January 2024 - ceph-users

Re: Stuck in upgrade process to reef
by Igor Fedotov 11 Jan '24

11 Jan '24

Hi Jan, unfortunately this wasn't very helpful. Moreover the log looks a bit messy - looks like a mixture of outputs from multiple running instances or something. I'm not an expert in using containerized setups though. Could you please simplify things by running ceph-osd process manually like you did for ceph-objectstore-tool. And enforce log output to a file. Command line should look somewhat the following: ceph-osd -i 0 --log-to-file --log-file <some-file> --debug-bluestore 5/20 --debug-prioritycache 10 Please don't forget to run repair prior to that. Also you haven't answered my questions about custom [memory] settings and RAM usage during OSD startup. It would be nice to hear some feedback. Thanks, Igor On 11/01/2024 16:47, Jan Marek wrote: > Hi Igor, > > I've tried to start osd.1 with debug_prioritycache and > debug_bluestore 5/20, see attached file... > > Sincerely > Jan > > Dne St, led 10, 2024 at 01:03:07 CET napsal(a) Igor Fedotov: >> Hi Jan, >> >> indeed this looks like some memory allocation problem - may be OSD's RAM >> usage threshold reached or something? >> >> Curious if you have any custom OSD settings or may be any memory caps for >> Ceph containers? >> >> Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 >> and try to start OSD once again. Please monitor process RAM usage along the >> process and share the resulting log. >> >> >> Thanks, >> >> Igor >> >> On 10/01/2024 11:20, Jan Marek wrote:

1 0

ceph-volume fails in all recent releases with IndexError
by Nico Schottelius 11 Jan '24

11 Jan '24

Hello dear fellow ceph users, it seems that for some months all current ceph releases (16.x, 17.x, 18.x) are having a bug in ceph-volume that causes disk activation to fail with the error "IndexError: list index out of range" (details below, [0]). It also seems there is already a fix for it available [1], but also that it hasn't been merged into any official release [2,3,4]. This has started to affect more and more nodes in our clusters and thus I was wondering if others are also seeing this issue and whether anyone knows whether it is planned to create a new release based on this soon? Best regards, Nico -------------------------------------------------------------------------------- [0] kubectl -n rook-ceph logs -c activate rook-ceph-osd-30-6558b7cf69-5cbbl + OSD_ID=30 + CEPH_FSID=bd3061a0-ecf3-4af6-9017-51b63c90b526 + OSD_UUID=319e5756-318c-46a0-b7e9-429e39069302 + OSD_STORE_FLAG=--bluestore + OSD_DATA_DIR=/var/lib/ceph/osd/ceph-30 + CV_MODE=raw + DEVICE=/dev/sdf + cp --no-preserve=mode /etc/temp-ceph/ceph.conf /etc/ceph/ceph.conf + python3 -c ' import configparser config = configparser.ConfigParser() config.read('\''/etc/ceph/ceph.conf'\'') if not config.has_section('\''global'\''): config['\''global'\''] = {} if not config.has_option('\''global'\'','\''fsid'\''): config['\''global'\'']['\''fsid'\''] = '\''....\'' with open('\''/etc/ceph/ceph.conf'\'', '\''w'\'') as configfile: config.write(configfile) ' + ceph -n client.admin auth get-or-create osd.30 mon 'allow profile osd' mgr 'allow profile osd' osd 'allow *' -k /etc/ceph/admin-keyring-store/keyring [osd.30] key = ... + [[ raw == \l\v\m ]] ++ mktemp + OSD_LIST=/tmp/tmp.OpZRJJOcrX + ceph-volume raw list /dev/sdf Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ self.main(self.argv) File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 166, in main self.list(args) File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 122, in list report = self.generate(args.device) File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 91, in generate info_device = [info for info in info_devices if info['NAME'] == dev][0] IndexError: list index out of range [1] https://github.com/ceph/ceph/pull/49954 [2] https://github.com/ceph/ceph/pull/54705 [3] https://github.com/ceph/ceph/pull/54706 [4] https://github.com/ceph/ceph/pull/54707 -- Sustainable and modern Infrastructures by ungleich.ch

2 1

Rack outage test failing when nodes get integrated again
by Steve Baker 11 Jan '24

11 Jan '24

Hi, we're currently testing a ceph (v 16.2.14) cluster, 3 mon nodes, 6 osd nodes à 8 nvme ssd osds distributed over 3 racks. Daemons are deployed in containers with cephadm / podman. We got 2 pools on it, one with 3x replication and min_size=2, one with an EC (k3m3). With 1 mon node and 2 osd nodes in each rack, the crush rules are configured in a way (for the 3x pool chooseleaf_firstn rack, for the ec pool choose_indep 3 rack / chooseleaf_indep 2 host), so that a full rack can go down while the cluster stays accessible for client operations. Other options we have set are mon_osd_down_out_subtree_limit=host so that in case of a host/rack outage the cluster does not automatically start to backfill, but will continue to run in a degraded state until human interaction comes to fix it. Also we set mon_osd_reporter_subtree_level=rack. We tested - while under (synthetic test-)client load - what happens if we take a full rack (one mon node and 2 osd nodes) out of the cluster. We did that using iptables to block the nodes of the rack from other nodes of the cluster (global and cluster network), as well as from the clients. As expected, the remainder of the cluster continues to run in a degraded state without to start any backfilling or recovery processes. All client requests gets served while the rack is out. But then a strange thing happens when we take the rack (1mon node, 2 osd nodes) back into the cluster again by deleting all firewall rules with iptables -F at once. Some osds get integrated in the cluster again immediatelly but some others remain in state "down" for exactly 10 minutes. These osds that stay down for the 10 minutes are in a state where they still seem to not be able to reach other osd nodes (see heartbeat_check logs below). After these 10 minutes have passed, these osds come up as well but then at exactly that time, many PGs get stuck in peering state and other osds that were all the time in the cluster get slow requests and the cluster blocks client traffic (I think it's just the PGs stuck in peering soaking all the client threads). Then, exactly 45 minutes after the nodes of the rack were made reachable by iptables -F again, the situation recovers, peering succeeds and client load gets handled again. We have repeated this test several times and it's always exactly the same 10 min "down interval" and a 45 min affected client requests. When we integrate the nodes into the cluster again one after another with a delay of some minutes inbetween, this does not happen at all. I wonder what's happening there. It must be some kind of split-brain situation having to do with blocking the nodes using iptables but not rebooting them completelly. The 10 min and 45 min intervals I described occure every time. For the 10 minutes, some osds stay down after the hosts got integrated again. It's not all of the 16 osds from the 2 osd hosts that got integrated again but just some of them. Which ones varies randomly. Sometimes it's also only just one. We also observerd, the longer the hosts were out of the cluster, the more osds are affected. Then even after they get up again after 10 minutes, it takes another 45 minutes until the stuck peering situation resolves. Also during these 45 minutes, we see slow ops on osds thet remained into the cluster. #################################################### See here some OSD logs that get written after the reintegration: #################################################### 2024-01-04T08:25:03.856+0000 7f369132b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-01-04T07:25:03.860426+0000) 2024-01-04T08:25:06.556+0000 7f3682882700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.0 down, but it is still running 2024-01-04T08:25:06.556+0000 7f3682882700 0 log_channel(cluster) log [DBG] : map e62160 wrongly marked me down at e62136 2024-01-04T08:25:06.556+0000 7f3682882700 1 osd.0 62160 start_waiting_for_healthy 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6810 osd.2 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6814 osd.3 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6822 osd.4 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6830 osd.5 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6830 osd.7 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6830 osd.9 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6802 osd.10 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6802 osd.11 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6802 osd.13 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6802 osd.15 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6806 osd.16 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6806 osd.17 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6806 osd.20 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6806 osd.21 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6810 osd.22 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6814 osd.23 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6810 osd.26 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6810 osd.27 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6814 osd.28 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6818 osd.29 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6814 osd.32 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6818 osd.33 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6818 osd.34 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6822 osd.35 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6818 osd.37 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6822 osd.39 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6822 osd.40 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6826 osd.41 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6826 osd.43 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6826 osd.44 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6826 osd.46 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6830 osd.47 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) [The block of loglines above gets repeated for 45 minutes until everything is fine again, the block below comes once at the beginning after the reintegration, then starts again after the 10 min interval and the repeats until the end of the 45 min interval until everything is fine again.] 2024-01-04T08:25:07.036+0000 7f3691b2c700 0 auth: could not find secret_id=1363 2024-01-04T08:25:07.036+0000 7f3691b2c700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1363 2024-01-04T08:25:07.036+0000 7f369132b700 0 auth: could not find secret_id=1363 2024-01-04T08:25:07.036+0000 7f369132b700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1363 2024-01-04T08:25:07.236+0000 7f369232d700 0 auth: could not find secret_id=1363 2024-01-04T08:25:07.236+0000 7f369232d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1363 2024-01-04T08:25:07.236+0000 7f369232d700 0 auth: could not find secret_id=1363 2024-01-04T08:25:07.236+0000 7f369232d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1363 2024-01-04T08:35:07.225+0000 7f369232d700 0 auth: could not find secret_id=1365 2024-01-04T08:35:07.225+0000 7f369232d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1365 [The block below gets logged for 10 minutes until the osd is not down anymore] 2024-01-04T08:25:08.368+0000 7f368d2a6700 1 osd.0 62162 is_healthy false -- only 0/10 up peers (less than 33%) 2024-01-04T08:25:08.368+0000 7f368d2a6700 1 osd.0 62162 not healthy; waiting to boot 2024-01-04T08:25:09.340+0000 7f368d2a6700 1 osd.0 62162 is_healthy false -- only 0/10 up peers (less than 33%) 2024-01-04T08:25:09.340+0000 7f368d2a6700 1 osd.0 62162 not healthy; waiting to boot 2024-01-04T08:25:10.316+0000 7f368d2a6700 1 osd.0 62162 is_healthy false -- only 0/10 up peers (less than 33%) 2024-01-04T08:25:10.316+0000 7f368d2a6700 1 osd.0 62162 not healthy; waiting to boot After 10 minutes then, the osd seems to reboot: 2024-01-04T08:35:07.005+0000 7f368d2a6700 1 osd.0 62509 start_boot 2024-01-04T08:35:07.009+0000 7f368b2a2700 1 osd.0 62509 set_numa_affinity storage numa node 0 2024-01-04T08:35:07.009+0000 7f368b2a2700 -1 osd.0 62509 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-04T08:35:07.009+0000 7f368b2a2700 1 osd.0 62509 set_numa_affinity not setting numa affinity 2024-01-04T08:35:07.197+0000 7f367ea40700 2 osd.0 62509 ms_handle_reset con 0x561d78ec6000 session 0x561d8ae5f0e0 2024-01-04T08:35:07.213+0000 7f3682882700 1 osd.0 62521 state: booting -> active ############################################################## See here some logs of the active mon that get written after the reintegration: ############################################################## 2024-01-04T08:25:06.486+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62160 send_latest to osd.0 v2:... start 62136 2024-01-04T08:25:06.486+0000 7ff5fa87b700 1 mon.ceph-mon01(a)0(leader).osd e62160 ignoring beacon from non-active osd.0 2024-01-04T08:25:06.490+0000 7ff5f9078700 0 log_channel(cluster) log [WRN] : osd.0 (root=default,rack=rack3,host=ceph-osd07) is down 2024-01-04T08:25:06.642+0000 7ff5f9078700 0 log_channel(cluster) log [INF] : osd.0 marked itself dead as of e62160 2024-01-04T08:25:07.434+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62161 preprocess_failure dne(/dup?): osd.0 [v2:...,v1:...], from osd.45 2024-01-04T08:29:59.998+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : osd.0 (root=default,rack=rack3,host=ceph-osd07) is down 2024-01-04T08:29:59.998+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Slow OSD heartbeats on back from osd.45 [rack3] to osd.0 [rack3] (down) 221701.273 msec 2024-01-04T08:29:59.998+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Slow OSD heartbeats on front from osd.45 [rack3] to osd.0 [rack3] (down) 221700.153 msec 2024-01-04T08:31:43.019+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62434 send_incremental [62416..62434] to osd.0 2024-01-04T08:32:13.915+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62443 send_incremental [62435..62443] to osd.0 2024-01-04T08:32:44.891+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62461 send_incremental [62444..62461] to osd.0 2024-01-04T08:33:15.752+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62465 send_incremental [62462..62465] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:51.240+0000 7ff5f9078700 5 mon.ceph-mon01(a)0(leader).osd e62484 send_incremental [62484..62484] to osd.0 2024-01-04T08:35:04.077+0000 7ff5fd080700 0 log_channel(cluster) log [INF] : Marking osd.0 out (has been down for 600 seconds) 2024-01-04T08:35:04.085+0000 7ff5fd080700 2 mon.ceph-mon01(a)0(leader).osd e62517 osd.0 OUT 2024-01-04T08:35:07.173+0000 7ff5fd080700 2 mon.ceph-mon01(a)0(leader).osd e62520 osd.0 UP [v2:...,v1:...] 2024-01-04T08:35:07.173+0000 7ff5fd080700 2 mon.ceph-mon01(a)0(leader).osd e62520 osd.0 IN This is what gets logged shortly before the situation recovers: 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health detail: HEALTH_WARN Reduced data availability: 292 pgs inactive, 292 pgs peering; Degraded data redundancy: 14614397/4091962683 objects degraded (0.357%), 322 pgs degraded, 360 pgs undersized; 2 pools have too many placement groups; 9472 slow ops, oldest one blocked for 2091 sec, daemons [osd.10,osd.11,osd.13,osd.14,osd.15,osd.16,osd.17> 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : [WRN] PG_AVAILABILITY: Reduced data availability: 292 pgs inactive, 292 pgs peering 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.212 is stuck peering for 34m, current state remapped+peering, last acting [4,21] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.220 is stuck peering for 34m, current state remapped+peering, last acting [3,35] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.227 is stuck peering for 34m, current state remapped+peering, last acting [44,26] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.228 is stuck peering for 34m, current state remapped+peering, last acting [23,5] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.22b is stuck peering for 34m, current state remapped+peering, last acting [32,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.234 is stuck peering for 34m, current state remapped+peering, last acting [32,44] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.24e is stuck peering for 34m, current state remapped+peering, last acting [17,32] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.255 is stuck peering for 34m, current state remapped+peering, last acting [4,22] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.260 is stuck peering for 34m, current state remapped+peering, last acting [17,47] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.267 is stuck peering for 34m, current state remapped+peering, last acting [44,23] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.27d is stuck peering for 34m, current state remapped+peering, last acting [4,21] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.289 is stuck peering for 34m, current state remapped+peering, last acting [13,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.292 is stuck peering for 34m, current state remapped+peering, last acting [15,10] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.297 is stuck peering for 34m, current state remapped+peering, last acting [2,21] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.29d is stuck peering for 34m, current state remapped+peering, last acting [40,23] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.2a8 is stuck peering for 34m, current state remapped+peering, last acting [33,4] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.20e is stuck inactive for 34m, current state remapped+peering, last acting [33,39,2147483647,2147483647,10,43] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.212 is stuck peering for 34m, current state remapped+peering, last acting [36,2147483647,22,40,10,43] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.219 is stuck peering for 34m, current state remapped+peering, last acting [13,10,21,34,2147483647,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.21c is stuck peering for 34m, current state remapped+peering, last acting [41,4,2147483647,14,44,15] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.222 is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,23,32,3,34] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.22d is stuck peering for 34m, current state remapped+peering, last acting [2147483647,45,41,20,17,33] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.233 is stuck peering for 34m, current state remapped+peering, last acting [4,2,27,34,14,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.23a is stuck peering for 34m, current state remapped+peering, last acting [41,43,19,2147483647,34,33] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.23b is stuck peering for 34m, current state remapped+peering, last acting [2147483647,30,7,41,34,15] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.23e is stuck peering for 34m, current state remapped+peering, last acting [10,37,2147483647,2147483647,11,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.243 is stuck peering for 34m, current state remapped+peering, last acting [23,13,11,15,2147483647,45] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.244 is stuck peering for 34m, current state remapped+peering, last acting [13,35,14,2147483647,17,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.249 is stuck peering for 34m, current state remapped+peering, last acting [32,47,2147483647,2147483647,46,17] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.24a is stuck peering for 34m, current state remapped+peering, last acting [47,7,15,5,2147483647,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.24b is stuck peering for 34m, current state remapped+peering, last acting [30,2147483647,46,28,4,29] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.24f is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,13,35,33,5] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.254 is stuck peering for 34m, current state remapped+peering, last acting [15,39,2147483647,2147483647,4,16] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.257 is stuck peering for 34m, current state remapped+peering, last acting [13,10,2147483647,2147483647,22,40] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.25d is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,20,16,34,46] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.26f is stuck peering for 34m, current state remapped+peering, last acting [33,17,2147483647,14,26,23] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.273 is stuck peering for 34m, current state remapped+peering, last acting [36,2147483647,29,4,17,21] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.27a is stuck peering for 34m, current state remapped+peering, last acting [40,34,2147483647,2147483647,26,23] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.27b is stuck peering for 34m, current state remapped+peering, last acting [41,37,2147483647,19,11,3] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.27e is stuck peering for 34m, current state remapped+peering, last acting [44,9,4,41,45,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.281 is stuck peering for 34m, current state remapped+peering, last acting [2,43,21,17,31,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.290 is stuck peering for 34m, current state remapped+peering, last acting [17,21,2147483647,2147483647,23,37] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.296 is stuck peering for 34m, current state remapped+peering, last acting [44,3,35,20,2147483647,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.298 is stuck peering for 34m, current state remapped+peering, last acting [2147483647,36,43,23,44,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2a5 is stuck peering for 34m, current state remapped+peering, last acting [33,11,19,2147483647,43,10] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2a6 is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,4,29,27,34] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2a8 is stuck peering for 34m, current state remapped+peering, last acting [47,4,2147483647,2147483647,15,5] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2ae is stuck peering for 34m, current state remapped+peering, last acting [23,32,2147483647,2147483647,3,22] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2b4 is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,23,20,15,34] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2b7 is stuck peering for 34m, current state remapped+peering, last acting [17,3,2147483647,45,41,32] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2b8 is stuck peering for 34m, current state remapped+peering, last acting [40,44,2147483647,2147483647,43,10] 2024-01-04T09:10:02.572+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 14296421/4091962644 objects degraded (0.349%), 322 pgs degraded, 360 pgs undersized (PG_DEGRADED) 2024-01-04T09:10:02.572+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: 9127 slow ops, oldest one blocked for 2096 sec, daemons [osd.10,osd.11,osd.13,osd.14,osd.15,osd.16,osd.17,osd.2,osd.20,osd.21]... have slow ops. (SLOW_OPS) 2024-01-04T09:10:27.608+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: Slow OSD heartbeats on back (longest 429402.973ms) (OSD_SLOW_PING_TIME_BACK) 2024-01-04T09:10:27.608+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: Slow OSD heartbeats on front (longest 429531.265ms) (OSD_SLOW_PING_TIME_FRONT) 2024-01-04T09:10:27.608+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 96918631/4091964189 objects degraded (2.369%), 404 pgs degraded, 241 pgs undersized (PG_DEGRADED) 2024-01-04T09:10:27.608+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: 706 slow ops, oldest one blocked for 2121 sec, daemons [osd.10,osd.11,osd.12,osd.13,osd.15,osd.16,osd.17,osd.18,osd.2,osd.20]... have slow ops. (SLOW_OPS) Any ideas what's going on here ?

2 1

Re: Sending notification after multiple objects are created in a ceph bucket.
by Lokendra Rathour 11 Jan '24

11 Jan '24

Facing a similar situation, any support would be helpful. -Lokendra On Tue, Jan 9, 2024 at 10:47 PM Kushagr Gupta <kushagrguptasps.mun(a)gmail.com> wrote: > Hi Team, > > Features used: Rados gateway, ceph S3 buckets > > We are trying to create a data pipeline using the S3 buckets capability > and rado gateway in ceph. > Our endpoint is a kafka topic. > > Currently, as soon as an object is created, a notification is sent to > kafka topic. > Our goal is to create multiple object first and then send the notification. > Also if there is a way such that we can assign objects to a group and > collectively send notification on an event on that group. > > Could anyone please help me? > > Thanks and Regards, > Kushagra Gupta > -- ~ Lokendra skype: lokendrarathour

2 1

physical vs osd performance
by Curt 10 Jan '24

10 Jan '24

Hello all, Looking at grafana reports, can anyone point me to documentation that outlines physical vs osd? https://docs.ceph.com/en/latest/monitoring/ gives some basic info, but I'm trying to get a better understanding. For instance, physical latency is 20ms and osd is 200ms, these are just made up for this example, why the huge difference? Same thing if for bytes or iops, just using latency as an example. Thanks, Curt

1 0

Re: Stuck in upgrade process to reef
by Igor Fedotov 10 Jan '24

10 Jan '24

Hi Jan, indeed this looks like some memory allocation problem - may be OSD's RAM usage threshold reached or something? Curious if you have any custom OSD settings or may be any memory caps for Ceph containers? Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 and try to start OSD once again. Please monitor process RAM usage along the process and share the resulting log. Thanks, Igor On 10/01/2024 11:20, Jan Marek wrote: > Hi Igor, > > I've tried to repair osd.1 with command: > > ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command repair > > and then start osd.1 ceph-osd podman service. > > It semms, that there is problem with memory allocation, see > attached log... > > Sincerely > Jan > > Dne Út, led 09, 2024 at 02:23:32 CET napsal(a) Igor Fedotov: >> Hi Marek, >> >> I haven't looked through those upgrade logs yet but here are some comments >> regarding last OSD startup attempt. >> >> First of answering your question >> >>> _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) >>> Is it a mandatory part of fsck? >> This is caused by previous non-graceful OSD process shutdown. BlueStore is unable to find up-to-date allocation map and recovers it from RocksDB. And since fsck is a read-only procedure the recovered allocmap is not saved - hence all the following BlueStore startups (within fsck or OSD init) cause another rebuild attempt. To avoid that you might want to run repair instead of fsck - this will persist up-to-date allocation map and avoid its rebuilding on the next startup. This will work till the next non-graceful shutdown only - hence unsuccessful OSD attempt might break the allocmap state again. >> >> Secondly - looking at OSD startup log one can see that actual OSD log ends with that allocmap recovery as well: >> >>> 2024-01-09T11:25:30.718449+01:00 osd1 ceph-osd[1734062]: bluestore(/var/lib/ceph/osd/ceph-1) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... >> Subsequent log line indicating OSD daemon termination is from systemd: >>> 2024-01-09T11:25:33.516258+01:00 osd1 systemd[1]: Stopping ceph-2c565e24-7850-47dc-a751-a6357cbbaf2a(a)osd.1.service - Ceph osd.1 for 2c565e24-7850-47dc-a751-a6357cbbaf2a... >> And honestly these lines provide almost no clue why termination happened. No obvious OSD failures or something are shown. Perhaps containerized environment hides the details e.g. by cutting off OSD log's tail. >> So you might want to proceed the investigation by running repair prior to starting the OSD as per above. This will result in no alloc map recovery and hopefully workaround the problem during startup - if the issue is caused by allocmap recovery. >> Additionally you might want to increase debug_bluestore log level for osd.1 before starting it up to get more insight on what's happening. >> >> Alternatively you might want to play with OSD log target settings to write OSD.1 log to some file rather than using system wide logging infra - hopefully this will be more helpful. >> >> Thanks, >> Igor >> >> On 09/01/2024 13:31, Jan Marek wrote: >>> Hi Igor, >>> >>> I've sent you logs via filesender.cesnet.cz, if someone would >>> be interested, they are here: >>> >>> https://filesender.cesnet.cz/?s=download&token=047b1ec4-4df0-4e8a-90fc-3170… >>> >>> Some points: >>> >>> 1) I've found, that on the osd1 server was bad time (3 minutes in >>> future). I've corrected that. Yes, I know, that it's bad, but we >>> moved servers to any other net segment, where they have no access >>> to the timeservers in Internet, then I must reconfigure it to use >>> our own NTP servers. >>> >>> 2) I've tried to start osd.1 service by this sequence: >>> >>> a) >>> >>> ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command fsck >>> >>> (without setting log properly :-( ) >>> >>> b) >>> >>> export CEPH_ARGS="--log-file osd.1.log --debug-bluestore 5/20" >>> ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command fsck >>> >>> - here I have one question: Why is it in this log stil this line: >>> >>> _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) >>> >>> Is it a mandatory part of fsck? >>> >>> Log is attached. >>> >>> c) >>> >>> systemctl start ceph-2c565e24-7850-47dc-a751-a6357cbbaf2a(a)osd.1.service >>> >>> still crashing, gzip-ed log attached too. >>> >>> Many thanks for exploring problem. >>> >>> Sincerely >>> Jan Marek >>> >>> Dne Po, led 08, 2024 at 12:00:05 CET napsal(a) Igor Fedotov: >>>> Hi Jan, >>>> >>>> indeed fsck logs for the OSDs other than osd.0 look good so it would be >>>> interesting to see OSD startup logs for them. Preferably to have that for >>>> multiple (e.g. 3-4) OSDs to get the pattern. >>>> >>>> Original upgrade log(s) would be nice to see as well. >>>> >>>> You might want to use Google Drive or any other publicly available file >>>> sharing site for that. >>>> >>>> >>>> Thanks, >>>> >>>> Igor >>>> >>>> On 05/01/2024 10:25, Jan Marek wrote: >>>>> Hi Igor, >>>>> >>>>> I've tried to start only osd.1, which seems to be fsck'd OK, but >>>>> it crashed :-( >>>>> >>>>> I search logs and I've found, that I have logs from 22.12.2023, >>>>> when I've did a upgrade (I have set logging to journald). >>>>> >>>>> Would you be interested in those logs? This file have 30MB in >>>>> bzip2 format, how I can share it with you? >>>>> >>>>> It contains crash log from start osd.1 too, but I can cut out >>>>> from it and send it to list... >>>>> >>>>> Sincerely >>>>> Jan Marek >>>>> >>>>> Dne Čt, led 04, 2024 at 02:43:48 CET napsal(a) Jan Marek: >>>>>> Hi Igor, >>>>>> >>>>>> I've ran this oneliner: >>>>>> >>>>>> for i in {0..12}; do export CEPH_ARGS="--log-file osd."${i}".log --debug-bluestore 5/20" ; ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.${i} --command fsck ; done; >>>>>> >>>>>> On osd.0 it crashed very quickly, on osd.1 it is still working. >>>>>> >>>>>> I've send those logs in one e-mail. >>>>>> >>>>>> But! >>>>>> >>>>>> I've tried to list disk devices in monitor view, and I've got >>>>>> very interesting screenshot - some part I've emphasized by red >>>>>> rectangulars. >>>>>> >>>>>> I've got a json from syslog, which was as a part cephadm call, >>>>>> where it seems to be correct (for my eyes). >>>>>> >>>>>> Can be this coincidence for this problem? >>>>>> >>>>>> Sincerely >>>>>> Jan Marek >>>>>> >>>>>> Dne Čt, led 04, 2024 at 12:32:47 CET napsal(a) Igor Fedotov: >>>>>>> Hi Jan, >>>>>>> >>>>>>> may I see the fsck logs from all the failing OSDs to see the pattern. IIUC >>>>>>> the full node is suffering from the issue, right? >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Igor >>>>>>> >>>>>>> On 1/2/2024 10:53 AM, Jan Marek wrote: >>>>>>>> Hello once again, >>>>>>>> >>>>>>>> I've tried this: >>>>>>>> >>>>>>>> export CEPH_ARGS="--log-file /tmp/osd.0.log --debug-bluestore 5/20" >>>>>>>> ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0 --command fsck >>>>>>>> >>>>>>>> And I've sending /tmp/osd.0.log file attached. >>>>>>>> >>>>>>>> Sincerely >>>>>>>> Jan Marek >>>>>>>> >>>>>>>> Dne Ne, pro 31, 2023 at 12:38:13 CET napsal(a) Igor Fedotov: >>>>>>>>> Hi Jan, >>>>>>>>> >>>>>>>>> this doesn't look like RocksDB corruption but rather like some BlueStore >>>>>>>>> metadata inconsistency. Also assertion backtrace in the new log looks >>>>>>>>> completely different from the original one. So in an attempt to find any >>>>>>>>> systematic pattern I'd suggest to run fsck with verbose logging for every >>>>>>>>> failing OSD. Relevant command line: >>>>>>>>> >>>>>>>>> CEPH_ARGS="--log-file osd.N.log --debug-bluestore 5/20" >>>>>>>>> bin/ceph-bluestore-tool --path <path-to-osd> --command fsck >>>>>>>>> >>>>>>>>> Unlikely this will fix anything it's rather a way to collect logs to get >>>>>>>>> better insight. >>>>>>>>> >>>>>>>>> >>>>>>>>> Additionally you might want to run similar fsck for a couple of healthy OSDs >>>>>>>>> - curious if it succeeds as I have a feeling that the problem with crashing >>>>>>>>> OSDs had been hidden before the upgrade and revealed rather than caused by >>>>>>>>> it. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Igor >>>>>>>>> >>>>>>>>> On 12/29/2023 3:28 PM, Jan Marek wrote: >>>>>>>>>> Hello Igor, >>>>>>>>>> >>>>>>>>>> I'm attaching a part of syslog creating while starting OSD.0. >>>>>>>>>> >>>>>>>>>> Many thanks for help. >>>>>>>>>> >>>>>>>>>> Sincerely >>>>>>>>>> Jan Marek >>>>>>>>>> >>>>>>>>>> Dne St, pro 27, 2023 at 04:42:56 CET napsal(a) Igor Fedotov: >>>>>>>>>>> Hi Jan, >>>>>>>>>>> >>>>>>>>>>> IIUC the attached log is for ceph-kvstore-tool, right? >>>>>>>>>>> >>>>>>>>>>> Can you please share full OSD startup log as well? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Igor >>>>>>>>>>> >>>>>>>>>>> On 12/27/2023 4:30 PM, Jan Marek wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> I've problem: my ceph cluster (3x mon nodes, 6x osd nodes, every >>>>>>>>>>>> osd node have 12 rotational disk and one NVMe device for >>>>>>>>>>>> bluestore DB). CEPH is installed by ceph orchestrator and have >>>>>>>>>>>> bluefs storage on osd. >>>>>>>>>>>> >>>>>>>>>>>> I've started process upgrade from version 17.2.6 to 18.2.1 by >>>>>>>>>>>> invocating: >>>>>>>>>>>> >>>>>>>>>>>> ceph orch upgrade start --ceph-version 18.2.1 >>>>>>>>>>>> >>>>>>>>>>>> After upgrade of mon and mgr processes orchestrator tried to >>>>>>>>>>>> upgrade the first OSD node, but they are falling down. >>>>>>>>>>>> >>>>>>>>>>>> I've stop the process of upgrade, but I have 1 osd node >>>>>>>>>>>> completely down. >>>>>>>>>>>> >>>>>>>>>>>> After upgrade I've got some error messages and I've found >>>>>>>>>>>> /var/lib/ceph/crashxxxx directories, I attach to this message >>>>>>>>>>>> files, which I've found here. >>>>>>>>>>>> >>>>>>>>>>>> Please, can you advice, what now I can do? It seems, that rocksdb >>>>>>>>>>>> is even non-compatible or corrupted :-( >>>>>>>>>>>> >>>>>>>>>>>> Thanks in advance. >>>>>>>>>>>> >>>>>>>>>>>> Sincerely >>>>>>>>>>>> Jan Marek >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>>> -- >>>>>>>>>>> Igor Fedotov >>>>>>>>>>> Ceph Lead Developer >>>>>>>>>>> >>>>>>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>>>>>> >>>>>>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Igor Fedotov >>>>>>>>> Ceph Lead Developer >>>>>>>>> >>>>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>>>> >>>>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>> -- >>>>>>> Igor Fedotov >>>>>>> Ceph Lead Developer >>>>>>> >>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>> >>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>> -- >>>>>> Ing. Jan Marek >>>>>> University of South Bohemia >>>>>> Academic Computer Centre >>>>>> Phone: +420389032080 >>>>>> http://www.gnu.org/philosophy/no-word-attachments.cs.html >>>>> >>>>> >>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

mds crashes after up:replay state
by Lars Köppel 09 Jan '24

09 Jan '24

Hello everyone, we are running a small cluster with 3 nodes and 25 osds per node. And Ceph version 17.2.6. Recently the active mds crashed and since then the new starting mds has always been in the up:replay state. In the output of the command 'ceph tell mds.cephfs:0 status' you can see that the journal is completely read in. As soon as it's finished, the mds crashes and the next one starts reading the journal. At the moment I have the journal inspection running ('cephfs-journal-tool --rank=cephfs:0 journal inspect'). Does anyone have any further suggestions on how I can get the cluster running again as quickly as possible? Best regards Lars [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel(a)ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai

4 7

How to configure something like osd_deep_scrub_min_interval?
by Frank Schilder 09 Jan '24

09 Jan '24

Hi folks, I am fighting a bit with odd deep-scrub behavior on HDDs and discovered a likely cause of why the distribution of last_deep_scrub_stamps is so weird. I wrote a small script to extract a histogram of scrubs by "days not scrubbed" (more precisely, intervals not scrubbed; see code) to find out how (deep-) scrub times are distributed. Output below. What I expected is along the lines that HDD-OSDs try to scrub every 1-3 days, while they try to deep-scrub every 7-14 days. In other words, OSDs that have been deep-scrubbed within the last 7 days would *never* be in scrubbing+deep state. However, what I see is completely different. There seems to be no distinction between scrub- and deep-scrub start times. This is really unexpected as nobody would try to deep-scrub HDDs every day. Weekly to bi-weekly is normal, specifically for large drives. Is there a way to configure something like osd_deep_scrub_min_interval (no, I don't want to run cron jobs for scrubbing yet)? In the output below, I would like to be able to configure a minimum period of 1-2 weeks before the next deep-scrub happens. How can I do that? The observed behavior is very unusual for RAID systems (if its not a bug in the report script). With this behavior its not surprising that people complain about "not deep-scrubbed in time" messages and too high deep-scrub IO load when such a large percentage of OSDs is needlessly deep-scrubbed after 1-6 days again already. Sample output: # scrub-report dumped pgs Scrub report: 4121 PGs not scrubbed since 1 intervals (6h) 3831 PGs not scrubbed since 2 intervals (6h) 4012 PGs not scrubbed since 3 intervals (6h) 3986 PGs not scrubbed since 4 intervals (6h) 2998 PGs not scrubbed since 5 intervals (6h) 1488 PGs not scrubbed since 6 intervals (6h) 909 PGs not scrubbed since 7 intervals (6h) 771 PGs not scrubbed since 8 intervals (6h) 582 PGs not scrubbed since 9 intervals (6h) 2 scrubbing 431 PGs not scrubbed since 10 intervals (6h) 333 PGs not scrubbed since 11 intervals (6h) 1 scrubbing 265 PGs not scrubbed since 12 intervals (6h) 195 PGs not scrubbed since 13 intervals (6h) 116 PGs not scrubbed since 14 intervals (6h) 78 PGs not scrubbed since 15 intervals (6h) 1 scrubbing 72 PGs not scrubbed since 16 intervals (6h) 37 PGs not scrubbed since 17 intervals (6h) 5 PGs not scrubbed since 18 intervals (6h) 14.237* 19.5cd* 19.12cc* 19.1233* 14.40e* 33 PGs not scrubbed since 20 intervals (6h) 23 PGs not scrubbed since 21 intervals (6h) 16 PGs not scrubbed since 22 intervals (6h) 12 PGs not scrubbed since 23 intervals (6h) 8 PGs not scrubbed since 24 intervals (6h) 2 PGs not scrubbed since 25 intervals (6h) 19.eef* 19.bb3* 4 PGs not scrubbed since 26 intervals (6h) 19.b4c* 19.10b8* 19.f13* 14.1ed* 5 PGs not scrubbed since 27 intervals (6h) 19.43f* 19.231* 19.1dbe* 19.1788* 19.16c0* 6 PGs not scrubbed since 28 intervals (6h) 2 PGs not scrubbed since 30 intervals (6h) 19.10f6* 14.9d* 3 PGs not scrubbed since 31 intervals (6h) 19.1322* 19.1318* 8.a* 1 PGs not scrubbed since 32 intervals (6h) 19.133f* 1 PGs not scrubbed since 33 intervals (6h) 19.1103* 3 PGs not scrubbed since 36 intervals (6h) 19.19cc* 19.12f4* 19.248* 1 PGs not scrubbed since 39 intervals (6h) 19.1984* 1 PGs not scrubbed since 41 intervals (6h) 14.449* 1 PGs not scrubbed since 44 intervals (6h) 19.179f* Deep-scrub report: 3723 PGs not deep-scrubbed since 1 intervals (24h) 4621 PGs not deep-scrubbed since 2 intervals (24h) 8 scrubbing+deep 3588 PGs not deep-scrubbed since 3 intervals (24h) 8 scrubbing+deep 2929 PGs not deep-scrubbed since 4 intervals (24h) 3 scrubbing+deep 1705 PGs not deep-scrubbed since 5 intervals (24h) 4 scrubbing+deep 1904 PGs not deep-scrubbed since 6 intervals (24h) 5 scrubbing+deep 1540 PGs not deep-scrubbed since 7 intervals (24h) 7 scrubbing+deep 1304 PGs not deep-scrubbed since 8 intervals (24h) 7 scrubbing+deep 923 PGs not deep-scrubbed since 9 intervals (24h) 5 scrubbing+deep 557 PGs not deep-scrubbed since 10 intervals (24h) 7 scrubbing+deep 501 PGs not deep-scrubbed since 11 intervals (24h) 2 scrubbing+deep 363 PGs not deep-scrubbed since 12 intervals (24h) 2 scrubbing+deep 377 PGs not deep-scrubbed since 13 intervals (24h) 1 scrubbing+deep 383 PGs not deep-scrubbed since 14 intervals (24h) 2 scrubbing+deep 252 PGs not deep-scrubbed since 15 intervals (24h) 2 scrubbing+deep 116 PGs not deep-scrubbed since 16 intervals (24h) 5 scrubbing+deep 47 PGs not deep-scrubbed since 17 intervals (24h) 2 scrubbing+deep 10 PGs not deep-scrubbed since 18 intervals (24h) 2 PGs not deep-scrubbed since 19 intervals (24h) 19.1c6c* 19.a01* 1 PGs not deep-scrubbed since 20 intervals (24h) 14.1ed* 2 PGs not deep-scrubbed since 21 intervals (24h) 19.1322* 19.10f6* 1 PGs not deep-scrubbed since 23 intervals (24h) 19.19cc* 1 PGs not deep-scrubbed since 24 intervals (24h) 19.179f* PGs marked with a * are on busy OSDs and not eligible for scrubbing. The script (pasted here because attaching doesn't work): # cat bin/scrub-report #!/bin/bash # Compute last scrub interval count. Scrub interval 6h, deep-scrub interval 24h. # Print how many PGs have not been (deep-)scrubbed since #intervals. ceph -f json pg dump pgs 2>&1 > /root/.cache/ceph/pgs_dump.json echo "" T0="$(date +%s)" scrub_info="$(jq --arg T0 "$T0" -rc '.pg_stats[] | [ .pgid, (.last_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*6)|ceil), (.last_deep_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*24)|ceil), .state, (.acting | join(" ")) ] | @tsv ' /root/.cache/ceph/pgs_dump.json)" # less <<<"$scrub_info" # 1 2 3 4 5..NF # pg_id scrub-ints deep-scrub-ints status acting[] awk <<<"$scrub_info" '{ for(i=5; i<=NF; ++i) pg_osds[$1]=pg_osds[$1] " " $i if($4 == "active+clean") { si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 } else if($4 ~ /scrubbing\+deep/) { deep_scrubbing[$3]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else if($4 ~ /scrubbing/) { scrubbing[$2]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else { unclean[$2]++ unclean_d[$3]++ si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 for(i=5; i<=NF; ++i) osd[$i]="busy" } } END { print "Scrub report:" for(si=1; si<=si_mx; ++si) { if(pg_sn[si]==0 && scrubbing[si]==0 && unclean[si]==0) continue; printf("%7d PGs not scrubbed since %2d intervals (6h)", pg_sn[si], si) if(scrubbing[si]) printf(" %d scrubbing", scrubbing[si]) if(unclean[si]) printf(" %d unclean", unclean[si]) if(pg_sn[si]<=5) { split(pg_sn_ids[si], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "Deep-scrub report:" for(dsi=1; dsi<=dsi_mx; ++dsi) { if(pg_dsn[dsi]==0 && deep_scrubbing[dsi]==0 && unclean_d[dsi]==0) continue; printf("%7d PGs not deep-scrubbed since %2d intervals (24h)", pg_dsn[dsi], dsi) if(deep_scrubbing[dsi]) printf(" %d scrubbing+deep", deep_scrubbing[dsi]) if(unclean_d[dsi]) printf(" %d unclean", unclean_d[dsi]) if(pg_dsn[dsi]<=5) { split(pg_dsn_ids[dsi], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "PGs marked with a * are on busy OSDs and not eligible for scrubbing." } ' Don't forget the last "'" when copy-pasting. Thanks for any pointers. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 6

Re: Stuck in upgrade process to reef
by Igor Fedotov 09 Jan '24

09 Jan '24

Hi Marek, I haven't looked through those upgrade logs yet but here are some comments regarding last OSD startup attempt. First of answering your question >_init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) >Is it a mandatory part of fsck? This is caused by previous non-graceful OSD process shutdown. BlueStore is unable to find up-to-date allocation map and recovers it from RocksDB. And since fsck is a read-only procedure the recovered allocmap is not saved - hence all the following BlueStore startups (within fsck or OSD init) cause another rebuild attempt. To avoid that you might want to run repair instead of fsck - this will persist up-to-date allocation map and avoid its rebuilding on the next startup. This will work till the next non-graceful shutdown only - hence unsuccessful OSD attempt might break the allocmap state again. Secondly - looking at OSD startup log one can see that actual OSD log ends with that allocmap recovery as well: > 2024-01-09T11:25:30.718449+01:00 osd1 ceph-osd[1734062]: bluestore(/var/lib/ceph/osd/ceph-1) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... Subsequent log line indicating OSD daemon termination is from systemd: > 2024-01-09T11:25:33.516258+01:00 osd1 systemd[1]: Stopping ceph-2c565e24-7850-47dc-a751-a6357cbbaf2a(a)osd.1.service - Ceph osd.1 for 2c565e24-7850-47dc-a751-a6357cbbaf2a... And honestly these lines provide almost no clue why termination happened. No obvious OSD failures or something are shown. Perhaps containerized environment hides the details e.g. by cutting off OSD log's tail. So you might want to proceed the investigation by running repair prior to starting the OSD as per above. This will result in no alloc map recovery and hopefully workaround the problem during startup - if the issue is caused by allocmap recovery. Additionally you might want to increase debug_bluestore log level for osd.1 before starting it up to get more insight on what's happening. Alternatively you might want to play with OSD log target settings to write OSD.1 log to some file rather than using system wide logging infra - hopefully this will be more helpful. Thanks, Igor On 09/01/2024 13:31, Jan Marek wrote: > Hi Igor, > > I've sent you logs via filesender.cesnet.cz, if someone would > be interested, they are here: > > https://filesender.cesnet.cz/?s=download&token=047b1ec4-4df0-4e8a-90fc-3170… > > Some points: > > 1) I've found, that on the osd1 server was bad time (3 minutes in > future). I've corrected that. Yes, I know, that it's bad, but we > moved servers to any other net segment, where they have no access > to the timeservers in Internet, then I must reconfigure it to use > our own NTP servers. > > 2) I've tried to start osd.1 service by this sequence: > > a) > > ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command fsck > > (without setting log properly :-( ) > > b) > > export CEPH_ARGS="--log-file osd.1.log --debug-bluestore 5/20" > ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command fsck > > - here I have one question: Why is it in this log stil this line: > > _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) > > Is it a mandatory part of fsck? > > Log is attached. > > c) > > systemctl start ceph-2c565e24-7850-47dc-a751-a6357cbbaf2a(a)osd.1.service > > still crashing, gzip-ed log attached too. > > Many thanks for exploring problem. > > Sincerely > Jan Marek > > Dne Po, led 08, 2024 at 12:00:05 CET napsal(a) Igor Fedotov: >> Hi Jan, >> >> indeed fsck logs for the OSDs other than osd.0 look good so it would be >> interesting to see OSD startup logs for them. Preferably to have that for >> multiple (e.g. 3-4) OSDs to get the pattern. >> >> Original upgrade log(s) would be nice to see as well. >> >> You might want to use Google Drive or any other publicly available file >> sharing site for that. >> >> >> Thanks, >> >> Igor >> >> On 05/01/2024 10:25, Jan Marek wrote: >>> Hi Igor, >>> >>> I've tried to start only osd.1, which seems to be fsck'd OK, but >>> it crashed :-( >>> >>> I search logs and I've found, that I have logs from 22.12.2023, >>> when I've did a upgrade (I have set logging to journald). >>> >>> Would you be interested in those logs? This file have 30MB in >>> bzip2 format, how I can share it with you? >>> >>> It contains crash log from start osd.1 too, but I can cut out >>> from it and send it to list... >>> >>> Sincerely >>> Jan Marek >>> >>> Dne Čt, led 04, 2024 at 02:43:48 CET napsal(a) Jan Marek: >>>> Hi Igor, >>>> >>>> I've ran this oneliner: >>>> >>>> for i in {0..12}; do export CEPH_ARGS="--log-file osd."${i}".log --debug-bluestore 5/20" ; ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.${i} --command fsck ; done; >>>> >>>> On osd.0 it crashed very quickly, on osd.1 it is still working. >>>> >>>> I've send those logs in one e-mail. >>>> >>>> But! >>>> >>>> I've tried to list disk devices in monitor view, and I've got >>>> very interesting screenshot - some part I've emphasized by red >>>> rectangulars. >>>> >>>> I've got a json from syslog, which was as a part cephadm call, >>>> where it seems to be correct (for my eyes). >>>> >>>> Can be this coincidence for this problem? >>>> >>>> Sincerely >>>> Jan Marek >>>> >>>> Dne Čt, led 04, 2024 at 12:32:47 CET napsal(a) Igor Fedotov: >>>>> Hi Jan, >>>>> >>>>> may I see the fsck logs from all the failing OSDs to see the pattern. IIUC >>>>> the full node is suffering from the issue, right? >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Igor >>>>> >>>>> On 1/2/2024 10:53 AM, Jan Marek wrote: >>>>>> Hello once again, >>>>>> >>>>>> I've tried this: >>>>>> >>>>>> export CEPH_ARGS="--log-file /tmp/osd.0.log --debug-bluestore 5/20" >>>>>> ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0 --command fsck >>>>>> >>>>>> And I've sending /tmp/osd.0.log file attached. >>>>>> >>>>>> Sincerely >>>>>> Jan Marek >>>>>> >>>>>> Dne Ne, pro 31, 2023 at 12:38:13 CET napsal(a) Igor Fedotov: >>>>>>> Hi Jan, >>>>>>> >>>>>>> this doesn't look like RocksDB corruption but rather like some BlueStore >>>>>>> metadata inconsistency. Also assertion backtrace in the new log looks >>>>>>> completely different from the original one. So in an attempt to find any >>>>>>> systematic pattern I'd suggest to run fsck with verbose logging for every >>>>>>> failing OSD. Relevant command line: >>>>>>> >>>>>>> CEPH_ARGS="--log-file osd.N.log --debug-bluestore 5/20" >>>>>>> bin/ceph-bluestore-tool --path <path-to-osd> --command fsck >>>>>>> >>>>>>> Unlikely this will fix anything it's rather a way to collect logs to get >>>>>>> better insight. >>>>>>> >>>>>>> >>>>>>> Additionally you might want to run similar fsck for a couple of healthy OSDs >>>>>>> - curious if it succeeds as I have a feeling that the problem with crashing >>>>>>> OSDs had been hidden before the upgrade and revealed rather than caused by >>>>>>> it. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Igor >>>>>>> >>>>>>> On 12/29/2023 3:28 PM, Jan Marek wrote: >>>>>>>> Hello Igor, >>>>>>>> >>>>>>>> I'm attaching a part of syslog creating while starting OSD.0. >>>>>>>> >>>>>>>> Many thanks for help. >>>>>>>> >>>>>>>> Sincerely >>>>>>>> Jan Marek >>>>>>>> >>>>>>>> Dne St, pro 27, 2023 at 04:42:56 CET napsal(a) Igor Fedotov: >>>>>>>>> Hi Jan, >>>>>>>>> >>>>>>>>> IIUC the attached log is for ceph-kvstore-tool, right? >>>>>>>>> >>>>>>>>> Can you please share full OSD startup log as well? >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Igor >>>>>>>>> >>>>>>>>> On 12/27/2023 4:30 PM, Jan Marek wrote: >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I've problem: my ceph cluster (3x mon nodes, 6x osd nodes, every >>>>>>>>>> osd node have 12 rotational disk and one NVMe device for >>>>>>>>>> bluestore DB). CEPH is installed by ceph orchestrator and have >>>>>>>>>> bluefs storage on osd. >>>>>>>>>> >>>>>>>>>> I've started process upgrade from version 17.2.6 to 18.2.1 by >>>>>>>>>> invocating: >>>>>>>>>> >>>>>>>>>> ceph orch upgrade start --ceph-version 18.2.1 >>>>>>>>>> >>>>>>>>>> After upgrade of mon and mgr processes orchestrator tried to >>>>>>>>>> upgrade the first OSD node, but they are falling down. >>>>>>>>>> >>>>>>>>>> I've stop the process of upgrade, but I have 1 osd node >>>>>>>>>> completely down. >>>>>>>>>> >>>>>>>>>> After upgrade I've got some error messages and I've found >>>>>>>>>> /var/lib/ceph/crashxxxx directories, I attach to this message >>>>>>>>>> files, which I've found here. >>>>>>>>>> >>>>>>>>>> Please, can you advice, what now I can do? It seems, that rocksdb >>>>>>>>>> is even non-compatible or corrupted :-( >>>>>>>>>> >>>>>>>>>> Thanks in advance. >>>>>>>>>> >>>>>>>>>> Sincerely >>>>>>>>>> Jan Marek >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>> -- >>>>>>>>> Igor Fedotov >>>>>>>>> Ceph Lead Developer >>>>>>>>> >>>>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>>>> >>>>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>>>> >>>>>>> -- >>>>>>> Igor Fedotov >>>>>>> Ceph Lead Developer >>>>>>> >>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>> >>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> -- >>>>> Igor Fedotov >>>>> Ceph Lead Developer >>>>> >>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>> >>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> -- >>>> Ing. Jan Marek >>>> University of South Bohemia >>>> Academic Computer Centre >>>> Phone: +420389032080 >>>> http://www.gnu.org/philosophy/no-word-attachments.cs.html >>> >>> >>> >>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

REST API Endpoint Failure - Request For Where To Look To Resolve
by duluxoz 09 Jan '24

09 Jan '24

Hi All, A little help please. TL/DR: Please help with error message: ~~~ REST API failure, code : 500 Unable to access the configuration object Unable to contact the local API endpoint (https://localhost:5000/api) ~~~ The Issue ------------ I've been through the documentation and can't find what I'm looking for - possibly because I'm not really sure what it is I *am* looking for, so if someone can point me in the right direction I would really appreciate it. I get the above error message when I run the `gwcli` command from inside a cephadm shell. What I'm trying to do is set up a set of iSCSI Gateways in our Ceph-Reef 18.2.1 Cluster (yes, I know its being depreciated as of Nov 22 - or whatever). We recently migrated 7 upgraded from a manual install of Quincy to a CephAdm install of Reef - everything went AOK *except* for the iSCSI Gateways. So we tore them down and then rebuilt them as per the latest documentation. So now we've got 3 gateways as per the Service page of the Dashboard and I'm trying to create the targets. I tried via the Dashboard but had errors, so instead I went in to do it via gwcli and hit the above error (which I now bevel to be the cause of the GUI creation I encountered. I have absolutely no experience with podman or containers in general, and can't work out how to fix the issue. So I'm requesting some help - not to solve the problem for me, but to point me in the right direction to solve it myself. :-) So, anyone? Cheers Dulux-Oz

2 9

Re: rbd persistent cache configuration
by Peter 08 Jan '24

08 Jan '24

rbd --version ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)

2 1

Radosgw not syncing files/folders with slashes in object name
by Matt Dunavant 08 Jan '24

08 Jan '24

Hi all, I need some help troubleshooting a strange issue where my relatively newly setup 2 ceph clusters (17.2.6) are setup for replication and I can get files with just regular names (example.txt for example) to sync but anything with a slash or a folder type (folder1/folder2/example.txt for example) won't sync over. Not sure exactly why this would be the case as I'm pretty sure slashes are allowed in object names, https://docs.ceph.com/en/latest/radosgw/layout/. Any ideas or something obvious I'm missing? Sync status looks normal and I have tested this with a variety of new and old buckets and the behavior always stays the same, nothing with a slash syncs but everything without does. Thanks in advance, -Matt Dunavant

1 0

Network Flapping Causing Slow Ops and Freezing VMs
by mahnoosh shahidi 08 Jan '24

08 Jan '24

Hi all, I hope this message finds you well. We recently encountered an issue on one of our OSD servers, leading to network flapping and subsequently causing significant performance degradation across our entire cluster. Although the OSDs were correctly marked as down in the monitor, slow ops persisted until we resolved the network issue. This incident resulted in a major disruption, especially affecting VMs with mapped RBD images, leading to their freezing. In light of this, I have two key questions for the community: 1. Why did slow ops persist even after marking the affected server as down in the monitor? 2.Are there any recommended configurations for OSD suicide or OSD down reports that could help us better handle similar network-related issues in the future? Best Regards, Mahnoosh

3 5

cephadm bootstrap on 3 network clusters
by Luis Domingues 08 Jan '24

08 Jan '24

Hi, I am bootstrapping a ceph cluster using cephadm, and our cluster uses 3 networks. We have - 1 network as public network (10.X.X.0/24) (pub) - 1 network as cluster network (10.X.Y.0/24) (cluster) - 1 network for management (172.Z.Z.0/24) (mgmt) The nodes are reachable using SSH only on mgmt network. However, they are reachable for our services using pub network. I want my MONs to be bind to this pub network. But when I bootstrap my cluster, I set my MON IP and CLUSTER NETWORK, and then the bootstrap process tries to add my bootstrap node using the MON IP. And then fails because it cannot reach the node. If I apply proper spec after it works fine, but the bootstrap process did not finish properly. Is there an option to tell cephadm to not use MON IP but another one for accessing the node during the bootstrap? If I tell it --skip-prepare-host, it tries to connect to it anyway, and then fails. Thanks, Luis Domingues Proton AG

4 6

osd_mclock_max_capacity_iops_hdd in Reef
by Luis Domingues 08 Jan '24

08 Jan '24

Hi all, We are testing migrations from a cluster running Pacific to Reef. In pacific we needed to tweak osd_mclock_max_capacity_iops_hdd to have decent performances of ou cluster. But in reef it looks like changing the value of osd_mclock_max_capacity_iops_hdd does not impact cluster performances. Did osd_mclock_max_capacity_iops_hdd became useless? I did not found anything regarding it on the changelogs, but I could have miss something. Luis Domingues Proton AG

2 3

Stuck in upgrade process to reef
by Jan Marek 08 Jan '24

08 Jan '24

Hello, I've problem: my ceph cluster (3x mon nodes, 6x osd nodes, every osd node have 12 rotational disk and one NVMe device for bluestore DB). CEPH is installed by ceph orchestrator and have bluefs storage on osd. I've started process upgrade from version 17.2.6 to 18.2.1 by invocating: ceph orch upgrade start --ceph-version 18.2.1 After upgrade of mon and mgr processes orchestrator tried to upgrade the first OSD node, but they are falling down. I've stop the process of upgrade, but I have 1 osd node completely down. After upgrade I've got some error messages and I've found /var/lib/ceph/crashxxxx directories, I attach to this message files, which I've found here. Please, can you advice, what now I can do? It seems, that rocksdb is even non-compatible or corrupted :-( Thanks in advance. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

2 8

ceph -s: wrong host count
by Jan Kasprzak 08 Jan '24

08 Jan '24

Hello, Ceph users! I have recently noticed that when I reboot a single ceph node, ceph -s reports "5 hosts down" instead of one. The following is captured during reboot of a node with two OSDs: health: HEALTH_WARN noout flag(s) set 2 osds down 5 hosts (2 osds) down [...] mon: 3 daemons, quorum mon1,mon3,mon2 (age 8h) mgr: mon2(active, since 2d), standbys: mon3, mon1 osd: 34 osds: 32 up (since 2m), 34 in (since 4M) flags noout rgw: 1 daemon active (1 hosts, 1 zones) After the node successfully reboots, ceph -s reports "HEALTH OK" and of course no OSDs and no hosts are reported as being down. Does anybody else see this as well? This is Ceph 18.2.1, but I think I have seen this on Ceph 17 as well. Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

Problems with "admin" bucket
by Manuel Holtgrewe 08 Jan '24

08 Jan '24

Hello, I have an issue with a Ceph 17.2.6 cluster. The dashboard says "The Object Gateway Service is not configured" when trying to access the Object Gateway section. It used to work before. One interesting symptom: the "admin" bucket exists in the output of "radosgw-admin bucket list" but it does not exist in "radosgw-admin bucket stats". Rather, I get a number of "ERROR: could not decode buffer info, caught buffer::error" messages from the "radosgw-admin bucket stats" command. Also, I cannot remove the "admin" bucket because I also get the same error (I thought about starting fresh with the admin bucket). Could someone help me debug this further and eventually resolve the issue? There is no critical data in the radosgw buckets (the cluster is primarily accessed via an CephFS cluster), so clearing all radosgw buckets is an option. Ideally, I could repair this, however. Kind regards, Manuel

1 1

No metrics shown in dashboard (18.2.1)
by Dietmar Rieder 06 Jan '24

06 Jan '24

Hi, I just freshly deployed a new cluster (v18.2.1) using cephadm. Now before creating pools, cephfs and so on I wanted to check if the dashboard is working and if I get some metrics. If I navigate to Cluster >> Hosts and open one of the OSD hosts the "Performance Details" tab is shown but all graphs display "no data". "OSDs" and "Raw Capacity" in that tab display "N/A". Prometheus is running: [root@cephmon-01 ~]# ceph orch ps --service_name prometheus NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID prometheus.cephmon-01 cephmon-01 *:9095 running (54m) 9m ago 3d 51.4M - 2.43.0 a07b618ecd1d 11b4b19df0d6 However it has no data collected: [root@cephmon-01 ~]# curl -s -XGET http://127.0.0.1:9095/api/v1/targets/metadata {"status":"success","data":[]} ceph-exporter services also seem to be running: [root@cephmon-01 ~]# ceph orch ps --service_name ceph-exporter NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID ceph-exporter.cephmon-01 cephmon-01 running (46h) 10m ago 3d 16.7M - 18.2.1 d2cdd87030d1 191eedccbcd8 ceph-exporter.cephmon-02 cephmon-02 running (3d) 2m ago 3d 16.5M - 18.2.1 d2cdd87030d1 7a08e9f1401c ceph-exporter.cephmon-03 cephmon-03 running (3d) 6m ago 3d 16.5M - 18.2.1 d2cdd87030d1 1eb4856a60d4 ceph-exporter.cephosd-01 cephosd-01 running (2d) 10m ago 2d 19.9M - 18.2.1 d2cdd87030d1 05642098f7de ceph-exporter.cephosd-02 cephosd-02 running (2d) 10m ago 2d 19.9M - 18.2.1 d2cdd87030d1 648715ecaa9d ceph-exporter.cephosd-03 cephosd-03 running (2d) 10m ago 2d 19.4M - 18.2.1 d2cdd87030d1 b8bb6dcb5386 ceph-exporter.cephosd-04 cephosd-04 running (2d) 10m ago 2d 19.5M - 18.2.1 d2cdd87030d1 4f1964f79ffe ceph-exporter.cephosd-05 cephosd-05 running (2d) 10m ago 2d 19.8M - 18.2.1 d2cdd87030d1 8ca8cbbf3984 ceph-exporter.cephosd-06 cephosd-06 running (2d) 10m ago 2d 19.4M - 18.2.1 d2cdd87030d1 a5e2860cc98e ceph-exporter.cephosd-07 cephosd-07 running (2d) 3m ago 2d 19.8M - 18.2.1 d2cdd87030d1 4eb01b8ebd33 ceph-exporter.cephosd-08 cephosd-08 running (2d) 3m ago 2d 19.9M - 18.2.1 d2cdd87030d1 b934866d2a1d ceph-exporter.cephosd-10 cephosd-10 running (2d) 3m ago 2d 19.4M - 18.2.1 d2cdd87030d1 457368d07579 ceph-exporter.cephosd-11 cephosd-11 running (2d) 3m ago 2d 19.5M - 18.2.1 d2cdd87030d1 e561cfac4209 ceph-exporter.cephosd-12 cephosd-12 running (2d) 9m ago 2d 19.9M - 18.2.1 d2cdd87030d1 0e5773c8e038 as well as node-exporter services: [root@cephmon-01 ~]# ceph orch ps --service_name node-exporter NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID node-exporter.cephmon-01 cephmon-01 *:9100 running (46h) 19s ago 3d 11.3M - 1.5.0 0da6a335fe13 72fefb7966ff node-exporter.cephmon-02 cephmon-02 *:9100 running (3d) 3m ago 3d 13.5M - 1.5.0 0da6a335fe13 2041d3d385b0 node-exporter.cephmon-03 cephmon-03 *:9100 running (3d) 7m ago 3d 12.9M - 1.5.0 0da6a335fe13 6ef204d12a7d node-exporter.cephosd-01 cephosd-01 *:9100 running (2d) 18s ago 2d 13.0M - 1.5.0 0da6a335fe13 6b05483b05c6 node-exporter.cephosd-02 cephosd-02 *:9100 running (2d) 18s ago 2d 11.4M - 1.5.0 0da6a335fe13 ede4995ffb1d node-exporter.cephosd-03 cephosd-03 *:9100 running (2d) 18s ago 2d 11.5M - 1.5.0 0da6a335fe13 cfbf15168667 node-exporter.cephosd-04 cephosd-04 *:9100 running (2d) 18s ago 2d 13.2M - 1.5.0 0da6a335fe13 5dc4794a7f6e node-exporter.cephosd-05 cephosd-05 *:9100 running (2d) 18s ago 2d 13.4M - 1.5.0 0da6a335fe13 8dfa1e252f82 node-exporter.cephosd-06 cephosd-06 *:9100 running (2d) 18s ago 2d 13.5M - 1.5.0 0da6a335fe13 93467e37df08 node-exporter.cephosd-07 cephosd-07 *:9100 running (2d) 4m ago 2d 13.2M - 1.5.0 0da6a335fe13 11795b83732d node-exporter.cephosd-08 cephosd-08 *:9100 running (2d) 4m ago 2d 13.5M - 1.5.0 0da6a335fe13 04197f3a6eb1 node-exporter.cephosd-10 cephosd-10 *:9100 running (2d) 4m ago 2d 13.2M - 1.5.0 0da6a335fe13 9e904581442c node-exporter.cephosd-11 cephosd-11 *:9100 running (2d) 4m ago 2d 13.0M - 1.5.0 0da6a335fe13 5164113044ed node-exporter.cephosd-12 cephosd-12 *:9100 running (2d) 16s ago 2d 13.3M - 1.5.0 0da6a335fe13 5c8af368eed4 I'm a bit lost, how can I get this running? Thanks for any help Dietmar

1 1

Re: rbd persistent cache configuration
by Peter 06 Jan '24

06 Jan '24

Thanks for ressponse! Yes, it is in use "watcher=10.1.254.51:0/1544956346 client.39553300 cookie=140244238214096" this is indicating the client is connect the image. I am using fio perform write task on it. I guess it is the feature not enable correctly or setting somewhere incorrect. Should I restart any process after modifying Ceph config? Any thought?

2 1

rbd persistent cache configuration
by Peter 05 Jan '24

05 Jan '24

I follow below document to setup image level rbd persistent cache, however I get error output while i using the command provide by the document. I have put my commands and descriptions below. Can anyone give some instructions? thanks in advance. https://docs.ceph.com/en/pacific/rbd/rbd-persistent-write-back-cache/ https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/b… [https://access.redhat.com/webassets/avalon/g/shadowman-200.png]<https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/b…> Chapter 2. Ceph block devices Red Hat Ceph Storage 5 | Red Hat Customer Portal<https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/b…> Access Red Hat’s knowledge, guidance, and support through your subscription. access.redhat.com I tried use host level client command, i got no error, however I won't be able to get cache usage output. "ceph config set client rbd_persistent_cache_mode ssd ceph config set client rbd_plugins pwl_cache" [root@master-node1 ceph]# rbd info sas-pool/testdrive rbd image 'testdrive': size 40 GiB in 10240 objects order 22 (4 MiB objects) snapshot_count: 0 id: 3de76a7e7c519 block_name_prefix: rbd_data.3de76a7e7c519 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Jun 29 02:03:41 2023 access_timestamp: Thu Jun 29 07:19:40 2023 modify_timestamp: Thu Jun 29 07:18:00 2023 I check feature exclusive-lock has been already enabled and I run following command get fault output. [root@master-node1 ceph]# rbd config image set sas-pool/testdrive rbd_persistent_cache_mode ssd rbd: invalid config key: rbd_persistent_cache_mode [root@master-node1 ceph]# rbd config image set sas-pool/testdrive rbd_plugins pwl_cache rbd: invalid config key: rbd_plugins root@node1:~# rbd status sas-pool/testdrive Watchers: watcher=10.1.254.51:0/1544956346 client.39553300 cookie=140244238214096 I hope to see the output include the persistent cache state like below: $ rbd status rbd/foo Watchers: watcher=10.10.0.102:0/1061883624 client.25496 cookie=140338056493088 Persistent cache state: host: sceph9 path: /mnt/nvme0/rbd-pwl.rbd.101e5824ad9a.pool size: 1 GiB mode: ssd stats_timestamp: Sun Apr 10 13:26:32 2022 present: true empty: false clean: false allocated: 509 MiB cached: 501 MiB dirty: 338 MiB free: 515 MiB hits_full: 1450 / 61% hits_partial: 0 / 0% misses: 924 hit_bytes: 192 MiB / 66% miss_bytes: 97 MiB

2 1

CEPH create an pool with 256 PGs stuck peering
by Phong Tran Thanh 05 Jan '24

05 Jan '24

Hi community. I' am running ceph cluster with 10 node and 180 osds, and i create an pool erasure code 4+2 with 256 PGs, but when create an pool PG too slow, and pg status stuck peering EALTH_WARN Reduced data availability: 5 pgs inactive, 5 pgs peering [WRN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive, 5 pgs peering pg 59.6b is stuck peering for 4m, current state creating+peering, last acting [17,87,92,117,71,149] pg 59.78 is stuck peering for 4m, current state creating+peering, last acting [94,16,137,98,41,79] pg 59.86 is stuck peering for 4m, current state creating+peering, last acting [37,107,24,138,144,25] and this is a pg query "recovery_state": [ { "name": "Started/Primary/Peering/GetInfo", "enter_time": "2024-01-04T11:02:09.208218+0000", "requested_info_from": [ { "osd": "101(4)" } ] }, { "name": "Started/Primary/Peering", "enter_time": "2024-01-04T11:02:09.208209+0000", "past_intervals": [ { "first": "0", "last": "0", "all_participants": [], "intervals": [] } ], "probing_osds": [ "0(3)", "36(5)", "74(2)", "100(0)", "101(4)", "150(1)" ], "down_osds_we_would_probe": [], "peering_blocked_by": [] }, { "name": "Started", "enter_time": "2024-01-04T11:02:09.208161+0000" } ], "agent_state": {} Why is the pg peering state so slow, it's affected by the network? My network lacp with two of 10Gbps NIC -- ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079(a)gmail.com Skype: tranphong079

2 1

Upgrading from 16.2.11?
by Jeremy Hansen 05 Jan '24

05 Jan '24

I’d like to upgrade from 16.2.11 to the latest version. Is it possible to do this in one jump or do I need to go from 16.2.11 -> 16.2.14 -> 17.1.0 -> 17.2.7 -> 18.1.0 -> 18.2.1? I’m using cephadm. Thanks -jeremy

2 1

Reef Dashboard Recovery Throughput empty
by Zoltán Beck 05 Jan '24

05 Jan '24

Hi All, we just upgraded to Reef, everything looks great, except the new Dashboard. The Recovery Throughput graph is empty, the recovery is ongoing for 18 hours and still no data. I tried to move the prometheus service to other node and redeployed couple of times, but still no data. Kind Regards Zoltan

2 1

How to increment osd_deep_scrub_interval
by Jorge JP 05 Jan '24

05 Jan '24

Hello! I want to increment the interval between deep scrub in all osd. I tryed this but not configured: ceph config set osd.* osd_deep_scrub_interval 1209600 I have 50 osd.. should config every osd? thanks for the support!

2 1

MacOS support for CephFS client
by Fabien Sirjean 04 Jan '24

04 Jan '24

Dear all, Could you please share recent experiences with support for cephfs access from MacOS clients ? I couldn't find any threads on this matter on the list since 2021 (Daniel Persson), where he stated that : > Mac Mini Intel Catalina - Connected and working fine. > Mac Mini M1 BigSur - Can't compile brew cask, no popups for extra > permissions in the GUI. Thanks a lot! Cheers, Fabien

1 0

Unable to find Refresh Interval Option in Ceph Dashboard (Ceph v18.2.1 "reef")- Seeking Assistance
by Alam Mohammad 04 Jan '24

04 Jan '24

Hello, We've been using Ceph for managing our storage infrastructure, and we recently upgraded to the latest version (Ceph v18.2.1 "reef"). However, We've noticed that the "refresh interval" option seems to be missing in the dashboard, and we are facing challenges with monitoring our cluster in real-time. In the earlier version of the Ceph dashboard, there was a useful "refresh interval" option that allowed us to customize the update frequency of the dashboard. This was particularly handy for monitoring changes and responding promptly. However, after the upgrade to Ceph v18.2.1 "reef", We can't seem to find this option anywhere in the dashboard. Additionally, we observed an automatic refresh occurring at every 25 seconds. Seeking guidance locating and tuning the refresh interval settings in the latest version of Ceph to potentially reduce this interval. We've explored the dashboard settings thoroughly and reviewed the release notes for Ceph v18.2.1 "reef", but we couldn't find any mention of the removal of the "refresh interval" option. Any guidance or insights would be greatly appreciated! Thanks, Mohammad Saif Ceph Enthusiast

3 2

Re: Ceph newbee questions
by Anthony D'Atri 03 Jan '24

03 Jan '24

>> >> You can do that for a PoC, but that's a bad idea for any production workload. You'd want at least three nodes with OSDs to use the default RF=3 replication. You can do RF=2, but at the peril of your mortal data. > > I'm not sure I agree - I think size=2, min_size=2 is no worse than > RAID1 for data security. size=2, min_size=2 *is* RAID1. Except that you become unavailable if a single drive is unavailable. >> That isn't even the main risk as I understand it. Of course a double > failure is going to be a problem with size=2, or traditional RAID1, > and I think anybody choosing this configuration accepts this risk. We see people often enough who don’t know that. I’ve seen double failures. ymmv. > As I understand it, the reason min_size=1 is a trap has nothing to do > with double failures per se. It’s one of the concerns. > > The issue is that Ceph OSDs are somewhat prone to flapping during > recovery (OOM, etc). So even if the disk is fine, an OSD can go down > for a short time. If you have size=2, min=1 configured, then when > this happens the PG will become degraded and will continue operating > on the other OSD, and the flapping OSD becomes stale. Then when it > comes back up it recovers. The problem is that if the other OSD has a > permanent failure (disk crash/etc) while the first OSD is flapping, > now you have no good OSDs, because when the flapping OSD comes back up > it is stale, and its PGs have no peer. Indeed, arguably that’s an overlapping failure. I’ve seen this too, and have a pg query to demonstrate it. > I suspect there are ways to re-activate it, though this will result in potential data > inconsistency since writes were allowed to the cluster and will then > get rolled back. Yep. > With only two OSDs I'm guessing that would be the > main impact (well, depending on journaling behavior/etc), but if you > have more OSDs than that then you could have situations where one file > is getting rolled back, and some other file isn't, and so on. But you’d have a voting majority. > > With min_size=2 you're fairly safe from flapping because there will > always be two replicas that have the most recent version of every PG, > and so you can still tolerate a permanent failure of one of them. Exactly. > > size=2, min=2 doesn't suffer this failure mode, because anytime there > is flapping the PG goes inactive and no writes can be made, so when > the other OSD comes back up there is nothing to recover. Of course > this results in IO blocks and downtime, which is obviously > undesirable, but it is likely a more recoverable state than > inconsistent writes. Agreed, the difference between availability and durability. Depends what’s important to you. > > Apologies if I've gotten any of that wrong, but my understanding is > that it is these sorts of failure modes that cause min_size=1 to be a > trap. This isn't the sort of thing that typically happens in a RAID1 > config, or at least that admins don't think about. It’s both.

2 4

CLT Meeting Minutes 2024-01-03
by David Orman 03 Jan '24

03 Jan '24

Happy 2024! Today's CLT meeting covered the following: 1. 2024 brings a focus on performance of Crimson (some information here: https://docs.ceph.com/en/reef/dev/crimson/crimson/ ) 1. Status is available here: https://github.com/ceph/ceph.io/pull/635 2. There will be a new Crimson performance weekly meeting that will be lead by Matan Breizman 1. This does not replace the existing performance weekly, and is focused on Crimson 2. An email will follow with more details about this meeting 2. Ceph Quarterly will be published on/around the 14th of January, 2024. 1. See https://ceph.io/en/community/cq/ for previous issues of CQ 3. A development freeze on Squid is tentatively scheduled for January 31, 2024 4. Upcoming releases 1. 16.2.15 is next (the last Pacific release) 1. Anticipated by the end of January 2. 17.2.8 will follow (Quincy) 3. 18.2.2 will follow this (Reef)

1 0

Ceph Docs: active releases outdated
by Eugen Block 03 Jan '24

03 Jan '24

Hello and a happy new year! I'm wondering if there are some structural changes or something regarding the release page [1]. It still doesn't contain version 18.2.1 (Reef) and the latest two Quincy releases (17.2.6, 17.2.7) are missing as well. And for Pacific it's even worse, the latest entry is for 16.2.11 although we're close to 16.2.5 being released. I wanted to check the changelog for 18.2.1 which can be found in the news blog [2]. I'd understand if the changelogs would be published only in one place (the blog?) but then it should be at least referenced in the docs. Thanks, Eugen [1] https://docs.ceph.com/en/reef/releases/ [2] https://ceph.io/en/news/blog/2023/v18-2-1-reef-released/

3 3

Radosgw replicated -> EC pool
by Jan Kasprzak 02 Jan '24

02 Jan '24

Hello, Ceph users, what is the best way how to change the storage layout of all buckets in radosgw? I have default.rgw.buckets.data pool as replicated, and I want to use an erasure-coded layout instead. One way is to use cache tiering as described here: https://cephnotes.ksperis.com/blog/2015/04/15/ceph-pool-migration/ Could this be done under the running radosgw? If I read this correctly, it should be done, because radosgw is just another RADOS client. Another possible approach would be to create a new erasure-coded pool, a new zone placement, and set it as default. But how can I migrate the existing data? If I understand it correctly, the default placement applies only to new buckets. Something like this: ceph osd erasure-code-profile set k5m2 k=5 m=2 ceph osd pool create default.rgw.buckets.ecdata erasure k5m2 ceph osd pool application enable default.rgw.buckets.ecdata radosgw radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id ecdata-placement radosgw-admin zone placement add --rgw-zone default --placement-id ecdata-placement --data-pool default.rgw.buckets.ecdata --index-pool default.rgw.buckets.index --data-extra-pool default.rgw.buckets.non-ec radosgw-admin zonegroup placement default --rgw-zonegroup default --placement-id ecdata-placement How to continue from this point? And a secondary question: what purpose does the data-extra-pool serve? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 1

ValueError: invalid literal for int() with base 10: '443 ssl_certificate=c
by Владимир Клеусов 02 Jan '24

02 Jan '24

Hi, When trying to log in to rgb via the dashboard, an error appears in the logs ValueError: invalid literal for int() with base 10: '443 ssl_certificate=config://rgw/cert/rgw.test' RGW with SSL If rgw is without ssl, everything works fine ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) Please tell me how to solve this?

2 2

ceph-iscsi on RL9
by duluxoz 02 Jan '24

02 Jan '24

Hi All, Just successfully(?) completed a "live" update of the first node of a Ceph Quincy cluster from RL8 to RL9. Everything "seems" to be working - EXCEPT the iSCSI Gateway on that box. During the update the ceph-iscsi package was removed (ie `ceph-iscsi-3.6-2.g97f5b02.el8.noarch.rpm` - this is the latest package available from the Ceph Repos). So, obviously, I reinstalled the package. However, `dnf` is throwing errors (unsurprisingly, as that package is an el8 package and this box is now running el9): that package requires python 3.6 and el9 runs with python 3.8 (I believe). So my question(s) is: Can I simply "downgrade" python to 3.6, or is there an el9-compatible version of `ceph-iscsi` somewhere, and/or is there some process I need to follow to get the iSCSI Gateway back up and running? Some further info: The next step in my "happy-happy-fun-time-holiday-ICT-maintenance" was to upgrade the current Ceph Cluster to use `cephadm` and to go from Ceph-Quincy to Ceph-Reef - is this my ultimate upgrade path to get the iSCSI G/W back? BTW the Ceph Cluster is used *only* to provide iSCSI LUNS to an oVirt (KVM) Cluster front-end. Because it is the holidays I can take the entire network down (ie shutdown all the VMs) to facilitate this update process, which also means that I can use some other way (ie a non-iSCSI way - I think) to connect the Ceph SAN Cluster to the oVirt VM-Hosting Cluster - if *this* is the solution (ie no iSCSI) does someone have any experience in running oVirt off of Ceph in a non-iSCSI way - and could you be so kind as to provide some pointers/documentation/help? And before anyone says it, let me: "I broke, now I own it" :-) Thanks in advance, and everyone have a Merry Christmas, Heavenly Hanukkah, Quality Kwanzaa, Really-good (upcoming) Ramadan, and/or a Happy Holidays. Cheers Dulux-Oz

2 2

What is the maximum number of Rados gateway objects in one cluster using the bucket index and in one bucket?
by steve jung 02 Jan '24

02 Jan '24

Hello. We are using Ceph storage to test whether we can run the service by uploading and saving more than 40 billion files. So I'd like to check the contents below. 1) Maximum number of Rados gateway objects that can be stored in one cluster using the bucket index 2) Maximum number of Rados gateway objects that can be stored in one bucket Although we have referred to the limitations on the number of Rados gateway objects mentioned in existing documents, it seems theoretically unlimited If you have operated the number of files at the level we think in actual services or products, we would appreciate it if you could share them. Below are related documents and related settings values. > Related documents - https://documentation.suse.com/ses/5.5/html/ses-all/cha-ceph-gw.html - https://www.ibm.com/docs/en/storage-ceph/6?topic=resharding-limitations-buc… - https://docs.ceph.com/en/latest/dev/radosgw/bucket_index/ > Related config - rgw_dynamic_resharding: true - rgw_max_objs_per_shard: 100000 - rgw_max_dynamic_shards : 65521

2 1

mds generates slow request: peer_request, how to deal with it?
by David Yang 02 Jan '24

02 Jan '24

I hope this message finds you well. I have a cephfs cluster with 3 active mds, and use 3-node samba to export through the kernel. Currently, there are 2 node mds experiencing slow requests. We have tried restarting the mds. After a few hours, the replay log status became active. But the slow request reappears. The slow request does not seem to come from the client, but from the request of the mds node. Looking forward to your prompt response. HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests mds.osd44(mds.0): 2 slow requests are blocked > 30 secs mds.osd43(mds.1): 2 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 2 MDSs behind on trimming mds.osd44(mds.0): Behind on trimming (18642/1024) max_segments: 1024, num_segments: 18642 mds.osd43(mds.1): Behind on trimming (976612/1024) max_segments: 1024, num_segments: 976612 mds.0 { "ops": [ { "description": "peer_request:mds.1:1", "initiated_at": "2023-12-31T11:19:38.679925+0800", "age": 4358.8009461359998, "duration": 4358.8009636369998, "type_data": { "flag_point": "dispatched", "reqid": "mds.1:1", "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2023-12-31T11:19:38.679925+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:38.679925+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:38.679925+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:38.679936+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:38.679940+0800", "event": "dispatched" } ] } }, { "description": "peer_request:mds.1:2", "initiated_at": "2023-12-31T11:19:38.679938+0800", "age": 4358.8009326969996, "duration": 4358.8009763549999, "type_data": { "flag_point": "dispatched", "reqid": "mds.1:2", "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2023-12-31T11:19:38.679938+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:38.679938+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:38.679938+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:38.679941+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:38.679991+0800", "event": "dispatched" } ] } } ], "complaint_time": 30, "num_blocked_ops": 2 } mds.1 { "ops": [ { "description": "internal op exportdir:mds.1:1", "initiated_at": "2023-12-31T11:19:34.416451+0800", "age": 4384.38814198, "duration": 4384.3881617610004, "type_data": { "flag_point": "failed to wrlock, waiting", "reqid": "mds.1:1", "op_type": "internal_op", "internal_op": 5377, "op_name": "exportdir", "events": [ { "time": "2023-12-31T11:19:34.416451+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "dispatched" }, { "time": "2023-12-31T11:19:38.679923+0800", "event": "requesting remote authpins" }, { "time": "2023-12-31T11:19:38.693981+0800", "event": "failed to wrlock, waiting" } ] } }, { "description": "internal op exportdir:mds.1:2", "initiated_at": "2023-12-31T11:19:34.416482+0800", "age": 4384.3881117999999, "duration": 4384.3881714600002, "type_data": { "flag_point": "failed to wrlock, waiting", "reqid": "mds.1:2", "op_type": "internal_op", "internal_op": 5377, "op_name": "exportdir", "events": [ { "time": "2023-12-31T11:19:34.416482+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "dispatched" }, { "time": "2023-12-31T11:19:38.679929+0800", "event": "requesting remote authpins" }, { "time": "2023-12-31T11:19:38.693995+0800", "event": "failed to wrlock, waiting" } ] } } ], "complaint_time": 30, "num_blocked_ops": 2 } I can't find any other solution other than restarting the mds service with slow requests. Currently, the backlog of mds logs in the metadata pool exceeds 4TB. Best regards,

2 2