cephadm auto disk preparation and OSD installation incomplete
Dear ceph community, We have trouble with new disks not being properly prepared resp. OSDs not being fully installed by cephadm. We just added one new node each with ~40 HDDs each to two of our ceph clusters. In one cluster all but 5 disks got installed automatically. In the other none got installed. We are on ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) on both clusters. (I haven't added new disks since the last upgrade if I recall correctly). This is our OSD service definition: ``` 0|0[root@ceph-3-10 ~]# ceph orch ls osd --export service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: unmanaged service_name: osd.unmanaged unmanaged: true spec: filter_logic: AND objectstore: bluestore ``` Usually, new disks are installed properly (as expected due to all-available-devices). This time, I can see that LVs were created (via `lsblk`, `lvs`, `cephadm ceph-volume lvm list`). And OSDs are entered to the crushmap. However, they are not assigned to a host yet, nor do they have a type or weight, e.g.: ``` 0|0[root@ceph-2-10 ~]# ceph osd tree | grep "0 osd" 518 0 osd.518 down 0 1.00000 519 0 osd.519 down 0 1.00000 520 0 osd.520 down 0 1.00000 521 0 osd.521 down 0 1.00000 522 0 osd.522 down 0 1.00000 ``` And there is also no OSD daemon created (no docker container). So, OSD creation is somehow stuck halfway. I thought of fully cleaning up the OSD/disks. Hopping cephadm might pick them up properly next time. Just zapping was not possible, e.g. `cephadm ceph-volume lvm zap --destroy /dev/sdab` results in these errors: ``` /usr/bin/docker: stderr stderr: wipefs: error: /dev/sdab: probing initialization failed: Device or resource busy /usr/bin/docker: stderr --> failed to wipefs device, will try again to workaround probable race condition ``` So, I cleaned up more manually with purging them from crush and "resetting" disk and LV with dd and dmsetup, resp.: ``` ceph osd purge 480 --force dd if=/dev/zero of=/dev/sdab bs=1M count=1 dmsetup remove ceph--e10e0f08--8705--441a--8caa--4590de22a611-osd--block--d464211c--f513--4513--86c1--c7ad63e6c142 ``` ceph-volume still reported the old volumes, but then zapping actually got rid of them (only cleaned out the left-over entries, I guess). Now, cephadm was able to get one OSD up, when I did this cleanup for only one disk. When I did it in bulk for the rest, they all got stuck again the same way. Looking into ceph-volume logs (here for osd.522 as representative): ``` 0|0[root@ceph-2-11 /var/log/ceph/55633ec3-6c0c-4a02-990c-0f87e0f7a01f]# ll *20240316 -rw-r--r-- 1 ceph ceph 613789 Mar 14 17:10 ceph-osd.522.log-20240316 -rw-r--r-- 1 root root 42473553 Mar 16 03:13 ceph-volume.log-20240316 ``` ceph-volume only reports keyring creation: ``` [2024-03-14 16:10:19,509][ceph_volume.util.prepare][INFO ] Creating keyring file for osd.522 [2024-03-14 16:10:19,510][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-522/keyring --create-keyring --name osd.522 --add-key AQBfIfNlinc7EBAAHeFicrjmLEjRPGSjuFuLiQ== ``` In the OSD logs I found a couple of these, but don't know if they are related: ``` 2024-03-14T16:10:54.706+0000 7fab26988540 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) ``` Has anyone seen this behaviour before? Or could tell me where I should look next to troubleshoot this (which logs)? Any help is appreciated. Best Wishes, Mathias
Hi, before getting into that the first thing I would do is to fail the mgr. There have been too many issues where failing over the mgr resolved many of them. If that doesn't help, the cephadm.log should show something useful (/var/log/ceph/cephadm.log on the OSD hosts, I'm still not too familiar with the whole 'ceph log last 200 debug cephadm' thing). I remember reports in earlier versions of ceph-volume (probably pre-cephadm) where not all OSDs were created if the host had many disks to deploy. But I can't find those threads right now. And it's strange that on the second cluster no OSD is created at all, but again, maybe fail the mgr first before looking deeper into it. Regards, Eugen Zitat von "Kuhring, Mathias" <mathias.kuhring@bih-charite.de>:
Dear ceph community,
We have trouble with new disks not being properly prepared resp. OSDs not being fully installed by cephadm. We just added one new node each with ~40 HDDs each to two of our ceph clusters. In one cluster all but 5 disks got installed automatically. In the other none got installed.
We are on ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) on both clusters. (I haven't added new disks since the last upgrade if I recall correctly).
This is our OSD service definition: ``` 0|0[root@ceph-3-10 ~]# ceph orch ls osd --export service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: unmanaged service_name: osd.unmanaged unmanaged: true spec: filter_logic: AND objectstore: bluestore ```
Usually, new disks are installed properly (as expected due to all-available-devices). This time, I can see that LVs were created (via `lsblk`, `lvs`, `cephadm ceph-volume lvm list`). And OSDs are entered to the crushmap. However, they are not assigned to a host yet, nor do they have a type or weight, e.g.: ``` 0|0[root@ceph-2-10 ~]# ceph osd tree | grep "0 osd" 518 0 osd.518 down 0 1.00000 519 0 osd.519 down 0 1.00000 520 0 osd.520 down 0 1.00000 521 0 osd.521 down 0 1.00000 522 0 osd.522 down 0 1.00000 ```
And there is also no OSD daemon created (no docker container). So, OSD creation is somehow stuck halfway.
I thought of fully cleaning up the OSD/disks. Hopping cephadm might pick them up properly next time. Just zapping was not possible, e.g. `cephadm ceph-volume lvm zap --destroy /dev/sdab` results in these errors: ``` /usr/bin/docker: stderr stderr: wipefs: error: /dev/sdab: probing initialization failed: Device or resource busy /usr/bin/docker: stderr --> failed to wipefs device, will try again to workaround probable race condition ```
So, I cleaned up more manually with purging them from crush and "resetting" disk and LV with dd and dmsetup, resp.: ``` ceph osd purge 480 --force dd if=/dev/zero of=/dev/sdab bs=1M count=1 dmsetup remove ceph--e10e0f08--8705--441a--8caa--4590de22a611-osd--block--d464211c--f513--4513--86c1--c7ad63e6c142 ```
ceph-volume still reported the old volumes, but then zapping actually got rid of them (only cleaned out the left-over entries, I guess).
Now, cephadm was able to get one OSD up, when I did this cleanup for only one disk. When I did it in bulk for the rest, they all got stuck again the same way.
Looking into ceph-volume logs (here for osd.522 as representative): ``` 0|0[root@ceph-2-11 /var/log/ceph/55633ec3-6c0c-4a02-990c-0f87e0f7a01f]# ll *20240316 -rw-r--r-- 1 ceph ceph 613789 Mar 14 17:10 ceph-osd.522.log-20240316 -rw-r--r-- 1 root root 42473553 Mar 16 03:13 ceph-volume.log-20240316 ```
ceph-volume only reports keyring creation: ``` [2024-03-14 16:10:19,509][ceph_volume.util.prepare][INFO ] Creating keyring file for osd.522 [2024-03-14 16:10:19,510][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-522/keyring --create-keyring --name osd.522 --add-key AQBfIfNlinc7EBAAHeFicrjmLEjRPGSjuFuLiQ== ```
In the OSD logs I found a couple of these, but don't know if they are related: ``` 2024-03-14T16:10:54.706+0000 7fab26988540 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) ```
Has anyone seen this behaviour before? Or could tell me where I should look next to troubleshoot this (which logs)? Any help is appreciated.
Best Wishes, Mathias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Eugen, Thank you for the quick reply. The 5 missing disks on the one host were completely installed after I fully cleaned them up as I described. So, seems a smaller number of disks can make it. Regarding the other host with 40 disks: Failing the MGR didn't have any effect. There are nor errors in `/var/log/ceph/cephadm.log`. But a bunch of repeating image listings like: cephadm --image quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae165222496e09 --timeout 895 ls But `ceph log last 200 debug cephadm` gave me a bunch of interesting errors (Excerpt below. Is there any preferred method to provide bigger logs?). So, there are some timeouts, which might play into the assumption that ceph-volume is a bit overwhelmed by the number of disks. Shy assumption, but maybe LV creation is taking way too long (is cephadm waiting for all of them in bulk?) and times out with the default 900 secs. However, LVs are created and cephadm will not consider them next round ("has a filesystem"). I'm testing this theory right now by bumping up the limit to 2 hours (and the restart with "fresh" disks again): ceph config set mgr mgr/cephadm/default_cephadm_command_timeout 7200 However, there are also mentions of the host being not reachable: "Unable to reach remote host ceph-3-11" But this seems to be limited to cephadm / ceph orch, so basically MGR but not the rest of the cluster (i.e. MONs, OSDs, etc. are communicating happily, as far as I can tell). During my fresh run, I do notice more hosts being apperently down: 0|0[root@ceph-3-10 ~]# ceph orch host ls | grep Offline ceph-3-7 172.16.62.38 rgw,osd,_admin Offline ceph-3-10 172.16.62.41 rgw,osd,_admin,prometheus Offline ceph-3-11 172.16.62.43 rgw,osd,_admin Offline osd-mirror-2 172.16.62.23 rgw,osd,_admin Offline osd-mirror-3 172.16.62.24 rgw,osd,_admin Offline But I wonder if this just a side effect of the MGR (cephadm/orch) being too busy/overwhelmed with e.g. deploying the new OSDs. I will update you once the next round is done or failed. Best Wishes, Mathias ceph log last 200 debug cephadm ... 2024-03-20T09:19:24.917834+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 82122 : cephadm [INF] Detected new or changed devices on ceph-3-11 2024-03-20T09:34:28.877718+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 83339 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore ''')): Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) ... raise TimeoutError() concurrent.futures._base.TimeoutError During handling of the above exception, another exception occurred: ... orchestrator._interface.OrchestratorError: Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) 2024-03-20T09:34:28.881472+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 83340 : cephadm [ERR] Task exception was never retrieved future: <Task finished coro=<OSDService.create_from_spec.<locals>.all_hosts() done, defined at /usr/share/ceph/mgr/cephadm/services/osd.py:72> exception=RuntimeError('cephadm exited with an error code: 1, stderr:Unable to reach remote host ceph-3-11. ',)> Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 75, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 64, in create_from_spec_one replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 96, in create_single_host code, '\n'.join(err))) RuntimeError: cephadm exited with an error code: 1, stderr:Unable to reach remote host ceph-3-11. ... ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/config/ceph.conf Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae165222496e09 -e NODE_NAME=ceph-3-11 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b:/var/run/ceph:z -v /var/log/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b:/var/log/ceph:z -v /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpfoibulv3:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpjq5uxhj1:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae165222496e09 lvm batch --no-auto /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd ... /usr/bin/docker: stderr raise RuntimeError("Device {} has a filesystem.".format(self.dev_path)) /usr/bin/docker: stderr RuntimeError: Device /dev/sdm has a filesystem. ... RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/config/ceph.conf ... -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Thursday, March 21, 2024 3:28 PM To: ceph-users@ceph.io Subject: [ext] [ceph-users] Re: cephadm auto disk preparation and OSD installation incomplete Hi, before getting into that the first thing I would do is to fail the mgr. There have been too many issues where failing over the mgr resolved many of them. If that doesn't help, the cephadm.log should show something useful (/var/log/ceph/cephadm.log on the OSD hosts, I'm still not too familiar with the whole 'ceph log last 200 debug cephadm' thing). I remember reports in earlier versions of ceph-volume (probably pre-cephadm) where not all OSDs were created if the host had many disks to deploy. But I can't find those threads right now. And it's strange that on the second cluster no OSD is created at all, but again, maybe fail the mgr first before looking deeper into it. Regards, Eugen Zitat von "Kuhring, Mathias" <mathias.kuhring@bih-charite.de>:
Dear ceph community,
We have trouble with new disks not being properly prepared resp. OSDs not being fully installed by cephadm. We just added one new node each with ~40 HDDs each to two of our ceph clusters. In one cluster all but 5 disks got installed automatically. In the other none got installed.
We are on ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) on both clusters. (I haven't added new disks since the last upgrade if I recall correctly).
This is our OSD service definition: ``` 0|0[root@ceph-3-10 ~]# ceph orch ls osd --export service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: unmanaged service_name: osd.unmanaged unmanaged: true spec: filter_logic: AND objectstore: bluestore ```
Usually, new disks are installed properly (as expected due to all-available-devices). This time, I can see that LVs were created (via `lsblk`, `lvs`, `cephadm ceph-volume lvm list`). And OSDs are entered to the crushmap. However, they are not assigned to a host yet, nor do they have a type or weight, e.g.: ``` 0|0[root@ceph-2-10 ~]# ceph osd tree | grep "0 osd" 518 0 osd.518 down 0 1.00000 519 0 osd.519 down 0 1.00000 520 0 osd.520 down 0 1.00000 521 0 osd.521 down 0 1.00000 522 0 osd.522 down 0 1.00000 ```
And there is also no OSD daemon created (no docker container). So, OSD creation is somehow stuck halfway.
I thought of fully cleaning up the OSD/disks. Hopping cephadm might pick them up properly next time. Just zapping was not possible, e.g. `cephadm ceph-volume lvm zap --destroy /dev/sdab` results in these errors: ``` /usr/bin/docker: stderr stderr: wipefs: error: /dev/sdab: probing initialization failed: Device or resource busy /usr/bin/docker: stderr --> failed to wipefs device, will try again to workaround probable race condition ```
So, I cleaned up more manually with purging them from crush and "resetting" disk and LV with dd and dmsetup, resp.: ``` ceph osd purge 480 --force dd if=/dev/zero of=/dev/sdab bs=1M count=1 dmsetup remove ceph--e10e0f08--8705--441a--8caa--4590de22a611-osd--block--d464211c--f 513--4513--86c1--c7ad63e6c142 ```
ceph-volume still reported the old volumes, but then zapping actually got rid of them (only cleaned out the left-over entries, I guess).
Now, cephadm was able to get one OSD up, when I did this cleanup for only one disk. When I did it in bulk for the rest, they all got stuck again the same way.
Looking into ceph-volume logs (here for osd.522 as representative): ``` 0|0[root@ceph-2-11 /var/log/ceph/55633ec3-6c0c-4a02-990c-0f87e0f7a01f]# ll *20240316 -rw-r--r-- 1 ceph ceph 613789 Mar 14 17:10 ceph-osd.522.log-20240316 -rw-r--r-- 1 root root 42473553 Mar 16 03:13 ceph-volume.log-20240316 ```
ceph-volume only reports keyring creation: ``` [2024-03-14 16:10:19,509][ceph_volume.util.prepare][INFO ] Creating keyring file for osd.522 [2024-03-14 16:10:19,510][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-522/keyring --create-keyring --name osd.522 --add-key AQBfIfNlinc7EBAAHeFicrjmLEjRPGSjuFuLiQ== ```
In the OSD logs I found a couple of these, but don't know if they are related: ``` 2024-03-14T16:10:54.706+0000 7fab26988540 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) ```
Has anyone seen this behaviour before? Or could tell me where I should look next to troubleshoot this (which logs)? Any help is appreciated.
Best Wishes, Mathias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm afraid the parameter mgr mgr/cephadm/default_cephadm_command_timeout is buggy. Once not on default anymore, MGR is preparing the parameter a bit (e.g. substracting 5 secs) And there making it float, but cephadm is not having it (not even if I try the default 900 myself): [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices osd.all-available-devices: cephadm exited with an error code: 2, stderr:usage: cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] [--sysctl-dir SYSCTL_DIR] [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] [--retry RETRY] [--env ENV] [--no-container-init] [--no-cgroups-split] {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan} ... cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b: error: argument --timeout: invalid int value: '895.0' This also let to a status panic spiral reporting plenty of host and services missing failing (I assume orch failing due to cephadm complaining about the parameter). I got it under control by removing the parameter again from the config (ceph config rm mgr mgr/cephadm/default_cephadm_command_timeout). And the restarting all MGRs manually (systemctl restart..., again since orch was kinda useless at this stage). Anyhow, is there any other way I can adapt this parameter? Or maybe look into speeding up LV creation (if this is the bootleneck)? Thanks a lot, Mathias -----Original Message----- From: Kuhring, Mathias <mathias.kuhring@bih-charite.de> Sent: Friday, March 22, 2024 5:38 PM To: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io Subject: [ceph-users] Re: [ext] Re: cephadm auto disk preparation and OSD installation incomplete Hey Eugen, Thank you for the quick reply. The 5 missing disks on the one host were completely installed after I fully cleaned them up as I described. So, seems a smaller number of disks can make it. Regarding the other host with 40 disks: Failing the MGR didn't have any effect. There are nor errors in `/var/log/ceph/cephadm.log`. But a bunch of repeating image listings like: cephadm --image quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae165222496e09 --timeout 895 ls But `ceph log last 200 debug cephadm` gave me a bunch of interesting errors (Excerpt below. Is there any preferred method to provide bigger logs?). So, there are some timeouts, which might play into the assumption that ceph-volume is a bit overwhelmed by the number of disks. Shy assumption, but maybe LV creation is taking way too long (is cephadm waiting for all of them in bulk?) and times out with the default 900 secs. However, LVs are created and cephadm will not consider them next round ("has a filesystem"). I'm testing this theory right now by bumping up the limit to 2 hours (and the restart with "fresh" disks again): ceph config set mgr mgr/cephadm/default_cephadm_command_timeout 7200 However, there are also mentions of the host being not reachable: "Unable to reach remote host ceph-3-11" But this seems to be limited to cephadm / ceph orch, so basically MGR but not the rest of the cluster (i.e. MONs, OSDs, etc. are communicating happily, as far as I can tell). During my fresh run, I do notice more hosts being apperently down: 0|0[root@ceph-3-10 ~]# ceph orch host ls | grep Offline ceph-3-7 172.16.62.38 rgw,osd,_admin Offline ceph-3-10 172.16.62.41 rgw,osd,_admin,prometheus Offline ceph-3-11 172.16.62.43 rgw,osd,_admin Offline osd-mirror-2 172.16.62.23 rgw,osd,_admin Offline osd-mirror-3 172.16.62.24 rgw,osd,_admin Offline But I wonder if this just a side effect of the MGR (cephadm/orch) being too busy/overwhelmed with e.g. deploying the new OSDs. I will update you once the next round is done or failed. Best Wishes, Mathias ceph log last 200 debug cephadm ... 2024-03-20T09:19:24.917834+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 82122 : cephadm [INF] Detected new or changed devices on ceph-3-11 2024-03-20T09:34:28.877718+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 83339 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore ''')): Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) ... raise TimeoutError() concurrent.futures._base.TimeoutError During handling of the above exception, another exception occurred: ... orchestrator._interface.OrchestratorError: Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) 2024-03-20T09:34:28.881472+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 83340 : cephadm [ERR] Task exception was never retrieved future: <Task finished coro=<OSDService.create_from_spec.<locals>.all_hosts() done, defined at /usr/share/ceph/mgr/cephadm/services/osd.py:72> exception=RuntimeError('cephadm exited with an error code: 1, stderr:Unable to reach remote host ceph-3-11. ',)> Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 75, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 64, in create_from_spec_one replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 96, in create_single_host code, '\n'.join(err))) RuntimeError: cephadm exited with an error code: 1, stderr:Unable to reach remote host ceph-3-11. ... ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/config/ceph.conf Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae165222496e09 -e NODE_NAME=ceph-3-11 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b:/var/run/ceph:z -v /var/log/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b:/var/log/ceph:z -v /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpfoibulv3:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpjq5uxhj1:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae16522249 6e09 lvm batch --no-auto /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd ... /usr/bin/docker: stderr raise RuntimeError("Device {} has a filesystem.".format(self.dev_path)) /usr/bin/docker: stderr RuntimeError: Device /dev/sdm has a filesystem. ... RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/config/ceph.conf ... -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Thursday, March 21, 2024 3:28 PM To: ceph-users@ceph.io Subject: [ext] [ceph-users] Re: cephadm auto disk preparation and OSD installation incomplete Hi, before getting into that the first thing I would do is to fail the mgr. There have been too many issues where failing over the mgr resolved many of them. If that doesn't help, the cephadm.log should show something useful (/var/log/ceph/cephadm.log on the OSD hosts, I'm still not too familiar with the whole 'ceph log last 200 debug cephadm' thing). I remember reports in earlier versions of ceph-volume (probably pre-cephadm) where not all OSDs were created if the host had many disks to deploy. But I can't find those threads right now. And it's strange that on the second cluster no OSD is created at all, but again, maybe fail the mgr first before looking deeper into it. Regards, Eugen Zitat von "Kuhring, Mathias" <mathias.kuhring@bih-charite.de>:
Dear ceph community,
We have trouble with new disks not being properly prepared resp. OSDs not being fully installed by cephadm. We just added one new node each with ~40 HDDs each to two of our ceph clusters. In one cluster all but 5 disks got installed automatically. In the other none got installed.
We are on ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) on both clusters. (I haven't added new disks since the last upgrade if I recall correctly).
This is our OSD service definition: ``` 0|0[root@ceph-3-10 ~]# ceph orch ls osd --export service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: unmanaged service_name: osd.unmanaged unmanaged: true spec: filter_logic: AND objectstore: bluestore ```
Usually, new disks are installed properly (as expected due to all-available-devices). This time, I can see that LVs were created (via `lsblk`, `lvs`, `cephadm ceph-volume lvm list`). And OSDs are entered to the crushmap. However, they are not assigned to a host yet, nor do they have a type or weight, e.g.: ``` 0|0[root@ceph-2-10 ~]# ceph osd tree | grep "0 osd" 518 0 osd.518 down 0 1.00000 519 0 osd.519 down 0 1.00000 520 0 osd.520 down 0 1.00000 521 0 osd.521 down 0 1.00000 522 0 osd.522 down 0 1.00000 ```
And there is also no OSD daemon created (no docker container). So, OSD creation is somehow stuck halfway.
I thought of fully cleaning up the OSD/disks. Hopping cephadm might pick them up properly next time. Just zapping was not possible, e.g. `cephadm ceph-volume lvm zap --destroy /dev/sdab` results in these errors: ``` /usr/bin/docker: stderr stderr: wipefs: error: /dev/sdab: probing initialization failed: Device or resource busy /usr/bin/docker: stderr --> failed to wipefs device, will try again to workaround probable race condition ```
So, I cleaned up more manually with purging them from crush and "resetting" disk and LV with dd and dmsetup, resp.: ``` ceph osd purge 480 --force dd if=/dev/zero of=/dev/sdab bs=1M count=1 dmsetup remove ceph--e10e0f08--8705--441a--8caa--4590de22a611-osd--block--d464211c--f 513--4513--86c1--c7ad63e6c142 ```
ceph-volume still reported the old volumes, but then zapping actually got rid of them (only cleaned out the left-over entries, I guess).
Now, cephadm was able to get one OSD up, when I did this cleanup for only one disk. When I did it in bulk for the rest, they all got stuck again the same way.
Looking into ceph-volume logs (here for osd.522 as representative): ``` 0|0[root@ceph-2-11 /var/log/ceph/55633ec3-6c0c-4a02-990c-0f87e0f7a01f]# ll *20240316 -rw-r--r-- 1 ceph ceph 613789 Mar 14 17:10 ceph-osd.522.log-20240316 -rw-r--r-- 1 root root 42473553 Mar 16 03:13 ceph-volume.log-20240316 ```
ceph-volume only reports keyring creation: ``` [2024-03-14 16:10:19,509][ceph_volume.util.prepare][INFO ] Creating keyring file for osd.522 [2024-03-14 16:10:19,510][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-522/keyring --create-keyring --name osd.522 --add-key AQBfIfNlinc7EBAAHeFicrjmLEjRPGSjuFuLiQ== ```
In the OSD logs I found a couple of these, but don't know if they are related: ``` 2024-03-14T16:10:54.706+0000 7fab26988540 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) ```
Has anyone seen this behaviour before? Or could tell me where I should look next to troubleshoot this (which logs)? Any help is appreciated.
Best Wishes, Mathias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi and sorry for the delay, I was on vacation last week. :-) I just read your responses. I have no idea how to modify the default timeout for cephadm, maybe Adam or someone else can comment on that. But everytime I've been watching cephadm (ceph-volume) create new OSDs they are not created in parallel but sequentially. That would probably explain why it takes that long and eventually runs into a timeout. I can't really confirm but it would make sense to me if this would be the reason for the "offline" hosts during the operation. Does it resolve after the process has finished?
(Excerpt below. Is there any preferred method to provide bigger logs?).
It was just an example, I would focus on one OSD host and inspect the cephadm.log. The command I mentioned collects logs from all hosts, and it sometimes can be misleading because of missing timestamps. Did you collect more infos in the meantime? Thanks, Eugen Zitat von "Kuhring, Mathias" <mathias.kuhring@bih-charite.de>:
I'm afraid the parameter mgr mgr/cephadm/default_cephadm_command_timeout is buggy. Once not on default anymore, MGR is preparing the parameter a bit (e.g. substracting 5 secs) And there making it float, but cephadm is not having it (not even if I try the default 900 myself):
[WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices osd.all-available-devices: cephadm exited with an error code: 2, stderr:usage: cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] [--sysctl-dir SYSCTL_DIR] [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] [--retry RETRY] [--env ENV] [--no-container-init] [--no-cgroups-split]
{version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan} ... cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b: error: argument --timeout: invalid int value: '895.0'
This also let to a status panic spiral reporting plenty of host and services missing failing (I assume orch failing due to cephadm complaining about the parameter). I got it under control by removing the parameter again from the config (ceph config rm mgr mgr/cephadm/default_cephadm_command_timeout). And the restarting all MGRs manually (systemctl restart..., again since orch was kinda useless at this stage).
Anyhow, is there any other way I can adapt this parameter? Or maybe look into speeding up LV creation (if this is the bootleneck)?
Thanks a lot, Mathias
-----Original Message----- From: Kuhring, Mathias <mathias.kuhring@bih-charite.de> Sent: Friday, March 22, 2024 5:38 PM To: Eugen Block <eblock@nde.ag>; ceph-users@ceph.io Subject: [ceph-users] Re: [ext] Re: cephadm auto disk preparation and OSD installation incomplete
Hey Eugen,
Thank you for the quick reply.
The 5 missing disks on the one host were completely installed after I fully cleaned them up as I described. So, seems a smaller number of disks can make it.
Regarding the other host with 40 disks: Failing the MGR didn't have any effect. There are nor errors in `/var/log/ceph/cephadm.log`. But a bunch of repeating image listings like: cephadm --image quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae165222496e09 --timeout 895 ls
But `ceph log last 200 debug cephadm` gave me a bunch of interesting errors (Excerpt below. Is there any preferred method to provide bigger logs?).
So, there are some timeouts, which might play into the assumption that ceph-volume is a bit overwhelmed by the number of disks. Shy assumption, but maybe LV creation is taking way too long (is cephadm waiting for all of them in bulk?) and times out with the default 900 secs. However, LVs are created and cephadm will not consider them next round ("has a filesystem").
I'm testing this theory right now by bumping up the limit to 2 hours (and the restart with "fresh" disks again): ceph config set mgr mgr/cephadm/default_cephadm_command_timeout 7200
However, there are also mentions of the host being not reachable: "Unable to reach remote host ceph-3-11" But this seems to be limited to cephadm / ceph orch, so basically MGR but not the rest of the cluster (i.e. MONs, OSDs, etc. are communicating happily, as far as I can tell).
During my fresh run, I do notice more hosts being apperently down: 0|0[root@ceph-3-10 ~]# ceph orch host ls | grep Offline ceph-3-7 172.16.62.38 rgw,osd,_admin Offline ceph-3-10 172.16.62.41 rgw,osd,_admin,prometheus Offline ceph-3-11 172.16.62.43 rgw,osd,_admin Offline osd-mirror-2 172.16.62.23 rgw,osd,_admin Offline osd-mirror-3 172.16.62.24 rgw,osd,_admin Offline
But I wonder if this just a side effect of the MGR (cephadm/orch) being too busy/overwhelmed with e.g. deploying the new OSDs.
I will update you once the next round is done or failed.
Best Wishes, Mathias
ceph log last 200 debug cephadm ... 2024-03-20T09:19:24.917834+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 82122 : cephadm [INF] Detected new or changed devices on ceph-3-11 2024-03-20T09:34:28.877718+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 83339 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore ''')): Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) ...
raise TimeoutError() concurrent.futures._base.TimeoutError
During handling of the above exception, another exception occurred:
... orchestrator._interface.OrchestratorError: Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) 2024-03-20T09:34:28.881472+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 83340 : cephadm [ERR] Task exception was never retrieved future: <Task finished coro=<OSDService.create_from_spec.<locals>.all_hosts() done, defined at /usr/share/ceph/mgr/cephadm/services/osd.py:72> exception=RuntimeError('cephadm exited with an error code: 1, stderr:Unable to reach remote host ceph-3-11. ',)> Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 75, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 64, in create_from_spec_one replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 96, in create_single_host code, '\n'.join(err))) RuntimeError: cephadm exited with an error code: 1, stderr:Unable to reach remote host ceph-3-11. ...
''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/config/ceph.conf Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae165222496e09 -e NODE_NAME=ceph-3-11 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b:/var/run/ceph:z -v /var/log/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b:/var/log/ceph:z -v /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpfoibulv3:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpjq5uxhj1:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae16522249 6e09 lvm batch --no-auto /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd ...
/usr/bin/docker: stderr raise RuntimeError("Device {} has a filesystem.".format(self.dev_path)) /usr/bin/docker: stderr RuntimeError: Device /dev/sdm has a filesystem. ...
RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/config/ceph.conf ...
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Thursday, March 21, 2024 3:28 PM To: ceph-users@ceph.io Subject: [ext] [ceph-users] Re: cephadm auto disk preparation and OSD installation incomplete
Hi,
before getting into that the first thing I would do is to fail the mgr. There have been too many issues where failing over the mgr resolved many of them. If that doesn't help, the cephadm.log should show something useful (/var/log/ceph/cephadm.log on the OSD hosts, I'm still not too familiar with the whole 'ceph log last 200 debug cephadm' thing). I remember reports in earlier versions of ceph-volume (probably pre-cephadm) where not all OSDs were created if the host had many disks to deploy. But I can't find those threads right now. And it's strange that on the second cluster no OSD is created at all, but again, maybe fail the mgr first before looking deeper into it.
Regards, Eugen
Zitat von "Kuhring, Mathias" <mathias.kuhring@bih-charite.de>:
Dear ceph community,
We have trouble with new disks not being properly prepared resp. OSDs not being fully installed by cephadm. We just added one new node each with ~40 HDDs each to two of our ceph clusters. In one cluster all but 5 disks got installed automatically. In the other none got installed.
We are on ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) on both clusters. (I haven't added new disks since the last upgrade if I recall correctly).
This is our OSD service definition: ``` 0|0[root@ceph-3-10 ~]# ceph orch ls osd --export service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: unmanaged service_name: osd.unmanaged unmanaged: true spec: filter_logic: AND objectstore: bluestore ```
Usually, new disks are installed properly (as expected due to all-available-devices). This time, I can see that LVs were created (via `lsblk`, `lvs`, `cephadm ceph-volume lvm list`). And OSDs are entered to the crushmap. However, they are not assigned to a host yet, nor do they have a type or weight, e.g.: ``` 0|0[root@ceph-2-10 ~]# ceph osd tree | grep "0 osd" 518 0 osd.518 down 0 1.00000 519 0 osd.519 down 0 1.00000 520 0 osd.520 down 0 1.00000 521 0 osd.521 down 0 1.00000 522 0 osd.522 down 0 1.00000 ```
And there is also no OSD daemon created (no docker container). So, OSD creation is somehow stuck halfway.
I thought of fully cleaning up the OSD/disks. Hopping cephadm might pick them up properly next time. Just zapping was not possible, e.g. `cephadm ceph-volume lvm zap --destroy /dev/sdab` results in these errors: ``` /usr/bin/docker: stderr stderr: wipefs: error: /dev/sdab: probing initialization failed: Device or resource busy /usr/bin/docker: stderr --> failed to wipefs device, will try again to workaround probable race condition ```
So, I cleaned up more manually with purging them from crush and "resetting" disk and LV with dd and dmsetup, resp.: ``` ceph osd purge 480 --force dd if=/dev/zero of=/dev/sdab bs=1M count=1 dmsetup remove ceph--e10e0f08--8705--441a--8caa--4590de22a611-osd--block--d464211c--f 513--4513--86c1--c7ad63e6c142 ```
ceph-volume still reported the old volumes, but then zapping actually got rid of them (only cleaned out the left-over entries, I guess).
Now, cephadm was able to get one OSD up, when I did this cleanup for only one disk. When I did it in bulk for the rest, they all got stuck again the same way.
Looking into ceph-volume logs (here for osd.522 as representative): ``` 0|0[root@ceph-2-11 /var/log/ceph/55633ec3-6c0c-4a02-990c-0f87e0f7a01f]# ll *20240316 -rw-r--r-- 1 ceph ceph 613789 Mar 14 17:10 ceph-osd.522.log-20240316 -rw-r--r-- 1 root root 42473553 Mar 16 03:13 ceph-volume.log-20240316 ```
ceph-volume only reports keyring creation: ``` [2024-03-14 16:10:19,509][ceph_volume.util.prepare][INFO ] Creating keyring file for osd.522 [2024-03-14 16:10:19,510][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-522/keyring --create-keyring --name osd.522 --add-key AQBfIfNlinc7EBAAHeFicrjmLEjRPGSjuFuLiQ== ```
In the OSD logs I found a couple of these, but don't know if they are related: ``` 2024-03-14T16:10:54.706+0000 7fab26988540 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) ```
Has anyone seen this behaviour before? Or could tell me where I should look next to troubleshoot this (which logs)? Any help is appreciated.
Best Wishes, Mathias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Kuhring, Mathias