Hi, first, there's a command to manage and unmanage services: ceph orch set-managed <service> ceph orch set-unmanaged <service> So you don't need to apply them each time. The OSD replacement has different aspects, for example if you have a failing drive which the kernel loses access to, you won't be able to drain and/or zap anything. In that case some manual intervention would be required to purge the OSD from the crush tree. If the OSD is responsive and draining works, you can (or should) use the orchestrator as I already mentioned. Managing/Unmanaging an OSD service can be necessary if you need to zap the drive (with or without a separate DB/WAL device). Because if your service is managed and you zap the OSD it could be immediately redeployed, depending on the Ceph version even as a standalone OSD although your spec requires a separate DB device (a known bug). So here's an example from one of my test clusters: # I want to replace osd.7 host1:~ # ceph osd metadata 7 | grep osdspec "osdspec_affinity": "osd-hdd-ssd-mix", # The osd has a separate DB/WAL host1:~ # ceph device ls | grep osd.7 147249cb-9f8b-40c8-b host1:vdb osd.7 4a4167cb-8e42-4e1e-b host1:vdd osd.10 osd.7 # Set service to unmanaged host1:~ # ceph orch set-unmanaged osd.osd-hdd-ssd-mix # Destroy and zap the drives (incl. DB/WAL) host1:~ # ceph orch osd rm 7 --replace --zap After draining and zap have completed the drive can be replaced. # Manage the service host1:~ # ceph orch set-managed osd.osd-hdd-ssd-mix After some time the destroyed OSD will be recreated: 2026-09-08T09:15:01.280+0000 7f54bb9da640 0 [cephadm INFO cephadm.services.osd] Found osd claims for drivegroup osd-hdd-ssd-mix -> {'host1': ['7']} 2026-09-08T09:15:01.280+0000 7f54bb9da640 0 log_channel(cephadm) log [INF] : Found osd claims for drivegroup osd-hdd-ssd-mix -> {'host1': ['7']} 2026-09-08T09:15:17.676+0000 7f54b89d4640 0 [cephadm INFO cephadm.serve] Deploying daemon osd.7 on host1 2026-09-08T09:15:17.676+0000 7f54b89d4640 0 log_channel(cephadm) log [INF] : Deploying daemon osd.7 on host1 The OSD has been redeployed successfully. Hope this helps! Zitat von Patrick Begou <ceph-users@ceph.io>:
Situation is back to normal now.
I've run: ceph orch osd rm stop 2 ceph osd tree -> 2 hdd 12.73340 osd.2 destroyed 1.00000 1.00000 ceph orch ps --daemon_type osd -> no osd.2 was running ceph orch apply osd --all-available-devices --dry-run -> after some amount of time show:
################ OSDSPEC PREVIEWS ################
+---------+-----------------------+-----------------+----------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+-----------------------+-----------------+----------+----+-----+ |osd |all-available-devices |whitaker03-ceph |/dev/sda |- |- |
+---------+-----------------------+-----------------+----------+----+-----+
ceph orch apply osd --all-available-devices --unmanaged=false ceph orch ps --daemon_type osd -> after some amount of time it shows:
osd.2 whitaker03-ceph starting
Then:
osd.2 whitaker03-ceph running (39s)
and now datas are moved back to the new HDD of osd.2: ceph -s
cluster: id: xxxxxxxxxxxxxxxxxxxxx health: HEALTH_OK
services: mon: xxxxxxxxxxxxxx mgr: mds: xxxxxxxxxxxxxx osd: 20 osds: 20 up (since 25m), 20 in (since 25m); 89 remapped pgs
data: volumes: 2/2 healthy pools: 8 pools, 705 pgs objects: 17.65M objects, 37 TiB usage: 60 TiB used, 194 TiB / 255 TiB avail pgs: 6270145/120559591 objects misplaced (5.201%) 591 active+clean 88 active+remapped+backfill_wait 20 active+clean+scrubbing 5 active+clean+scrubbing+deep 1 active+remapped+backfilling
But I'm not really sure about what is the right procedure for a next HDD replacement....
All is running fine however. Thank you Eugen for your help.
Patrick
Le 07/09/2026 à 15:33, Eugen Block a écrit :
Try this:
ceph orch osd rm stop 2
This should clear the orch state and potentially redeploy the OSD if there's an applicable spec.
Zitat von Patrick Begou <ceph-users@ceph.io>:
Le 07/09/2026 à 14:44, Eugen Block a écrit :
# cephadm ceph-volume lvm create --osd-id 2 --data /dev/sda
This will not work with cephadm as expected and as you already found out.
It's not a good idea to use ceph-volume to create OSDs, let the orchestrator handle it for you. Either via osd spec or for unmanaged OSDs like this:
ceph orch daemon add osd <host>:/dev/sdX
Run the `ceph-volume lvm zap` command with `--destroy` against the VG/LV if you want them to be destroyed.
I created a tracker for this misleading message:
https://tracker.ceph.com/issues/80083
It should not suggest to run ceph-volume directly.
The dashboard do not show any error or warning (cluster health is now OK because osd.2 is removed I think) but osd2 remains in this state since nearly 1 hour. Should I unset 'ceph orch apply osd --all-available-devices --unmanaged=true' now to finish ?
Which state exactly? Has the OSD been drained successfully? What's the output of:
ceph orch osd rm status
If the OSD is empty but hasn't been zapped, it should be marked as "destroyed" in the osd tree, correct? In that case you can replace the drive now, cephadm should eventually recognize it as available and then deploy an OSD on it, at least if the osd spec is applicable.
Hi Eugen, this is the status:
Not using image '4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51' as it's not in list of non-dangling images with ceph=True label [ceph: root@whitaker03-ceph /]# ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 2 whitaker03-ceph done, waiting for purge 0 True False True
Ceph show this OSD as destroyed, the HDD has been replaced and zapped:
[ceph: root@whitaker03-ceph /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 254.66789 root default .... -5 50.93358 host whitaker03-ceph 2 hdd 12.73340 osd.2 destroyed 0 1.00000 5 hdd 12.73340 osd.5 up 1.00000 1.00000 12 hdd 12.73340 osd.12 up 1.00000 1.00000 15 hdd 12.73340 osd.15 up 1.00000 1.00000 ....
[ceph: root@whitaker03-ceph /]# ceph health HEALTH_OK
I do not know how to break this failing command "ceph orch osd rm 2 --replace" now.
Patrick
Zitat von Patrick Begou <ceph-users@ceph.io>:
Hi Eugen,
thank you for this quick response.
- Yes it is a cephadm cluster. - I had launched "cephadm shell" before the "ceph-volume lvm zap /dev/sda" documented in https://docs.ceph.com/en/squid/cephadm/services/osd/#replacing-an-osd but this was not working. - I saw that /dev/mapper/osvg-rootvol does not exists in the cephadm container, but exists on the host. But ceph-volume command is only available in cephadm container. - your suggestion to launch from the cluster node "cephadm ceph-volume lvm zap /dev/sda" works ! - I try the same approach (login root on the node) for:
# cephadm ceph-volume lvm create --osd-id 2 --data /dev/sda
but it fails with:
Inferring fsid aa64f278-3ba8-11f0-b327-303ea701bc10 Not using image '4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51' as it's not in list of non-dangling images with ceph=True label Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=whitaker03-ceph -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10:/var/run/ceph:z -v /var/log/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10:/var/log/ceph:z -v /var/lib/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsuhjdojs:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm create --osd-id 2 --data /dev/sda /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.282+0000 7f3a7a2f8640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.282+0000 7f3a7a2f8640 -1 AuthRegistry(0x7f3a740669a0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.282+0000 7f3a7a2f8640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.282+0000 7f3a7a2f8640 -1 AuthRegistry(0x7f3a7406a450) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.283+0000 7f3a7a2f8640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.283+0000 7f3a7a2f8640 -1 AuthRegistry(0x7f3a7a2f6ff0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.285+0000 7f3a73fff640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.285+0000 7f3a737fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.285+0000 7f3a7886e640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.285+0000 7f3a7a2f8640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication /usr/bin/podman: stderr stderr: [errno 13] RADOS permission denied (error connecting to the cluster) /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module> /usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 54, in __init__ /usr/bin/podman: stderr self.main(self.argv) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr return f(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 166, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/main.py", line 47, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/create.py", line 84, in main /usr/bin/podman: stderr self.create() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/create.py", line 31, in create /usr/bin/podman: stderr self.objectstore.safe_prepare() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py", line 118, in safe_prepare /usr/bin/podman: stderr self.prepare() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py", line 131, in prepare /usr/bin/podman: stderr self.pre_prepare() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py", line 39, in pre_prepare /usr/bin/podman: stderr self.osd_id = prepare_utils.create_id(self.osd_fsid, /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 142, in create_id /usr/bin/podman: stderr if osd_id_available(osd_id): /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 180, in osd_id_available /usr/bin/podman: stderr raise RuntimeError('Unable check if OSD id exists: %s' % osd_id) /usr/bin/podman: stderr RuntimeError: Unable check if OSD id exists: 2 Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/sbin/cephadm/__main__.py", line 5581, in <module> File "/usr/sbin/cephadm/__main__.py", line 5569, in main File "/usr/sbin/cephadm/__main__.py", line 409, in _infer_config File "/usr/sbin/cephadm/__main__.py", line 353, in _infer_fsid File "/usr/sbin/cephadm/__main__.py", line 437, in _infer_image File "/usr/sbin/cephadm/__main__.py", line 311, in _validate_fsid File "/usr/sbin/cephadm/__main__.py", line 3314, in command_ceph_volume File "/usr/sbin/cephadm/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=whitaker03-ceph -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10:/var/run/ceph:z -v /var/log/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10:/var/log/ceph:z -v /var/lib/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsuhjdojs:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm create --osd-id 2 --data /dev/sda [root@whitaker03-ceph ~]# cephadm ceph osd tree usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] [--sysctl-dir SYSCTL_DIR] [--unit-dir UNIT_DIR] [--verbose] [--log-dest {file,syslog}] [--timeout TIMEOUT] [--retry RETRY] [--env ENV] [--no-container-init] [--no-cgroups-split] {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,unit-install,logs,bootstrap,deploy,_orch,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan} ... cephadm: error: argument {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,unit-install,logs,bootstrap,deploy,_orch,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan}: invalid choice: 'ceph' (choose from 'version', 'pull', 'inspect-image', 'ls', 'list-networks', 'adopt', 'rm-daemon', 'rm-cluster', 'run', 'shell', 'enter', 'ceph-volume', 'zap-osds', 'unit', 'unit-install', 'logs', 'bootstrap', 'deploy', '_orch', 'check-host', 'prepare-host', 'add-repo', 'rm-repo', 'install', 'registry-login', 'gather-facts', 'host-maintenance', 'agent', 'disk-rescan')
So I try with the orchestrator as suggested
# cephadm shell [ceph: root@whitaker03-ceph /]# ceph orch osd rm 2 --replace Scheduled OSD(s) for removal. VG/LV for the OSDs won't be zapped (--zap wasn't passed). Run the `ceph-volume lvm zap` command with `--destroy` against the VG/LV if you want them to be destroyed.
The dashboard do not show any error or warning (cluster health is now OK because osd.2 is removed I think) but osd2 remains in this state since nearly 1 hour.
Should I unset 'ceph orch apply osd --all-available-devices --unmanaged=true' now to finish ?
Patrick
Le 07/09/2026 à 11:59, Eugen Block a écrit :
Is it a cephadm managed cluster? In that case don't use ceph-volume outside of containers. Instead you'd need to execute:
cephadm ceph-volume lvm zap /dev/sda
Or via orchestrator:
ceph orch device zap <host> <device_path>
And if it's orchestrated, you could just use orch as well to remove the OSD as described here:
https://docs.ceph.com/en/squid/cephadm/services/osd/#replacing-an-osd
If it's not cephadm managed, look at the ceph-volume.log to find out more about the root cause. But that error message reminds me of other user's attempts to use ceph-volume outside of containers.
Zitat von Patrick Begou <ceph-users@ceph.io>:
> Better layout of the text: > > > I'm replacing the HDD of osd.2 in my squid cluster following > the provided documentation: > https://docs.ceph.com/en/squid/rados/operations/add-or-rm-osds/ > > The OSD was yet out and the daemon down. > > The osd.2 is destroyed: > > ceph osd destroy 2 --yes-i-really-mean-it > > > I've stopped the node and replaced the HDD. Ceph (graphic > board) see the HDD as available. But I'm unable to go further > with the ceph-volume commands: > > # ceph-volume lvm zap /dev/sda > > --> RuntimeError: /dev/mapper/osvg-rootvol not found. > > # ceph-volume lvm prepare --osd-id 2 --data /dev/sda > > --> RuntimeError: /dev/mapper/osvg-rootvol not found. > > Something wrong in my cluster or a step is missing in the > documentation ? > > > > Le 07/09/2026 à 11:38, Patrick Begou a écrit : >> Hi >> >> I'm replacing the HDD of osd.2 in my squid cluster following >> the provided documentation: >> https://docs.ceph.com/en/squid/rados/operations/add-or-rm-osds/ >> >> The OSD was yet out and the daemon down. >> >> The osd.2 is destroyed: >> >> cephosddestroy2--yes-i-really-mean-it I've stopped the node >> and replaced the HDD. Ceph (graphic board) see the HDD as >> available. But I'm unable to go further with the ceph-volume >> commands: # ceph-volume lvm zap /dev/sda --> RuntimeError: >> /dev/mapper/osvg-rootvol not found. # ceph-volume lvm prepare >> --osd-id 2 --data /dev/sda --> RuntimeError: >> /dev/mapper/osvg-rootvol not found. >> >> Something wrong in my cluster or a step is missing in the >> documentation ? >> >> Thank >> >> Patrick >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> >> > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io