Hi Eugen, thank you for this quick response. - Yes it is a cephadm cluster. - I had launched "cephadm shell" before the "ceph-volume lvm zap /dev/sda" documented in https://docs.ceph.com/en/squid/cephadm/services/osd/#replacing-an-osd but this was not working. - I saw that /dev/mapper/osvg-rootvol does not exists in the cephadm container, but exists on the host. But ceph-volume command is only available in cephadm container. - your suggestion to launch from the cluster node "cephadm ceph-volume lvm zap /dev/sda" works ! - I try the same approach (login root on the node) for: # cephadm ceph-volume lvm create --osd-id 2 --data /dev/sda but it fails with: Inferring fsid aa64f278-3ba8-11f0-b327-303ea701bc10 Not using image '4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51' as it's not in list of non-dangling images with ceph=True label Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=whitaker03-ceph -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10:/var/run/ceph:z -v /var/log/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10:/var/log/ceph:z -v /var/lib/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsuhjdojs:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm create --osd-id 2 --data /dev/sda /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.282+0000 7f3a7a2f8640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.282+0000 7f3a7a2f8640 -1 AuthRegistry(0x7f3a740669a0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.282+0000 7f3a7a2f8640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.282+0000 7f3a7a2f8640 -1 AuthRegistry(0x7f3a7406a450) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.283+0000 7f3a7a2f8640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.283+0000 7f3a7a2f8640 -1 AuthRegistry(0x7f3a7a2f6ff0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.285+0000 7f3a73fff640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.285+0000 7f3a737fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.285+0000 7f3a7886e640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] /usr/bin/podman: stderr stderr: 2026-09-07T10:22:55.285+0000 7f3a7a2f8640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication /usr/bin/podman: stderr stderr: [errno 13] RADOS permission denied (error connecting to the cluster) /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module> /usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 54, in __init__ /usr/bin/podman: stderr self.main(self.argv) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr return f(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 166, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/main.py", line 47, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/create.py", line 84, in main /usr/bin/podman: stderr self.create() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/create.py", line 31, in create /usr/bin/podman: stderr self.objectstore.safe_prepare() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py", line 118, in safe_prepare /usr/bin/podman: stderr self.prepare() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py", line 131, in prepare /usr/bin/podman: stderr self.pre_prepare() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py", line 39, in pre_prepare /usr/bin/podman: stderr self.osd_id = prepare_utils.create_id(self.osd_fsid, /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 142, in create_id /usr/bin/podman: stderr if osd_id_available(osd_id): /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 180, in osd_id_available /usr/bin/podman: stderr raise RuntimeError('Unable check if OSD id exists: %s' % osd_id) /usr/bin/podman: stderr RuntimeError: Unable check if OSD id exists: 2 Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/sbin/cephadm/__main__.py", line 5581, in <module> File "/usr/sbin/cephadm/__main__.py", line 5569, in main File "/usr/sbin/cephadm/__main__.py", line 409, in _infer_config File "/usr/sbin/cephadm/__main__.py", line 353, in _infer_fsid File "/usr/sbin/cephadm/__main__.py", line 437, in _infer_image File "/usr/sbin/cephadm/__main__.py", line 311, in _validate_fsid File "/usr/sbin/cephadm/__main__.py", line 3314, in command_ceph_volume File "/usr/sbin/cephadm/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=whitaker03-ceph -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10:/var/run/ceph:z -v /var/log/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10:/var/log/ceph:z -v /var/lib/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/aa64f278-3ba8-11f0-b327-303ea701bc10/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsuhjdojs:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm create --osd-id 2 --data /dev/sda [root@whitaker03-ceph ~]# cephadm ceph osd tree usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] [--sysctl-dir SYSCTL_DIR] [--unit-dir UNIT_DIR] [--verbose] [--log-dest {file,syslog}] [--timeout TIMEOUT] [--retry RETRY] [--env ENV] [--no-container-init] [--no-cgroups-split] {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,unit-install,logs,bootstrap,deploy,_orch,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan} ... cephadm: error: argument {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,unit-install,logs,bootstrap,deploy,_orch,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan}: invalid choice: 'ceph' (choose from 'version', 'pull', 'inspect-image', 'ls', 'list-networks', 'adopt', 'rm-daemon', 'rm-cluster', 'run', 'shell', 'enter', 'ceph-volume', 'zap-osds', 'unit', 'unit-install', 'logs', 'bootstrap', 'deploy', '_orch', 'check-host', 'prepare-host', 'add-repo', 'rm-repo', 'install', 'registry-login', 'gather-facts', 'host-maintenance', 'agent', 'disk-rescan') So I try with the orchestrator as suggested # cephadm shell [ceph: root@whitaker03-ceph /]# ceph orch osd rm 2 --replace Scheduled OSD(s) for removal. VG/LV for the OSDs won't be zapped (--zap wasn't passed). Run the `ceph-volume lvm zap` command with `--destroy` against the VG/LV if you want them to be destroyed. The dashboard do not show any error or warning (cluster health is now OK because osd.2 is removed I think) but osd2 remains in this state since nearly 1 hour. Should I unset 'ceph orch apply osd --all-available-devices --unmanaged=true' now to finish ? Patrick Le 07/09/2026 à 11:59, Eugen Block a écrit :
Is it a cephadm managed cluster? In that case don't use ceph-volume outside of containers. Instead you'd need to execute:
cephadm ceph-volume lvm zap /dev/sda
Or via orchestrator:
ceph orch device zap <host> <device_path>
And if it's orchestrated, you could just use orch as well to remove the OSD as described here:
https://docs.ceph.com/en/squid/cephadm/services/osd/#replacing-an-osd
If it's not cephadm managed, look at the ceph-volume.log to find out more about the root cause. But that error message reminds me of other user's attempts to use ceph-volume outside of containers.
Zitat von Patrick Begou <ceph-users@ceph.io>:
Better layout of the text:
I'm replacing the HDD of osd.2 in my squid cluster following the provided documentation: https://docs.ceph.com/en/squid/rados/operations/add-or-rm-osds/
The OSD was yet out and the daemon down.
The osd.2 is destroyed:
ceph osd destroy 2 --yes-i-really-mean-it
I've stopped the node and replaced the HDD. Ceph (graphic board) see the HDD as available. But I'm unable to go further with the ceph-volume commands:
# ceph-volume lvm zap /dev/sda
--> RuntimeError: /dev/mapper/osvg-rootvol not found.
# ceph-volume lvm prepare --osd-id 2 --data /dev/sda
--> RuntimeError: /dev/mapper/osvg-rootvol not found.
Something wrong in my cluster or a step is missing in the documentation ?
Le 07/09/2026 à 11:38, Patrick Begou a écrit :
Hi
I'm replacing the HDD of osd.2 in my squid cluster following the provided documentation: https://docs.ceph.com/en/squid/rados/operations/add-or-rm-osds/
The OSD was yet out and the daemon down.
The osd.2 is destroyed:
cephosddestroy2--yes-i-really-mean-it I've stopped the node and replaced the HDD. Ceph (graphic board) see the HDD as available. But I'm unable to go further with the ceph-volume commands: # ceph-volume lvm zap /dev/sda --> RuntimeError: /dev/mapper/osvg-rootvol not found. # ceph-volume lvm prepare --osd-id 2 --data /dev/sda --> RuntimeError: /dev/mapper/osvg-rootvol not found.
Something wrong in my cluster or a step is missing in the documentation ?
Thank
Patrick
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io