Hi, I would appreciate some guidance here I am trying to reconfigure OSD to put DB/WAL on NVME so I "out" and "ceph osd rm" them ( because they were automatically configured and I did not "set-unamage" them properly ) then "wipefs -af " them Now CEPH is seeing drives available for deploying OSD on but it fails doing it due to "device or resource busy" Attempting to destroy and zap it fails due to "device or resource busy" even after I "wipefs -af /dev/sdaa" it this is what it is on it ( output of lsblk) sdaa 18.2T disk └─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm lvs,vgs does not find any logical disks /volume cephadm ceph-volume inventory is seeing the disks as "available = True" cephadm ceph-volume lvm list is NOT finding any LVM on /dev/sdaa docker ps -a does not show any "left over" OSD docker There is a reference ( block link) to the /dev/ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f016 in /var/lib/cephFSID/removed/osd.NUMER_DATE How can I 'clean up' the disk ? Many thanks Steven cephadm ceph-volume lvm zap --destroy /dev/sdaa Inferring fsid 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 Not using image 'sha256:4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51' as it's not in list of non-dangling images with ceph=True label Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v /var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /tmp/ceph-tmp419ql4q7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm zap --destroy /dev/sdaa /usr/bin/docker: stderr --> Zapping: /dev/sdaa /usr/bin/docker: stderr --> Removing all BlueStore signature on /dev/sdaa if any... /usr/bin/docker: stderr Running command: /usr/bin/ceph-bluestore-tool zap-device --dev /dev/sdaa --yes-i-really-really-mean-it /usr/bin/docker: stderr stderr: error from zap: (16) Device or resource busy /usr/bin/docker: stderr 2025-09-02T17:41:02.385+0000 72def157f9c0 -1 bdev(0x57b37fe52000 /dev/sdaa) open open got: (16) Device or resource busy Some extra info cephadm ceph-volume inventory output Device Path Size Device nodes rotates available Model name /dev/nvme3n1 1.46 TB nvme3n1 False True MTFDKCC1T6TGQ-1BK1DABYY /dev/sdaa 18.19 TB sdaa True True ST20000NM007D-3D /dev/sdv 18.19 TB sdv True True ST20000NM007D-3D /dev/sdw 18.19 TB sdw True True ST20000NM007D-3D /dev/sdx 18.19 TB sdx True True ST20000NM007D-3D /dev/sdy 18.19 TB sdy True True ST20000NM007D-3D /dev/sdz 18.19 TB sdz True True ST20000NM007D-3D lsblk output NAME SIZE TYPE MOUNTPOINT sdaa 18.2T disk └─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm lvs output ( greped for lsblk /dev/sdaa and couldn't find it ) osd-block-29a5b1f5-d442-4756-a61d-e230832a3577 ceph-013dc6ae-90c7-4271-b308-895bbafe8840 -wi-ao---- <6.99t osd-block-5f7471f4-22ee-4cf9-9a16-2854a0c26496 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 -wi-ao---- <6.99t osd-block-ff6ca83d-80c5-4fff-a50d-399d94ba1b1b ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea -wi-ao---- 18.19t osd-block-fc022b7e-3d85-4435-998e-6513020bbdf3 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 -wi-ao---- 13.97t osd-block-347afb43-812c-4330-a795-060e76f33f89 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 -wi-ao---- <6.99t osd-block-c96901b7-8b7e-4a94-be0a-ab1316fd114e ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 -wi-ao---- <6.99t osd-block-5e069648-e0ec-4490-a3a9-e758d6e3e1d4 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 -wi-ao---- 13.97t osd-block-5d7dc85e-dcc8-4d87-b7ba-d61201dccd5e ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e -wi-ao---- <6.99t osd-block-d18a4abc-f723-4ac8-9d05-d02edd7e6018 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 -wi-ao---- <6.99t osd-block-33404537-6f45-4f10-af8a-8de491e8bcc9 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 -wi-ao---- 18.19t osd-block-153d1ed5-007c-439e-ad9e-c20d1880ff4d ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac -wi-ao---- 18.19t osd-block-15f14591-43c9-42ed-8410-d7f216019b55 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 -wi-ao---- 18.19t osd-block-44c310eb-2002-4817-8b2e-beacb0a70806 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 -wi-ao---- <6.99t osd-block-b55d7d9f-8e2a-4da5-b9c5-ace96975286d ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a -wi-ao---- 13.97t osd-block-a97d2620-d07a-4bd7-8f19-66801f5d613b ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 -wi-ao---- 18.19t osd-block-5c134f31-3a17-4fe9-989f-fb3a1dc2c0e9 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 -wi-ao---- <6.99t osd-block-89390a0d-6bc3-41a5-a79b-ebac922d540f ceph-d8a23938-be11-4cee-ba13-951996ca9328 -wi-ao---- <6.99t osd-block-cfabfe08-5e42-46ba-8412-1d264e9312d3 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 -wi-ao---- <6.99t osd-block-2e86866b-b537-4b29-86e7-490e627f82be ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e -wi-ao---- 18.19t osd-block-ab0b170e-91e0-41d4-a5ea-b5c4fef6c3d8 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 -wi-ao---- <6.99t osd-db-012f9187-1596-4db8-96ba-809a96bce8fd ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-124f39e2-f28d-47a8-acce-fb1364f7a638 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-17edf308-b36b-42be-8855-55833d35f983 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-55eb88b3-89a0-4931-a1be-d73e2aaf527f ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-5e03ba4b-5641-49c6-a477-1b37dffcd398 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-b22a1142-8dec-4035-96b1-7717c4587bcb ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-block-f9c81b89-a903-4a11-9c2a-9171b4a5760a ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 -wi-ao---- <6.99t VGS output ( ( greped for lsblk /dev/sdaa result and couldn't find it ) VG #PV #LV #SN Attr VSize VFree ceph-013dc6ae-90c7-4271-b308-895bbafe8840 1 1 0 wz--n- <6.99t 0 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 1 1 0 wz--n- <6.99t 0 ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea 1 1 0 wz--n- 18.19t 0 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 1 1 0 wz--n- 13.97t 0 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 1 1 0 wz--n- <6.99t 0 ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 1 1 0 wz--n- <6.99t 0 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 1 1 0 wz--n- 13.97t 0 ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e 1 1 0 wz--n- <6.99t 0 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 1 1 0 wz--n- <6.99t 0 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 1 1 0 wz--n- 18.19t 0 ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac 1 1 0 wz--n- 18.19t 0 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 1 1 0 wz--n- 18.19t 0 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 1 1 0 wz--n- <6.99t 0 ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a 1 1 0 wz--n- 13.97t 0 ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 1 1 0 wz--n- 18.19t 0 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 1 1 0 wz--n- <6.99t 0 ceph-d8a23938-be11-4cee-ba13-951996ca9328 1 1 0 wz--n- <6.99t 0 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 1 1 0 wz--n- <6.99t 0 ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e 1 1 0 wz--n- 18.19t 0 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 1 1 0 wz--n- <6.99t 0 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 1 6 0 wz--n- <1.46t 8.00m ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 1 1 0 wz--n- <6.99t 0