squid 19.2.2 - drive in use but ceph is seeing it as "available"
Hi, I would appreciate some guidance here I am trying to reconfigure OSD to put DB/WAL on NVME so I "out" and "ceph osd rm" them ( because they were automatically configured and I did not "set-unamage" them properly ) then "wipefs -af " them Now CEPH is seeing drives available for deploying OSD on but it fails doing it due to "device or resource busy" Attempting to destroy and zap it fails due to "device or resource busy" even after I "wipefs -af /dev/sdaa" it this is what it is on it ( output of lsblk) sdaa 18.2T disk └─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm lvs,vgs does not find any logical disks /volume cephadm ceph-volume inventory is seeing the disks as "available = True" cephadm ceph-volume lvm list is NOT finding any LVM on /dev/sdaa docker ps -a does not show any "left over" OSD docker There is a reference ( block link) to the /dev/ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f016 in /var/lib/cephFSID/removed/osd.NUMER_DATE How can I 'clean up' the disk ? Many thanks Steven cephadm ceph-volume lvm zap --destroy /dev/sdaa Inferring fsid 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 Not using image 'sha256:4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51' as it's not in list of non-dangling images with ceph=True label Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v /var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /tmp/ceph-tmp419ql4q7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm zap --destroy /dev/sdaa /usr/bin/docker: stderr --> Zapping: /dev/sdaa /usr/bin/docker: stderr --> Removing all BlueStore signature on /dev/sdaa if any... /usr/bin/docker: stderr Running command: /usr/bin/ceph-bluestore-tool zap-device --dev /dev/sdaa --yes-i-really-really-mean-it /usr/bin/docker: stderr stderr: error from zap: (16) Device or resource busy /usr/bin/docker: stderr 2025-09-02T17:41:02.385+0000 72def157f9c0 -1 bdev(0x57b37fe52000 /dev/sdaa) open open got: (16) Device or resource busy Some extra info cephadm ceph-volume inventory output Device Path Size Device nodes rotates available Model name /dev/nvme3n1 1.46 TB nvme3n1 False True MTFDKCC1T6TGQ-1BK1DABYY /dev/sdaa 18.19 TB sdaa True True ST20000NM007D-3D /dev/sdv 18.19 TB sdv True True ST20000NM007D-3D /dev/sdw 18.19 TB sdw True True ST20000NM007D-3D /dev/sdx 18.19 TB sdx True True ST20000NM007D-3D /dev/sdy 18.19 TB sdy True True ST20000NM007D-3D /dev/sdz 18.19 TB sdz True True ST20000NM007D-3D lsblk output NAME SIZE TYPE MOUNTPOINT sdaa 18.2T disk └─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm lvs output ( greped for lsblk /dev/sdaa and couldn't find it ) osd-block-29a5b1f5-d442-4756-a61d-e230832a3577 ceph-013dc6ae-90c7-4271-b308-895bbafe8840 -wi-ao---- <6.99t osd-block-5f7471f4-22ee-4cf9-9a16-2854a0c26496 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 -wi-ao---- <6.99t osd-block-ff6ca83d-80c5-4fff-a50d-399d94ba1b1b ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea -wi-ao---- 18.19t osd-block-fc022b7e-3d85-4435-998e-6513020bbdf3 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 -wi-ao---- 13.97t osd-block-347afb43-812c-4330-a795-060e76f33f89 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 -wi-ao---- <6.99t osd-block-c96901b7-8b7e-4a94-be0a-ab1316fd114e ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 -wi-ao---- <6.99t osd-block-5e069648-e0ec-4490-a3a9-e758d6e3e1d4 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 -wi-ao---- 13.97t osd-block-5d7dc85e-dcc8-4d87-b7ba-d61201dccd5e ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e -wi-ao---- <6.99t osd-block-d18a4abc-f723-4ac8-9d05-d02edd7e6018 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 -wi-ao---- <6.99t osd-block-33404537-6f45-4f10-af8a-8de491e8bcc9 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 -wi-ao---- 18.19t osd-block-153d1ed5-007c-439e-ad9e-c20d1880ff4d ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac -wi-ao---- 18.19t osd-block-15f14591-43c9-42ed-8410-d7f216019b55 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 -wi-ao---- 18.19t osd-block-44c310eb-2002-4817-8b2e-beacb0a70806 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 -wi-ao---- <6.99t osd-block-b55d7d9f-8e2a-4da5-b9c5-ace96975286d ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a -wi-ao---- 13.97t osd-block-a97d2620-d07a-4bd7-8f19-66801f5d613b ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 -wi-ao---- 18.19t osd-block-5c134f31-3a17-4fe9-989f-fb3a1dc2c0e9 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 -wi-ao---- <6.99t osd-block-89390a0d-6bc3-41a5-a79b-ebac922d540f ceph-d8a23938-be11-4cee-ba13-951996ca9328 -wi-ao---- <6.99t osd-block-cfabfe08-5e42-46ba-8412-1d264e9312d3 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 -wi-ao---- <6.99t osd-block-2e86866b-b537-4b29-86e7-490e627f82be ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e -wi-ao---- 18.19t osd-block-ab0b170e-91e0-41d4-a5ea-b5c4fef6c3d8 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 -wi-ao---- <6.99t osd-db-012f9187-1596-4db8-96ba-809a96bce8fd ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-124f39e2-f28d-47a8-acce-fb1364f7a638 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-17edf308-b36b-42be-8855-55833d35f983 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-55eb88b3-89a0-4931-a1be-d73e2aaf527f ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-5e03ba4b-5641-49c6-a477-1b37dffcd398 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-b22a1142-8dec-4035-96b1-7717c4587bcb ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-block-f9c81b89-a903-4a11-9c2a-9171b4a5760a ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 -wi-ao---- <6.99t VGS output ( ( greped for lsblk /dev/sdaa result and couldn't find it ) VG #PV #LV #SN Attr VSize VFree ceph-013dc6ae-90c7-4271-b308-895bbafe8840 1 1 0 wz--n- <6.99t 0 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 1 1 0 wz--n- <6.99t 0 ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea 1 1 0 wz--n- 18.19t 0 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 1 1 0 wz--n- 13.97t 0 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 1 1 0 wz--n- <6.99t 0 ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 1 1 0 wz--n- <6.99t 0 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 1 1 0 wz--n- 13.97t 0 ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e 1 1 0 wz--n- <6.99t 0 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 1 1 0 wz--n- <6.99t 0 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 1 1 0 wz--n- 18.19t 0 ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac 1 1 0 wz--n- 18.19t 0 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 1 1 0 wz--n- 18.19t 0 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 1 1 0 wz--n- <6.99t 0 ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a 1 1 0 wz--n- 13.97t 0 ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 1 1 0 wz--n- 18.19t 0 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 1 1 0 wz--n- <6.99t 0 ceph-d8a23938-be11-4cee-ba13-951996ca9328 1 1 0 wz--n- <6.99t 0 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 1 1 0 wz--n- <6.99t 0 ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e 1 1 0 wz--n- 18.19t 0 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 1 1 0 wz--n- <6.99t 0 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 1 6 0 wz--n- <1.46t 8.00m ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 1 1 0 wz--n- <6.99t 0
Hi, If you want to remove the ceph lv partition you can try : dmsetup remove ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 ________________________________ De : Steven Vacaroaia <stef97@gmail.com> Envoyé : mercredi 3 septembre 2025 15:11:48 À : ceph-users Objet : [ceph-users] squid 19.2.2 - drive in use but ceph is seeing it as "available" Hi, I would appreciate some guidance here I am trying to reconfigure OSD to put DB/WAL on NVME so I "out" and "ceph osd rm" them ( because they were automatically configured and I did not "set-unamage" them properly ) then "wipefs -af " them Now CEPH is seeing drives available for deploying OSD on but it fails doing it due to "device or resource busy" Attempting to destroy and zap it fails due to "device or resource busy" even after I "wipefs -af /dev/sdaa" it this is what it is on it ( output of lsblk) sdaa 18.2T disk └─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm lvs,vgs does not find any logical disks /volume cephadm ceph-volume inventory is seeing the disks as "available = True" cephadm ceph-volume lvm list is NOT finding any LVM on /dev/sdaa docker ps -a does not show any "left over" OSD docker There is a reference ( block link) to the /dev/ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f016 in /var/lib/cephFSID/removed/osd.NUMER_DATE How can I 'clean up' the disk ? Many thanks Steven cephadm ceph-volume lvm zap --destroy /dev/sdaa Inferring fsid 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 Not using image 'sha256:4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51' as it's not in list of non-dangling images with ceph=True label Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v /var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /tmp/ceph-tmp419ql4q7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm zap --destroy /dev/sdaa /usr/bin/docker: stderr --> Zapping: /dev/sdaa /usr/bin/docker: stderr --> Removing all BlueStore signature on /dev/sdaa if any... /usr/bin/docker: stderr Running command: /usr/bin/ceph-bluestore-tool zap-device --dev /dev/sdaa --yes-i-really-really-mean-it /usr/bin/docker: stderr stderr: error from zap: (16) Device or resource busy /usr/bin/docker: stderr 2025-09-02T17:41:02.385+0000 72def157f9c0 -1 bdev(0x57b37fe52000 /dev/sdaa) open open got: (16) Device or resource busy Some extra info cephadm ceph-volume inventory output Device Path Size Device nodes rotates available Model name /dev/nvme3n1 1.46 TB nvme3n1 False True MTFDKCC1T6TGQ-1BK1DABYY /dev/sdaa 18.19 TB sdaa True True ST20000NM007D-3D /dev/sdv 18.19 TB sdv True True ST20000NM007D-3D /dev/sdw 18.19 TB sdw True True ST20000NM007D-3D /dev/sdx 18.19 TB sdx True True ST20000NM007D-3D /dev/sdy 18.19 TB sdy True True ST20000NM007D-3D /dev/sdz 18.19 TB sdz True True ST20000NM007D-3D lsblk output NAME SIZE TYPE MOUNTPOINT sdaa 18.2T disk └─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm lvs output ( greped for lsblk /dev/sdaa and couldn't find it ) osd-block-29a5b1f5-d442-4756-a61d-e230832a3577 ceph-013dc6ae-90c7-4271-b308-895bbafe8840 -wi-ao---- <6.99t osd-block-5f7471f4-22ee-4cf9-9a16-2854a0c26496 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 -wi-ao---- <6.99t osd-block-ff6ca83d-80c5-4fff-a50d-399d94ba1b1b ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea -wi-ao---- 18.19t osd-block-fc022b7e-3d85-4435-998e-6513020bbdf3 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 -wi-ao---- 13.97t osd-block-347afb43-812c-4330-a795-060e76f33f89 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 -wi-ao---- <6.99t osd-block-c96901b7-8b7e-4a94-be0a-ab1316fd114e ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 -wi-ao---- <6.99t osd-block-5e069648-e0ec-4490-a3a9-e758d6e3e1d4 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 -wi-ao---- 13.97t osd-block-5d7dc85e-dcc8-4d87-b7ba-d61201dccd5e ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e -wi-ao---- <6.99t osd-block-d18a4abc-f723-4ac8-9d05-d02edd7e6018 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 -wi-ao---- <6.99t osd-block-33404537-6f45-4f10-af8a-8de491e8bcc9 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 -wi-ao---- 18.19t osd-block-153d1ed5-007c-439e-ad9e-c20d1880ff4d ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac -wi-ao---- 18.19t osd-block-15f14591-43c9-42ed-8410-d7f216019b55 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 -wi-ao---- 18.19t osd-block-44c310eb-2002-4817-8b2e-beacb0a70806 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 -wi-ao---- <6.99t osd-block-b55d7d9f-8e2a-4da5-b9c5-ace96975286d ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a -wi-ao---- 13.97t osd-block-a97d2620-d07a-4bd7-8f19-66801f5d613b ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 -wi-ao---- 18.19t osd-block-5c134f31-3a17-4fe9-989f-fb3a1dc2c0e9 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 -wi-ao---- <6.99t osd-block-89390a0d-6bc3-41a5-a79b-ebac922d540f ceph-d8a23938-be11-4cee-ba13-951996ca9328 -wi-ao---- <6.99t osd-block-cfabfe08-5e42-46ba-8412-1d264e9312d3 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 -wi-ao---- <6.99t osd-block-2e86866b-b537-4b29-86e7-490e627f82be ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e -wi-ao---- 18.19t osd-block-ab0b170e-91e0-41d4-a5ea-b5c4fef6c3d8 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 -wi-ao---- <6.99t osd-db-012f9187-1596-4db8-96ba-809a96bce8fd ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-124f39e2-f28d-47a8-acce-fb1364f7a638 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-17edf308-b36b-42be-8855-55833d35f983 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-55eb88b3-89a0-4931-a1be-d73e2aaf527f ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-5e03ba4b-5641-49c6-a477-1b37dffcd398 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-b22a1142-8dec-4035-96b1-7717c4587bcb ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-block-f9c81b89-a903-4a11-9c2a-9171b4a5760a ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 -wi-ao---- <6.99t VGS output ( ( greped for lsblk /dev/sdaa result and couldn't find it ) VG #PV #LV #SN Attr VSize VFree ceph-013dc6ae-90c7-4271-b308-895bbafe8840 1 1 0 wz--n- <6.99t 0 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 1 1 0 wz--n- <6.99t 0 ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea 1 1 0 wz--n- 18.19t 0 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 1 1 0 wz--n- 13.97t 0 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 1 1 0 wz--n- <6.99t 0 ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 1 1 0 wz--n- <6.99t 0 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 1 1 0 wz--n- 13.97t 0 ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e 1 1 0 wz--n- <6.99t 0 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 1 1 0 wz--n- <6.99t 0 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 1 1 0 wz--n- 18.19t 0 ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac 1 1 0 wz--n- 18.19t 0 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 1 1 0 wz--n- 18.19t 0 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 1 1 0 wz--n- <6.99t 0 ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a 1 1 0 wz--n- 13.97t 0 ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 1 1 0 wz--n- 18.19t 0 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 1 1 0 wz--n- <6.99t 0 ceph-d8a23938-be11-4cee-ba13-951996ca9328 1 1 0 wz--n- <6.99t 0 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 1 1 0 wz--n- <6.99t 0 ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e 1 1 0 wz--n- 18.19t 0 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 1 1 0 wz--n- <6.99t 0 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 1 6 0 wz--n- <1.46t 8.00m ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 1 1 0 wz--n- <6.99t 0 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks dmsetup remove solved the issue Steven On Wed, 3 Sept 2025 at 10:47, GLE, Vivien <Vivien.GLE@inist.fr> wrote:
Hi,
If you want to remove the ceph lv partition you can try :
dmsetup remove ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 ------------------------------ *De :* Steven Vacaroaia <stef97@gmail.com> *Envoyé :* mercredi 3 septembre 2025 15:11:48 *À :* ceph-users *Objet :* [ceph-users] squid 19.2.2 - drive in use but ceph is seeing it as "available"
Hi,
I would appreciate some guidance here
I am trying to reconfigure OSD to put DB/WAL on NVME so I "out" and "ceph osd rm" them ( because they were automatically configured and I did not "set-unamage" them properly ) then "wipefs -af " them
Now CEPH is seeing drives available for deploying OSD on but it fails doing it due to "device or resource busy"
Attempting to destroy and zap it fails due to "device or resource busy" even after I "wipefs -af /dev/sdaa" it
this is what it is on it ( output of lsblk)
sdaa 18.2T disk
└─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm
lvs,vgs does not find any logical disks /volume cephadm ceph-volume inventory is seeing the disks as "available = True" cephadm ceph-volume lvm list is NOT finding any LVM on /dev/sdaa docker ps -a does not show any "left over" OSD docker
There is a reference ( block link) to the
/dev/ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f016 in /var/lib/cephFSID/removed/osd.NUMER_DATE
How can I 'clean up' the disk ?
Many thanks Steven
cephadm ceph-volume lvm zap --destroy /dev/sdaa Inferring fsid 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 Not using image 'sha256:4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51' as it's not in list of non-dangling images with ceph=True label Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v /var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v
/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /tmp/ceph-tmp419ql4q7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm zap --destroy /dev/sdaa /usr/bin/docker: stderr --> Zapping: /dev/sdaa /usr/bin/docker: stderr --> Removing all BlueStore signature on /dev/sdaa if any... /usr/bin/docker: stderr Running command: /usr/bin/ceph-bluestore-tool zap-device --dev /dev/sdaa --yes-i-really-really-mean-it /usr/bin/docker: stderr stderr: error from zap: (16) Device or resource busy /usr/bin/docker: stderr 2025-09-02T17:41:02.385+0000 72def157f9c0 -1 bdev(0x57b37fe52000 /dev/sdaa) open open got: (16) Device or resource busy
Some extra info
cephadm ceph-volume inventory output
Device Path Size Device nodes rotates available Model name /dev/nvme3n1 1.46 TB nvme3n1 False True MTFDKCC1T6TGQ-1BK1DABYY /dev/sdaa 18.19 TB sdaa True True ST20000NM007D-3D /dev/sdv 18.19 TB sdv True True ST20000NM007D-3D /dev/sdw 18.19 TB sdw True True ST20000NM007D-3D /dev/sdx 18.19 TB sdx True True ST20000NM007D-3D /dev/sdy 18.19 TB sdy True True ST20000NM007D-3D /dev/sdz 18.19 TB sdz True True ST20000NM007D-3D
lsblk output NAME SIZE TYPE MOUNTPOINT sdaa 18.2T disk
└─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm
lvs output ( greped for lsblk /dev/sdaa and couldn't find it ) osd-block-29a5b1f5-d442-4756-a61d-e230832a3577 ceph-013dc6ae-90c7-4271-b308-895bbafe8840 -wi-ao---- <6.99t
osd-block-5f7471f4-22ee-4cf9-9a16-2854a0c26496 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 -wi-ao---- <6.99t
osd-block-ff6ca83d-80c5-4fff-a50d-399d94ba1b1b ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea -wi-ao---- 18.19t
osd-block-fc022b7e-3d85-4435-998e-6513020bbdf3 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 -wi-ao---- 13.97t
osd-block-347afb43-812c-4330-a795-060e76f33f89 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 -wi-ao---- <6.99t
osd-block-c96901b7-8b7e-4a94-be0a-ab1316fd114e ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 -wi-ao---- <6.99t
osd-block-5e069648-e0ec-4490-a3a9-e758d6e3e1d4 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 -wi-ao---- 13.97t
osd-block-5d7dc85e-dcc8-4d87-b7ba-d61201dccd5e ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e -wi-ao---- <6.99t
osd-block-d18a4abc-f723-4ac8-9d05-d02edd7e6018 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 -wi-ao---- <6.99t
osd-block-33404537-6f45-4f10-af8a-8de491e8bcc9 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 -wi-ao---- 18.19t
osd-block-153d1ed5-007c-439e-ad9e-c20d1880ff4d ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac -wi-ao---- 18.19t
osd-block-15f14591-43c9-42ed-8410-d7f216019b55 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 -wi-ao---- 18.19t
osd-block-44c310eb-2002-4817-8b2e-beacb0a70806 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 -wi-ao---- <6.99t
osd-block-b55d7d9f-8e2a-4da5-b9c5-ace96975286d ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a -wi-ao---- 13.97t
osd-block-a97d2620-d07a-4bd7-8f19-66801f5d613b ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 -wi-ao---- 18.19t
osd-block-5c134f31-3a17-4fe9-989f-fb3a1dc2c0e9 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 -wi-ao---- <6.99t
osd-block-89390a0d-6bc3-41a5-a79b-ebac922d540f ceph-d8a23938-be11-4cee-ba13-951996ca9328 -wi-ao---- <6.99t
osd-block-cfabfe08-5e42-46ba-8412-1d264e9312d3 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 -wi-ao---- <6.99t
osd-block-2e86866b-b537-4b29-86e7-490e627f82be ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e -wi-ao---- 18.19t
osd-block-ab0b170e-91e0-41d4-a5ea-b5c4fef6c3d8 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 -wi-ao---- <6.99t
osd-db-012f9187-1596-4db8-96ba-809a96bce8fd ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-124f39e2-f28d-47a8-acce-fb1364f7a638 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-17edf308-b36b-42be-8855-55833d35f983 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-55eb88b3-89a0-4931-a1be-d73e2aaf527f ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-5e03ba4b-5641-49c6-a477-1b37dffcd398 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-b22a1142-8dec-4035-96b1-7717c4587bcb ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-block-f9c81b89-a903-4a11-9c2a-9171b4a5760a ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 -wi-ao---- <6.99t
VGS output ( ( greped for lsblk /dev/sdaa result and couldn't find it )
VG #PV #LV #SN Attr VSize VFree ceph-013dc6ae-90c7-4271-b308-895bbafe8840 1 1 0 wz--n- <6.99t 0 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 1 1 0 wz--n- <6.99t 0 ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea 1 1 0 wz--n- 18.19t 0 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 1 1 0 wz--n- 13.97t 0 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 1 1 0 wz--n- <6.99t 0 ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 1 1 0 wz--n- <6.99t 0 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 1 1 0 wz--n- 13.97t 0 ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e 1 1 0 wz--n- <6.99t 0 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 1 1 0 wz--n- <6.99t 0 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 1 1 0 wz--n- 18.19t 0 ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac 1 1 0 wz--n- 18.19t 0 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 1 1 0 wz--n- 18.19t 0 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 1 1 0 wz--n- <6.99t 0 ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a 1 1 0 wz--n- 13.97t 0 ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 1 1 0 wz--n- 18.19t 0 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 1 1 0 wz--n- <6.99t 0 ceph-d8a23938-be11-4cee-ba13-951996ca9328 1 1 0 wz--n- <6.99t 0 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 1 1 0 wz--n- <6.99t 0 ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e 1 1 0 wz--n- 18.19t 0 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 1 1 0 wz--n- <6.99t 0 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 1 6 0 wz--n- <1.46t 8.00m ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 1 1 0 wz--n- <6.99t 0 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
GLE, Vivien
-
Steven Vacaroaia