Preventing device zapping while replacing faulty drive (Squid 19.2.2)
Hi List! We have Squid 19.2.2. It is cephadm/docker based deployment (recently upgraded from Pacific 16.2.15). We are using 8 SAS drives for Block and 2 SSD drives for DB on every OSD Host. Problem: One of the SAS Block drives failed on OSD Host and we need to replace it. When our Ceph cluster was running on Pacific, we usually performed drive replacement using this steps: 1) Edit -> MARK osd.xx as OUT [re-balancing starts, wait until it is completed] 2) Edit -> MARK osd.xx as DOWN 3) Edit -> DELETE osd.xx [put check on "Preserve OSD ID" and "Yes, I am sure"] 4) Edit -> DESTROY osd.xx 5) Edit -> PURGE osd.xx [re-balancing starts, wait until it is completed] 6) Set "noout" and "norebalance" flags. Put OSD Host in maintenance mode. Shutdown OSD Host. Replace failed drive. Start OSD Host. 7) Wipe old DB Logical Volume (LV) [dd if=/dev/zero of=/dev/ceph-xxx/osd-db-xxx bs=1M count=10 conv=fsync]. 8) Wipe new Block disk. Destroy old DB LV. Wait for automatic discovery and creation of new osd.xx instance. In Pacific 16.2.15, after execution of PURGE command, Ceph just removed old osd.xx instance from cluster without deletion/zapping of DB and Block LVs. Now in Squid 19.2.2 we see what Ceph behaves differently. Execution of step 3 (Edit -> DELETE osd.xx) automatically executes DESTROY and PURGE, and after that Ceph automatically performs zapping and deletion of DB LV and Block LV! And after that it's automatic discovery finds "clean" SAS disk + free space on SSD drive and happily forms new osd.xx instance form failed drive what we need to replace :) Questions: 1) What is correct procedure how-to replace failed Block drive in Ceph Squid 19.2.2? 2) Is it possible to disable Zapping? 3) Is it possible to temporary disable automatic discovery of new drives for OSD service? P.S. Here is our Pacement Specification for OSD service: [ceph: root@ceph-mon12 /]# ceph orch ls osd osd.dashboard-admin-1633624229976 --export service_type: osd service_id: dashboard-admin-1633624229976 service_name: osd.dashboard-admin-1633624229976 placement: host_pattern: '*' spec: data_devices: rotational: true db_devices: rotational: false db_slots: 4 filter_logic: AND objectstore: bluestore Logs from Ceph: 12/8/25 08:54 AM [INF] Cluster is now healthy 12/8/25 08:54 AM [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded) 12/8/25 08:54 AM [WRN] Health check update: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 12/8/25 08:54 AM [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 12/8/25 08:54 AM [WRN] Health check failed: Degraded data redundancy: 12/121226847 objects degraded (0.000%), 2 pgs degraded (PG_DEGRADED) 12/8/25 08:54 AM [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 12/8/25 08:54 AM [INF] osd.33 [v2:10.10.10.105:6824/297218036,v1:10.10.10.105:6825/297218036] boot 12/8/25 08:54 AM [WRN] OSD bench result of 1909.305644 IOPS is not within the threshold limit range of 50.000000 IOPS and 500.000000 IOPS for osd.33. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. 12/8/25 08:53 AM [INF] Deploying daemon osd.33 on ceph-osd15 12/8/25 08:53 AM [INF] Found osd claims for drivegroup dashboard-admin-1633624229976 -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Found osd claims -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Detected new or changed devices on ceph-osd15 12/8/25 08:52 AM [INF] Successfully zapped devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Zapping devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Successfully destroyed old osd.33 on ceph-osd15; ready for replacement 12/8/25 08:52 AM [INF] Successfully removed osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Removing key for osd.33 12/8/25 08:52 AM [INF] Removing daemon osd.33 from ceph-osd15 -- ports []
On 12.08.25 10:47, Dmitrijs Demidovs wrote:
1) What is correct procedure how-to replace failed Block drive in Ceph Squid 19.2.2? 2) Is it possible to disable Zapping?
On the CLI you can run "ceph orch osd rm NN" without "--zap".
3) Is it possible to temporary disable automatic discovery of new drives for OSD service?
ceph orch set-unmanaged osd.dashboard-admin-1633624229976 and afterwards ceph orch set-managed osd.dashboard-admin-1633624229976 Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Hi, I think I found the right place [0]: ---snip--- if any_replace_params: # mark destroyed in osdmap if not osd.destroy(): raise orchestrator.OrchestratorError( f"Could not destroy {osd}") logger.info( f"Successfully destroyed old {osd} on {osd.hostname}; ready for replacement") if any_replace_params: osd.zap = True ... if osd.zap: # throws an exception if the zap fails logger.info(f"Zapping devices for {osd} on {osd.hostname}") osd.do_zap() ---snip--- So if the replace flag is set, Ceph will zap the device(s). I compared the versions, the change was between 19.2.0 and 19.2.1. On the one hand, I agree with OP, if Ceph immediately zaps the drive(s) it will redeploy the destroyed OSD with the faulty disk. On the other hand, if you don't let cephadm zap the drives you'll need manual intervention during the actual disk replacement. So the OSD will be purged, leaving the DB/WAL LVs on the disk. It would be interesting to learn what led to the decision to implement it like this, but I also don't see an "optimal" way doing this. I wonder if it could make sense to zap only DB/WAL devices, not the data device in case of replacement. Then when the faulty disk gets replaced, the orchestrator could redeploy an OSD since the data drive should be clean and there should be space for DB/WAL. Regards, Eugen [0] https://github.com/ceph/ceph/blob/v19.2.3/src/pybind/mgr/cephadm/services/os... Zitat von Dmitrijs Demidovs <dmitrijs.demidovs@carminered.eu>:
Hi List!
We have Squid 19.2.2. It is cephadm/docker based deployment (recently upgraded from Pacific 16.2.15). We are using 8 SAS drives for Block and 2 SSD drives for DB on every OSD Host.
Problem:
One of the SAS Block drives failed on OSD Host and we need to replace it. When our Ceph cluster was running on Pacific, we usually performed drive replacement using this steps:
1) Edit -> MARK osd.xx as OUT [re-balancing starts, wait until it is completed] 2) Edit -> MARK osd.xx as DOWN 3) Edit -> DELETE osd.xx [put check on "Preserve OSD ID" and "Yes, I am sure"] 4) Edit -> DESTROY osd.xx 5) Edit -> PURGE osd.xx [re-balancing starts, wait until it is completed] 6) Set "noout" and "norebalance" flags. Put OSD Host in maintenance mode. Shutdown OSD Host. Replace failed drive. Start OSD Host. 7) Wipe old DB Logical Volume (LV) [dd if=/dev/zero of=/dev/ceph-xxx/osd-db-xxx bs=1M count=10 conv=fsync]. 8) Wipe new Block disk. Destroy old DB LV. Wait for automatic discovery and creation of new osd.xx instance.
In Pacific 16.2.15, after execution of PURGE command, Ceph just removed old osd.xx instance from cluster without deletion/zapping of DB and Block LVs.
Now in Squid 19.2.2 we see what Ceph behaves differently. Execution of step 3 (Edit -> DELETE osd.xx) automatically executes DESTROY and PURGE, and after that Ceph automatically performs zapping and deletion of DB LV and Block LV! And after that it's automatic discovery finds "clean" SAS disk + free space on SSD drive and happily forms new osd.xx instance form failed drive what we need to replace :)
Questions:
1) What is correct procedure how-to replace failed Block drive in Ceph Squid 19.2.2? 2) Is it possible to disable Zapping? 3) Is it possible to temporary disable automatic discovery of new drives for OSD service?
P.S.
Here is our Pacement Specification for OSD service:
[ceph: root@ceph-mon12 /]# ceph orch ls osd osd.dashboard-admin-1633624229976 --export service_type: osd service_id: dashboard-admin-1633624229976 service_name: osd.dashboard-admin-1633624229976 placement: host_pattern: '*' spec: data_devices: rotational: true db_devices: rotational: false db_slots: 4 filter_logic: AND objectstore: bluestore
Logs from Ceph:
12/8/25 08:54 AM [INF] Cluster is now healthy 12/8/25 08:54 AM [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded) 12/8/25 08:54 AM [WRN] Health check update: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 12/8/25 08:54 AM [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 12/8/25 08:54 AM [WRN] Health check failed: Degraded data redundancy: 12/121226847 objects degraded (0.000%), 2 pgs degraded (PG_DEGRADED) 12/8/25 08:54 AM [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 12/8/25 08:54 AM [INF] osd.33 [v2:10.10.10.105:6824/297218036,v1:10.10.10.105:6825/297218036] boot 12/8/25 08:54 AM [WRN] OSD bench result of 1909.305644 IOPS is not within the threshold limit range of 50.000000 IOPS and 500.000000 IOPS for osd.33. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. 12/8/25 08:53 AM [INF] Deploying daemon osd.33 on ceph-osd15 12/8/25 08:53 AM [INF] Found osd claims for drivegroup dashboard-admin-1633624229976 -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Found osd claims -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Detected new or changed devices on ceph-osd15 12/8/25 08:52 AM [INF] Successfully zapped devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Zapping devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Successfully destroyed old osd.33 on ceph-osd15; ready for replacement 12/8/25 08:52 AM [INF] Successfully removed osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Removing key for osd.33 12/8/25 08:52 AM [INF] Removing daemon osd.33 from ceph-osd15 -- ports []
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, One way to avoid the OSD from being recreated with the faulty drive as soon as it was zapped (without disabling the OSD service entirely) is to set the _no_schedule label on the host with 'ceph orch host label add <hostname> _no_schedule' and remove the label after the drive has been replaced. Best regards, Frédéric. Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> frederic.nass@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Le mer. 20 août 2025 à 10:42, Eugen Block <eblock@nde.ag> a écrit :
Hi,
I think I found the right place [0]:
---snip--- if any_replace_params: # mark destroyed in osdmap if not osd.destroy(): raise orchestrator.OrchestratorError( f"Could not destroy {osd}") logger.info( f"Successfully destroyed old {osd} on {osd.hostname}; ready for replacement") if any_replace_params: osd.zap = True ...
if osd.zap: # throws an exception if the zap fails logger.info(f"Zapping devices for {osd} on {osd.hostname}") osd.do_zap() ---snip---
So if the replace flag is set, Ceph will zap the device(s). I compared the versions, the change was between 19.2.0 and 19.2.1.
On the one hand, I agree with OP, if Ceph immediately zaps the drive(s) it will redeploy the destroyed OSD with the faulty disk. On the other hand, if you don't let cephadm zap the drives you'll need manual intervention during the actual disk replacement. So the OSD will be purged, leaving the DB/WAL LVs on the disk.
It would be interesting to learn what led to the decision to implement it like this, but I also don't see an "optimal" way doing this. I wonder if it could make sense to zap only DB/WAL devices, not the data device in case of replacement. Then when the faulty disk gets replaced, the orchestrator could redeploy an OSD since the data drive should be clean and there should be space for DB/WAL.
Regards, Eugen
[0]
https://github.com/ceph/ceph/blob/v19.2.3/src/pybind/mgr/cephadm/services/os...
Zitat von Dmitrijs Demidovs <dmitrijs.demidovs@carminered.eu>:
Hi List!
We have Squid 19.2.2. It is cephadm/docker based deployment (recently upgraded from Pacific 16.2.15). We are using 8 SAS drives for Block and 2 SSD drives for DB on every OSD Host.
Problem:
One of the SAS Block drives failed on OSD Host and we need to replace it. When our Ceph cluster was running on Pacific, we usually performed drive replacement using this steps:
1) Edit -> MARK osd.xx as OUT [re-balancing starts, wait until it is completed] 2) Edit -> MARK osd.xx as DOWN 3) Edit -> DELETE osd.xx [put check on "Preserve OSD ID" and "Yes, I am sure"] 4) Edit -> DESTROY osd.xx 5) Edit -> PURGE osd.xx [re-balancing starts, wait until it is completed] 6) Set "noout" and "norebalance" flags. Put OSD Host in maintenance mode. Shutdown OSD Host. Replace failed drive. Start OSD Host. 7) Wipe old DB Logical Volume (LV) [dd if=/dev/zero of=/dev/ceph-xxx/osd-db-xxx bs=1M count=10 conv=fsync]. 8) Wipe new Block disk. Destroy old DB LV. Wait for automatic discovery and creation of new osd.xx instance.
In Pacific 16.2.15, after execution of PURGE command, Ceph just removed old osd.xx instance from cluster without deletion/zapping of DB and Block LVs.
Now in Squid 19.2.2 we see what Ceph behaves differently. Execution of step 3 (Edit -> DELETE osd.xx) automatically executes DESTROY and PURGE, and after that Ceph automatically performs zapping and deletion of DB LV and Block LV! And after that it's automatic discovery finds "clean" SAS disk + free space on SSD drive and happily forms new osd.xx instance form failed drive what we need to replace :)
Questions:
1) What is correct procedure how-to replace failed Block drive in Ceph Squid 19.2.2? 2) Is it possible to disable Zapping? 3) Is it possible to temporary disable automatic discovery of new drives for OSD service?
P.S.
Here is our Pacement Specification for OSD service:
[ceph: root@ceph-mon12 /]# ceph orch ls osd osd.dashboard-admin-1633624229976 --export service_type: osd service_id: dashboard-admin-1633624229976 service_name: osd.dashboard-admin-1633624229976 placement: host_pattern: '*' spec: data_devices: rotational: true db_devices: rotational: false db_slots: 4 filter_logic: AND objectstore: bluestore
Logs from Ceph:
12/8/25 08:54 AM [INF] Cluster is now healthy 12/8/25 08:54 AM [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded) 12/8/25 08:54 AM [WRN] Health check update: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 12/8/25 08:54 AM [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 12/8/25 08:54 AM [WRN] Health check failed: Degraded data redundancy: 12/121226847 objects degraded (0.000%), 2 pgs degraded (PG_DEGRADED) 12/8/25 08:54 AM [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 12/8/25 08:54 AM [INF] osd.33 [v2:10.10.10.105:6824/297218036,v1:10.10.10.105:6825/297218036] boot 12/8/25 08:54 AM [WRN] OSD bench result of 1909.305644 IOPS is not within the threshold limit range of 50.000000 IOPS and 500.000000 IOPS for osd.33. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. 12/8/25 08:53 AM [INF] Deploying daemon osd.33 on ceph-osd15 12/8/25 08:53 AM [INF] Found osd claims for drivegroup dashboard-admin-1633624229976 -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Found osd claims -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Detected new or changed devices on ceph-osd15 12/8/25 08:52 AM [INF] Successfully zapped devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Zapping devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Successfully destroyed old osd.33 on ceph-osd15; ready for replacement 12/8/25 08:52 AM [INF] Successfully removed osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Removing key for osd.33 12/8/25 08:52 AM [INF] Removing daemon osd.33 from ceph-osd15 -- ports []
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Dmitrijs Demidovs
-
Eugen Block
-
Frédéric Nass
-
Robert Sander