OSD failed: still recovering
Hello, We had a drive (OSD) failed innour 5 node cluster three days ago (late afternoon of Mar 20). The PGs have sorted themselves out, but the cluster has has been recovering with backfill since then. Every time I run a 'ceph -s' it shows a little over 5% misplaced objects with several jobs of backfill_wait and some scrubbing. What is sort of weird is that if I run the 'ceph -s' as few times in a row, I can see the the percentage of misplaced objects go down a bit but then if I leave it for a while and run 'ceph -s' again, it is still just over 5% misplaced objects but has typically slightly increased. For example, it might be 5.364% when I check it, and then after checking it several times in a row it might go down to 5.276% but then if I check it again after a few hours, it might be something like 5.478% (so still in the 5% range but slightly increased from last check) The cluster is on 10Gbit, and I have increased the max_backfills to 4 while the recovery runs, but it just doesn't seem to be making much progress. I know the failed drive needs to be replaced, but I think it is recommended to wait until the cluster is finished recovering? Your thoughts/advice (as usual) are greatly appreciated. Sent from my mobile device. Please excuse brevity and ttpos.
Hint: the default max misplaced setting for the balancer module is 5% . This is a common question. I should see if there’s somewhere in the docs where this could be called out. Most likely it IS making progress. In you look at `ceph health detail` periodically the misplaced PGs should be different over time. The stddev reported by `ceph osd df` should decrease over time too. Unless you have PGs stuck in backfill_wait or backfill_toofull forever.
On Mar 23, 2025, at 9:58 PM, Alan Murrell <Alan@t-net.ca> wrote:
Hello,
We had a drive (OSD) failed innour 5 node cluster three days ago (late afternoon of Mar 20). The PGs have sorted themselves out, but the cluster has has been recovering with backfill since then. Every time I run a 'ceph -s' it shows a little over 5% misplaced objects with several jobs of backfill_wait and some scrubbing.
What is sort of weird is that if I run the 'ceph -s' as few times in a row, I can see the the percentage of misplaced objects go down a bit but then if I leave it for a while and run 'ceph -s' again, it is still just over 5% misplaced objects but has typically slightly increased.
For example, it might be 5.364% when I check it, and then after checking it several times in a row it might go down to 5.276% but then if I check it again after a few hours, it might be something like 5.478% (so still in the 5% range but slightly increased from last check)
The cluster is on 10Gbit, and I have increased the max_backfills to 4 while the recovery runs, but it just doesn't seem to be making much progress.
I know the failed drive needs to be replaced, but I think it is recommended to wait until the cluster is finished recovering?
Your thoughts/advice (as usual) are greatly appreciated.
Sent from my mobile device. Please excuse brevity and ttpos. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, Thanks for the response. OK, good to know about the 5% misplaced objects report 😊 I just checked 'ceph -s' and the misplaced objects is showing 1.948%, but I suspect I will see this up to 5% or so later on 😊 It does finally look like there is progress being made, as my "active+clean" is currently 292 (out of a total 321 PGs), whereas it was seeming to progress beyond 287 or so. This is the result: --- START --- root@cephnode01:/# ceph -s cluster: id: 474264fe-b00e-11ee-b586-ac1f6b0ff21a health: HEALTH_ERR 1 failed cephadm daemon(s) 622 scrub errors Possible data damage: 8 pgs inconsistent services: mon: 5 daemons, quorum cephnode01,cephnode03,cephnode02,cephnode04,cephnode05 (age 6w) mgr: cephnode01.kefvmh(active, since 6w), standbys: cephnode03.clxwlu osd: 40 osds: 39 up (since 3d), 39 in (since 3d); 19 remapped pgs tcmu-runner: 1 portal active (1 hosts) data: pools: 4 pools, 321 pgs objects: 9.70M objects, 37 TiB usage: 117 TiB used, 234 TiB / 350 TiB avail pgs: 566646/29086680 objects misplaced (1.948%) 292 active+clean 13 active+remapped+backfilling 7 active+clean+inconsistent 5 active+remapped+backfill_wait 2 active+clean+scrubbing 1 active+clean+scrubbing+deep 1 active+remapped+inconsistent+backfilling io: client: 30 MiB/s rd, 2.5 MiB/s wr, 187 op/s rd, 240 op/s wr recovery: 189 MiB/s, 48 objects/s --- END --- I think the main numbers I need to keep an eye on are the ones that are "backfilling"? The "scrub" ones are just normal scrubs that are going on? I am waiting for a full "active+clean" before dealing with the Scrub errors (which are a result of the failed OSD). Also, my "TiB used" keeps going up as well; is that because of my lost HDD (the HDDs are 16TB).
Hi Alan, Just to share our experience, we have a similar cluster that was in a very similar state after a disk failure: 6 nodes, 168 OSDs, 12 PGs inconsistent, little over 5% misplaced objects. It took a good while for it to sort itself out, but eventually it did it. Repairing the PGs with `ceph pg repair <pgid>` seemed to do nothing at first, but worked. Changing the mClock profile to high_recovery_ops had a visible impact in recovery speed, though client I/O performance was very poor during recovery (we didn't mind). YMMV. - Gustavo ________________________________ From: Alan Murrell Sent: Monday, March 24, 2025 8:12 AM To: 'ceph-users' Subject: [ceph-users] Re: OSD failed: still recovering Hello, Thanks for the response. OK, good to know about the 5% misplaced objects report 😊 I just checked 'ceph -s' and the misplaced objects is showing 1.948%, but I suspect I will see this up to 5% or so later on 😊 It does finally look like there is progress being made, as my "active+clean" is currently 292 (out of a total 321 PGs), whereas it was seeming to progress beyond 287 or so. This is the result: --- START --- root@cephnode01:/# ceph -s cluster: id: 474264fe-b00e-11ee-b586-ac1f6b0ff21a health: HEALTH_ERR 1 failed cephadm daemon(s) 622 scrub errors Possible data damage: 8 pgs inconsistent services: mon: 5 daemons, quorum cephnode01,cephnode03,cephnode02,cephnode04,cephnode05 (age 6w) mgr: cephnode01.kefvmh(active, since 6w), standbys: cephnode03.clxwlu osd: 40 osds: 39 up (since 3d), 39 in (since 3d); 19 remapped pgs tcmu-runner: 1 portal active (1 hosts) data: pools: 4 pools, 321 pgs objects: 9.70M objects, 37 TiB usage: 117 TiB used, 234 TiB / 350 TiB avail pgs: 566646/29086680 objects misplaced (1.948%) 292 active+clean 13 active+remapped+backfilling 7 active+clean+inconsistent 5 active+remapped+backfill_wait 2 active+clean+scrubbing 1 active+clean+scrubbing+deep 1 active+remapped+inconsistent+backfilling io: client: 30 MiB/s rd, 2.5 MiB/s wr, 187 op/s rd, 240 op/s wr recovery: 189 MiB/s, 48 objects/s --- END --- I think the main numbers I need to keep an eye on are the ones that are "backfilling"? The "scrub" ones are just normal scrubs that are going on? I am waiting for a full "active+clean" before dealing with the Scrub errors (which are a result of the failed OSD). Also, my "TiB used" keeps going up as well; is that because of my lost HDD (the HDDs are 16TB). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
OK, so just an update that the recovery did finally complete, and I am pretty sure that the "inconsistent" PGs were PGs that the failed OSD were part of. Running 'ceph pg repair' has them sorted out, along with the 600+ "scrub errors" I had. I was able to remove the OSD from the cluster, and am now just awaiting a replacement drive. My cluster is now showing healthy. Related question: the OSD had its DB/WAL on a partition on an SSD. Would I just "zap" the partition like I would a drive, so it is available to be used again when I replace the HDD, or is there another method for "reclaiming" that DB/WAL partition?
Hi Alan, ----- Le 25 Mar 25, à 16:47, Alan Murrell Alan@t-net.ca a écrit :
OK, so just an update that the recovery did finally complete, and I am pretty sure that the "inconsistent" PGs were PGs that the failed OSD were part of. Running 'ceph pg repair' has them sorted out, along with the 600+ "scrub errors" I had.
I was able to remove the OSD from the cluster, and am now just awaiting a replacement drive. My cluster is now showing healthy.
Related question: the OSD had its DB/WAL on a partition on an SSD. Would I just "zap" the partition like I would a drive, so it is available to be used again when I replace the HDD,
Yes. Spot the 'db device' path to zap for this specific OSD in the output of 'cephadm ceph-volume lvm list', then zap it with: cephadm ceph-volume lvm zap --destroy /dev/ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/osd-db-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Refresh devices info with 'ceph orch device ls --hostname=<hostname> --wide --refresh' and the orchestrator should bootstrap a new OSD right away. Regards, Frédéric.
or is there another method for "reclaiming" that DB/WAL partition? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
OK, good to know about the 5% misplaced objects report 😊
I just checked 'ceph -s' and the misplaced objects is showing 1.948%, but I suspect I will see this up to 5% or so later on 😊
If you see a place in the docs where it would help to note the balancer phenomenon and mistakenly thinking that progress wasn’t, well, progressing, please let me know and I’ll make it so.
It does finally look like there is progress being made, as my "active+clean" is currently 292 (out of a total 321 PGs), whereas it was seeming to progress beyond 287 or so. This is the result:
--- START --- root@cephnode01:/# ceph -s cluster: id: 474264fe-b00e-11ee-b586-ac1f6b0ff21a health: HEALTH_ERR 1 failed cephadm daemon(s) 622 scrub errors Possible data damage: 8 pgs inconsistent
Inconsistent isn’t good. Look for medium errors on the OSDs in those PGs, grown errors, etc. Try repairs, but if there are bad drives, the problem likely will come back.
pgs: 566646/29086680 objects misplaced (1.948%) 292 active+clean 13 active+remapped+backfilling 7 active+clean+inconsistent 5 active+remapped+backfill_wait 2 active+clean+scrubbing 1 active+clean+scrubbing+deep 1 active+remapped+inconsistent+backfilling
io: client: 30 MiB/s rd, 2.5 MiB/s wr, 187 op/s rd, 240 op/s wr recovery: 189 MiB/s, 48 objects/s --- END ---
I think the main numbers I need to keep an eye on are the ones that are "backfilling"? The "scrub" ones are just normal scrubs that are going on?
Yes, scrubs are normal. Shallow scrubs are lightweight so they typically are done daily, the default interval for seep scrubs is weekly. With HDDs, especially if EC, it is not unusual to extend the deep scrub interval so that the cluster can get through them without DoSing clients.
I am waiting for a full "active+clean" before dealing with the Scrub errors (which are a result of the failed OSD).
No reason to necessarily wait, though the above backfill shouldn’t take long.
Also, my "TiB used" keeps going up as well; is that because of my lost HDD (the HDDs are 16TB).
TiB used as reported by the dashboard or `ceph df`? Client data organic growth?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Alan Murrell
-
Anthony D'Atri
-
Anthony D'Atri
-
Frédéric Nass
-
Gustavo Garcia Rondina