Why does recovering objects take much longer than the outage that caused them?
I noticed that for my clusters, even a short 5-minute network outage or single-host reboot can cause pgs: 5586988/366684639 objects misplaced (1.524%) which at the speed of recovery: 2.2 GiB/s, 676 objects/s can take hours to recover. I don't understand how this can be. If it's down for so short, how can rebalancing can take this long? I'm using Ceph 19.2.2 on HDDs with SSDs as BlueStore "db" device. Is this perhaps that writes of new files are written linearly to HDD (fast) but recovery seeks around on my HDDs in random order (slow)? In any case, this asymmetry is quite annoying. Could anything be done against it? Thanks! Niklas
If you have misplaced objects, the OSDs were marked out and ceph started to move the PGs to other nodes. This usually happens after 5 minutes of OSD downtime If you do "ceph osd set noout" and restart a node it should backfill much faster, because it just needs to catch up with the changes. Mit freundlichen Grüßen - Boris Behrens
Am 19.09.2025 um 13:25 schrieb Niklas Hambüchen <mail@nh2.me>:
I noticed that for my clusters, even a short 5-minute network outage or single-host reboot can cause
pgs: 5586988/366684639 objects misplaced (1.524%)
which at the speed of
recovery: 2.2 GiB/s, 676 objects/s
can take hours to recover.
I don't understand how this can be. If it's down for so short, how can rebalancing can take this long?
I'm using Ceph 19.2.2 on HDDs with SSDs as BlueStore "db" device. Is this perhaps that writes of new files are written linearly to HDD (fast) but recovery seeks around on my HDDs in random order (slow)?
In any case, this asymmetry is quite annoying. Could anything be done against it?
Thanks! Niklas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Boris,
If you have misplaced objects, the OSDs were marked out and ceph started to move the PGs to other node Does this really affect the question though? Even if Ceph started to move objects to other nodes for 5 minutes, why would undoing *that* after the 5 minutes take 3 hours?
Shouldn't moving stuff back into a node take roughly as long as moving it out? Separate: As far as I can tell, my nodes were down too short to be marked as `out`.
Den fre 19 sep. 2025 kl 14:11 skrev Niklas Hambüchen <mail@nh2.me>:
Hi Boris,
If you have misplaced objects, the OSDs were marked out and ceph started to move the PGs to other node Does this really affect the question though? Even if Ceph started to move objects to other nodes for 5 minutes, why would undoing *that* after the 5 minutes take 3 hours?
Shouldn't moving stuff back into a node take roughly as long as moving it out?
Separate: As far as I can tell, my nodes were down too short to be marked as `out`.
Well, if they were away long enough to get "out", then it is somewhat reasonable even for ~5m downtimes. In that case, I think the scenario is like this, you have an OSD.1 with x PGs on it, lets say one of them is PG 3.34a. It normally lives on this OSD.1 and also on OSD.12 and OSD.23. Then OSD.1 is gone so long the cluster starts "repairing" the hole it left, so it grabs "next OSD to hold 3.34a" which is OSD.34, so .34 starts creating PGan empty 3.34a and tries to fill it with data for a few minutes, while OSD.12 takes on writes which goes to OSD.23 aswell, and would end up on OSD.34 too, except its busy backfilling from scratch. This means the PG 3.34a version or date or whatever keeps track of history moves forward a lot while OSD.1 is gone. Then 5m later, OSD.1 comes back, but it has version-number too-old so neither OSD.12 or OSD.23 has enough history to just replay what happened last minutes (for recovery), in which case the temp-copy on OSD.34 gets erased and OSD.1 starts a full backfill from OSD.12 or OSD.23 in order to become fully updated copy of PG 3.34a. And this backfill is the one that takes hours, not just "the small diff of what occured in 5m". -- May the most significant bit of your life be positive.
Well, if they were away long enough to get "out", then it is somewhat reasonable even for ~5m downtimes.
Right, but what I'm saying is that this is not what happens. My reboot or disconnect takes < 5 minutes, and no OSD is `out` afterwards. When I say "down for 5 minutes", I literally mean that the node goes down, comes back up, and I'm sitting in front of its terminal and observe that all OSDs are `up` and `in`. Of course your explanation of what happens if it's `out` makes sense, but that isn't my scenario; if Ceph had 10 hours to move data off, of course I would have to expect at least 10 hours to move data back on. But it only has 5 minutes at max to move data off.
In this case it's not so much moving data off as those PGs catching up with writes that came in duringthat interval. That said, 1.5% and taking 10 hours seems like a lot. Something's awry. `ceph df` `ceph osd df` `ceph osd df tree` `ceph osd dump | grep pool` `ceph config dump`
On Sep 22, 2025, at 8:42 AM, Niklas Hambüchen <mail@nh2.me> wrote:
Well, if they were away long enough to get "out", then it is somewhat reasonable even for ~5m downtimes.
Right, but what I'm saying is that this is not what happens.
My reboot or disconnect takes < 5 minutes, and no OSD is `out` afterwards.
When I say "down for 5 minutes", I literally mean that the node goes down, comes back up, and I'm sitting in front of its terminal and observe that all OSDs are `up` and `in`.
Of course your explanation of what happens if it's `out` makes sense, but that isn't my scenario; if Ceph had 10 hours to move data off, of course I would have to expect at least 10 hours to move data back on. But it only has 5 minutes at max to move data off. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 9/22/25 14:42, Niklas Hambüchen wrote:
Well, if they were away long enough to get "out", then it is somewhat reasonable even for ~5m downtimes.
Right, but what I'm saying is that this is not what happens.
My reboot or disconnect takes < 5 minutes, and no OSD is `out` afterwards.
When I say "down for 5 minutes", I literally mean that the node goes down, comes back up, and I'm sitting in front of its terminal and observe that all OSDs are `up` and `in`.
Of course your explanation of what happens if it's `out` makes sense, but that isn't my scenario; if Ceph had 10 hours to move data off, of course I would have to expect at least 10 hours to move data back on. But it only has 5 minutes at max to move data off.
Are we talking about a replicated pool or EC here? And what is your failure domain? What might give insight is the following command: watch -n 5 ceph pg ls remapped In the first columns you can see how many objects / data are missing on each PG. Maybe note what the status is for a specific set of OSDs before a reboot. That might give a clue what happens. There is a difference in recovery strategy between replicated and EC [1]. Not sure about backfill, but this might be handled in a similar manner. In a replicated pool I would expect the time to backfill misplaced objects be roughly similar to the downtime. A bit longer actually as there might be a few OSDs that have multiple PGs on them and not enough backfill slots to do this in parallel (as was already mentioned in this thread). Gr. Stefan [1]: https://docs.ceph.com/en/latest/dev/osd_internals/log_based_pg/
On 9/23/25 13:45, Niklas Hambüchen wrote:
I'm in 3-replication, DC building as failure domain. Currently each DC has 1-2 hosts, and each host has ~10 OSDs.
watch -n 5 ceph pg ls remapped
Thanks, I'll try that next time it happens!
Hmm, we have similar (but larger) setups. We don't have misplaced objects after a node reboot, only degraded. Note that we have an even number of hosts in each DC. I also would not expect any misplaced objects (when OSDs do not get marked "out") at all. So I wonder how your CRUSH rule(s) look like, and if the DC placement CRUSH rule(s) are applied on all the pools. Can you provide your crush rules? ceph osd crush rule dump Example: <snip> "op": "chooseleaf_firstn", "num": 0, "type": "datacenter" <snap> Gr. Stefan
Hi, On 9/19/25 13:23, Niklas Hambüchen wrote:
I noticed that for my clusters, even a short 5-minute network outage or single-host reboot can cause
pgs: 5586988/366684639 objects misplaced (1.524%)
which at the speed of
recovery: 2.2 GiB/s, 676 objects/s
can take hours to recover.
I don't understand how this can be. If it's down for so short, how can rebalancing can take this long?
If the objects are only misplaced, but not degraded, the full data availability and consistency according to the chosen crush rules is already guaranteed. The objects are "only" on the wrong OSDs. In my experiences the backfilling operation for misplaced objects have a lower priority than other operation. Things are different if objects are _degraded_. In this case CEPH pushes the backfill operations. You also have to consider how many backfill operations can run in parallel. Each OSD has a number of backfill slots; and all OSDs handling a PG have to provide a free slot for the operation. For replicated pools this means that three OSDs are involved, probably more for EC pools. And finally there's Murphy....the last objects have to be handled by the same set of OSDS, resulting in a queue up of backfill operations and slow progress. There are a number of settings for backfilling, but these differ e.g. depending on the select osd_op_queue. Best regards, Burkhard
participants (6)
-
Anthony D'Atri
-
Boris
-
Burkhard Linke
-
Janne Johansson
-
Niklas Hambüchen
-
Stefan Kooman