On 9/22/25 14:42, Niklas Hambüchen wrote:
Well, if they were away long enough to get "out", then it is somewhat reasonable even for ~5m downtimes.
Right, but what I'm saying is that this is not what happens.
My reboot or disconnect takes < 5 minutes, and no OSD is `out` afterwards.
When I say "down for 5 minutes", I literally mean that the node goes down, comes back up, and I'm sitting in front of its terminal and observe that all OSDs are `up` and `in`.
Of course your explanation of what happens if it's `out` makes sense, but that isn't my scenario; if Ceph had 10 hours to move data off, of course I would have to expect at least 10 hours to move data back on. But it only has 5 minutes at max to move data off.
Are we talking about a replicated pool or EC here? And what is your failure domain? What might give insight is the following command: watch -n 5 ceph pg ls remapped In the first columns you can see how many objects / data are missing on each PG. Maybe note what the status is for a specific set of OSDs before a reboot. That might give a clue what happens. There is a difference in recovery strategy between replicated and EC [1]. Not sure about backfill, but this might be handled in a similar manner. In a replicated pool I would expect the time to backfill misplaced objects be roughly similar to the downtime. A bit longer actually as there might be a few OSDs that have multiple PGs on them and not enough backfill slots to do this in parallel (as was already mentioned in this thread). Gr. Stefan [1]: https://docs.ceph.com/en/latest/dev/osd_internals/log_based_pg/