OSD_FULL after OSD Node Failures
Hi, We recently had problems that meant 3 out of 32 OSD hosts went offline for about 10 minutes. The hosts are now back in the cluster as expected and backfilling going on. However we are seeing a couple of problems. We are seeing: 1. Ceph is flagging a handfull of PGs as backfill_toofull when they aren't. https://tracker.ceph.com/issues/61839. 2. Periodically it generates an OSD_FULL error. Are there any plans to look at resolving bug #61839? When an OSD_FULL error has been reported, the OSD in question has been <75% usage. The OSD wasn't used by any of the PGs reporting backfill_toofull. I currently have the full_ratio set to 0.97 and nearfull_ratio set to 0.87 so the OSDs are nowhere near these levels. The %raw usage of the OSDs in the cluster ranges from about 60-80% and the raw usage of the cluster is about 75%. We do not get any "near full" warnings prior to OSD_FULL being set. Having a production system instantly go offline without warning isn't ideal and these things seem to know the least convenient moment. These problems only happen after a host failure. Each time we have added additional OSD hosts into the cluster, the backfilling has finished without problems. We are currently running Reef 18.2.4 but I have experienced these same problems on Pacific 16.2.10 Has anyone else seen this behaviour? Thanks Gerard
Just to add a bit of additional information: - I have tried failing the active MGR but that appeared to make no difference. - I found restarting the primary OSD on a PG that is reporting backfill_toofull clears to toofull status. It could just be coincidence but since I've started clearing the backfill_toofull status as soon as it appears, I haven't had any OSDs reporting OSD_FULL. Gerard
Hi Gerard, You have very few PGs for very large disks. 50PGs account for 11TB of space. Also full, nearfull and backfillfull is the same value. You want to set them different. ceph osd set-nearfull-ratio .85 ceph osd set-backfillfull-ratio .90 ceph osd set-full-ratio .95 I could imagine a lot of PG movement is happening, and during the movement of a PG, the data is doubled until the new OSD can take over. What I usually did, was reweighting the full OSDs to .95 just to get things moving. When everything is back to normal try to set the amount of PGs higher (x2 or even x4). This also leads to a better distribution. Cheers Boris
Am 27.12.2024 um 05:09 schrieb Gerard Hand <g.hand@lancaster.ac.uk>:
Just to add a bit of additional information: - I have tried failing the active MGR but that appeared to make no difference. - I found restarting the primary OSD on a PG that is reporting backfill_toofull clears to toofull status.
It could just be coincidence but since I've started clearing the backfill_toofull status as soon as it appears, I haven't had any OSDs reporting OSD_FULL.
Gerard _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Boris
-
Gerard Hand