Hi Michel, ----- Le 27 Mar 25, à 18:52, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :
Hello,
It seems we are at the end of our stressful adventure!
Awesome!
After the big rebalancing finished, without errors but without any significant impact on the pool access problem, we decided to reboot all our OSD servers one by one. The first good news is that it cleared all the reported issues (MDS complaining about a damages rank, slow ops...) and we were able to recover the access to all pools and filesystems. The second good news is that the OSD server reboot triggered no new rebalancing meaning that the PG placement is again stable.
Ceph is sometimes stressful but demonstrated again it is a robust storage platform as the data have not been in danger at anytime (just the access)!
Indeed.
With a great community to support us! Thanks!
I agree.
That said, Frédéric and other experts, do you think it is worth to do a post-mortem analysis
Well nobody died in this episode :-)
to understand how we ended up in such a mess after an incident that looked somewhat trivial (a few OSD crashing)?
Yes, investigating the sequence of events (at what time each OSD crashed, if some of them crashed simultaneously) and the state of the cluster at each time would surely help to understand what happened. Most importantly, you would better understand why some OSDs were OOM-killed by the kernel, to avoid this in the future. Are you using swap on your OSD nodes? Regards, Frédéric.
Best regards,
Michel
Le 27/03/2025 à 14:26, Frédéric Nass a écrit :
Michel,
I can't recall any situations like that - maybe someone here does? - but I would advise that you restart all OSDs to trigger the re-peering of every PG. This should get your cluster back on track.
Just make sure the crush map / crush rules / bucket weights (including OSDs weights) haven't changed, as this would of course trigger rebalancing.
Regards, Frédéric.
----- Le 27 Mar 25, à 13:30, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :
Frédéric,
When I was writing the last email, my colleague launched a re-peering of the PG in activating state: the PG became active immediately but triggered a little bit of rebalancing of other PGs, not necessarily in the same pool. After this success, we decided to go for your approach, selected a not too critical pool and did a repeer command on all the pool PGs. This resulted in a huge rebalancing (5M objects, in progress, affecting many pools), basically a rebalancing similar (in size) to the unexpected one we have seen after the incident 2 days ago. Could it mean that the state of some OSD was improperly set/used after the restart of OSD servers after the incident and may have resulted in an inappropriate placement of PGs that is being currently fixed after the repeer command causes a reevaluation of the crush map ?
Cheers,
Michel
Le 27/03/2025 à 12:16, Michel Jouvin a écrit :
Frédéric,
Thanks for your answer. I checked the number of PG on osd.17: it is 164, very far from the hard limit (750, the default I think). So it doesn't seem to be the problem and may be the peering is a victim of the more general problem leading to many pools to be more or less inaccessible. What inaccessible means here is not entirely clear:
- We tested the ability to access the pool content with 'rados ls' as I said and we considered that a pool was inaccessible when the command was timing out after 10s (no explicit error). This happens also on empty pools.
- At the same time, on one such pool at least, we were able to successfully upload and download a large file with a S3 client (this pool is part of the data pool of a Swift RGW).
To be honest we have not checked all the logs yet! We concentrated mainly on the mon logs but we'll have a look to some OSD logs.
As for restarting daemons, I am not so reluctant to do it. I have the feeling that in the absence of any message related to inconsistencies, there is no real risk if we restart them one by one and check with ok-to-stop before doing it. What's your feeling? Is it worth restarting the 3 mon first (one by one)?
You mention as an alternative re-peering all PGs of one pool.I was not aware we could do it but I see that there is a 'ceph pg repeer' command. Anything else we should do before running the command? Does it make sense to try it on the PG stucked in activating+remapped state?
Best regards,
Michel
Le 27/03/2025 à 11:40, Frédéric Nass a écrit :
echo "`ceph config get osd.0 mon_max_pg_per_osd`*`ceph config get osd.0 osd_max_pg_per_osd_hard_ratio`" | bc