And sorry for all these mails, I forgot to mention that we are running 18.2.2. Michel Le 26/03/2025 à 21:51, Michel Jouvin a écrit :

Hi again,

Looking for more info on the degraded filesystem, I managed to connect to the dashboard where I see an error not reported as explicitely by 'ceph health' :

One or more metadata daemons (MDS ranks) are failed or in a damaged state. At best the filesystem is partially available, at worst the filesystem is completely unusable.

But I don't manage what can be done from this point... and I really don't understand how we ended up in such a state...

Cheers,

Michel

Le 26/03/2025 à 21:27, Michel Jouvin a écrit :

...

Hi,

We have a production cluster made of 3 mon+mgr, 18 OSD servers and ~500 OSDs and configured with ~50 pools, 1/2 EC (9+6) and 1/2 replica 3. It also has 2 CephFS filesystems with 1 MDS each.

2 days ago, in a period spanning 16 hours, 13 OSD crashed with an OOM. The OSD were first restarted but it was decided to reboot the server with a crashed OSD and "by mistake" (it was at least useless), the OSD of the rebooted server were set noout,norebalance before the reboot. The flags were removed after the reboot.

After all of this, 'ceph -s' started to report a lot of misplaced PG and recovery started. All the PGs but one were successfully reactivated. One stayed in the activating+remapped state (located in a pool used for tests). 'ceph health' (I don't put the details here to avoid a too long mail but I can shared them) says:

HEALTH_WARN 1 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow metadata IOs; Reduced data availability: 1 pg inactive; 13 daemons have recently crashed

and reports about one of the filesystem being degraded despite the only PG reported inactive is not part of a pool related to the FS.

The recovery was slow until we realized we should change the mclock profile to high_recovery_ops. Then it completed in a few hours. Unfortunately the degraded filesystem remains degraded without an obvious reason... and the inactive page is still in the activating+remapped state. We have not been able to identify a relevant error in the logs up to now (but we may have missed something...).

So far we have avoided restarting too many things until we have a better understanding of what happened and what is the current state. We only restarted the mgr which was using a lot of CPU and the MDS for the degraded FS, without any improvement.

We are looking on advices about where to start... It seems we have (at least) 2 independent problems:

- A PG that cannot be reactivated with a remap operation that doesn't proceed: would stopping osd.17 help (so that osd.460 is reused)?

[root@ijc-mon1 ~]# ceph pg dump_stuck PG_STAT  STATE                UP            UP_PRIMARY ACTING         ACTING_PRIMARY 32.7ef   activating+remapped  [100,154,17]         100 [100,154,460]             100

- 1 degraded filesystem: where to look for a reason?

Thanks in advance for any help?

Cheers,

Michel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Michel, A common reason for PGs being stuck during activation is reaching the hard limit of PGs per OSD. You might want to compare the number of PGs osd.17 has (ceph osd df tree | grep -E 'osd.17 |PGS') to the hard limit set in your cluster (echo "`ceph config get osd.0 mon_max_pg_per_osd`*`ceph config get osd.0 osd_max_pg_per_osd_hard_ratio`" | bc). If both values are close to each other, increasing osd_max_pg_per_osd_hard_ratio should help. Re-peering PG 32.7ef (ceph pg repeer 32.7ef) may also help. Also, considering the situation, make sure to disable the PG autoscaler if in use. Regarding the overall cluster state, it is difficult to make any assessment without analyzing events and logs from MONs and OSDs. The fact that the 'rados ls' command fails for more than half of the pools suggests that one (or more) OSDs is misbehaving. Have you checked all OSDs and MONs logs? If you're hesitant about restarting the OSDs, you could try re-peering all PGs of a single pool and see if it helps the 'rados ls'. Regards, Frédéric. ----- Le 27 Mar 25, à 10:42, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :

Hi,

I have not seen an answer yet, help would be very much appreciated as our production cluster seems in a worst shape that initially described...

After a deeper analysis, we found that more than half of the pools, despite reported as ok, are not accessible: the 'rados ls' command is stuck when we try to access them. It is not correlated to the EC versus 3 replica pool configuration (both are affected or can be ok). We don't have clear view whether we have a corruption problem (but it is unclear why it would have happened) or a communication problem between some cluster components that may explain that pools reported as good are not accessible and that one PG remains in the activating+remapped state, something that seems unusual (I could not find any reference to this with my Google searches).

Then we are hesitating between forcing a deep scrub of everything (we have not seen any errors reported by scrubs/deep scrubs run in the last days) or restarting the whole cluster in case there is a kind of deadlock in the communication between some mon and/or osd. In case the second approach is suggested (cluster restart), should we do it daemon by daemon or shutdown everything and do a cold restart of the cluster?

I don't want to clutter this already too long thread with too many details but one of my colleague gave me the 'ceph -s' and 'ceph osd status` output before he started to reboot servers. If it is useful, I can share them.

Again, thanks in advance for any help/hint.

Best regards,

Michel

Le 26/03/2025 à 21:54, Michel Jouvin a écrit :

...

And sorry for all these mails, I forgot to mention that we are running 18.2.2.

Michel

Le 26/03/2025 à 21:51, Michel Jouvin a écrit :

...

Hi again,

Looking for more info on the degraded filesystem, I managed to connect to the dashboard where I see an error not reported as explicitely by 'ceph health' :

One or more metadata daemons (MDS ranks) are failed or in a damaged state. At best the filesystem is partially available, at worst the filesystem is completely unusable.

But I don't manage what can be done from this point... and I really don't understand how we ended up in such a state...

Cheers,

Michel

Le 26/03/2025 à 21:27, Michel Jouvin a écrit :

...

Hi,

We have a production cluster made of 3 mon+mgr, 18 OSD servers and ~500 OSDs and configured with ~50 pools, 1/2 EC (9+6) and 1/2 replica 3. It also has 2 CephFS filesystems with 1 MDS each.

2 days ago, in a period spanning 16 hours, 13 OSD crashed with an OOM. The OSD were first restarted but it was decided to reboot the server with a crashed OSD and "by mistake" (it was at least useless), the OSD of the rebooted server were set noout,norebalance before the reboot. The flags were removed after the reboot.

After all of this, 'ceph -s' started to report a lot of misplaced PG and recovery started. All the PGs but one were successfully reactivated. One stayed in the activating+remapped state (located in a pool used for tests). 'ceph health' (I don't put the details here to avoid a too long mail but I can shared them) says:

HEALTH_WARN 1 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow metadata IOs; Reduced data availability: 1 pg inactive; 13 daemons have recently crashed

and reports about one of the filesystem being degraded despite the only PG reported inactive is not part of a pool related to the FS.

The recovery was slow until we realized we should change the mclock profile to high_recovery_ops. Then it completed in a few hours. Unfortunately the degraded filesystem remains degraded without an obvious reason... and the inactive page is still in the activating+remapped state. We have not been able to identify a relevant error in the logs up to now (but we may have missed something...).

So far we have avoided restarting too many things until we have a better understanding of what happened and what is the current state. We only restarted the mgr which was using a lot of CPU and the MDS for the degraded FS, without any improvement.

We are looking on advices about where to start... It seems we have (at least) 2 independent problems:

- A PG that cannot be reactivated with a remap operation that doesn't proceed: would stopping osd.17 help (so that osd.460 is reused)?

[root@ijc-mon1 ~]# ceph pg dump_stuck PG_STAT  STATE                UP            UP_PRIMARY ACTING         ACTING_PRIMARY 32.7ef   activating+remapped  [100,154,17]         100 [100,154,460]             100

- 1 degraded filesystem: where to look for a reason?

Thanks in advance for any help?

Cheers,

Michel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Frédéric, When I was writing the last email, my colleague launched a re-peering of the PG in activating state: the PG became active immediately but triggered a little bit of rebalancing of other PGs, not necessarily in the same pool. After this success, we decided to go for your approach, selected a not too critical pool and did a repeer command on all the pool PGs. This resulted in a huge rebalancing (5M objects, in progress, affecting many pools), basically a rebalancing similar (in size) to the unexpected one we have seen after the incident 2 days ago. Could it mean that the state of some OSD was improperly set/used after the restart of OSD servers after the incident and may have resulted in an inappropriate placement of PGs that is being currently fixed after the repeer command causes a reevaluation of the crush map ? Cheers, Michel Le 27/03/2025 à 12:16, Michel Jouvin a écrit :

Frédéric,

Thanks for your answer. I checked the number of PG on osd.17: it is 164, very far from the hard limit (750, the default I think). So it doesn't seem to be the problem and may be the peering is a victim of the more general problem leading to many pools to be more or less inaccessible. What inaccessible means here is not entirely clear:

- We tested the ability to access the pool content with 'rados ls' as I said and we considered that a pool was inaccessible when the command was timing out after 10s (no explicit error). This happens also on empty pools.

- At the same time, on one such pool at least, we were able to successfully upload and download a large file with a S3 client (this pool is part of the data pool of a Swift RGW).

To be honest we have not checked all the logs yet! We concentrated mainly on the mon logs but we'll have a look to some OSD logs.

As for restarting daemons, I am not so reluctant to do it. I have the feeling that in the absence of any message related to inconsistencies, there is no real risk if we restart them one by one and check with ok-to-stop before doing it. What's your feeling? Is it worth restarting the 3 mon first (one by one)?

You mention as an alternative re-peering all PGs of one pool.I was not aware we could do it but I see that there is a 'ceph pg repeer' command. Anything else we should do before running the command? Does it make sense to try it on the PG stucked in activating+remapped state?

Best regards,

Michel

Le 27/03/2025 à 11:40, Frédéric Nass a écrit :

...

echo "`ceph config get osd.0 mon_max_pg_per_osd`*`ceph config get osd.0 osd_max_pg_per_osd_hard_ratio`" | bc

Hello, It seems we are at the end of our stressful adventure! After the big rebalancing finished, without errors but without any significant impact on the pool access problem, we decided to reboot all our OSD servers one by one. The first good news is that it cleared all the reported issues (MDS complaining about a damages rank, slow ops...) and we were able to recover the access to all pools and filesystems. The second good news is that the OSD server reboot triggered no new rebalancing meaning that the PG placement is again stable. Ceph is sometimes stressful but demonstrated again it is a robust storage platform as the data have not been in danger at anytime (just the access)! With a great community to support us! Thanks! That said, Frédéric and other experts, do you think it is worth to do a post-mortem analysis to understand how we ended up in such a mess after an incident that looked somewhat trivial (a few OSD crashing)? Best regards, Michel Le 27/03/2025 à 14:26, Frédéric Nass a écrit :

Michel,

I can't recall any situations like that - maybe someone here does? - but I would advise that you restart all OSDs to trigger the re-peering of every PG. This should get your cluster back on track.

Just make sure the crush map / crush rules / bucket weights (including OSDs weights) haven't changed, as this would of course trigger rebalancing.

Regards, Frédéric.

----- Le 27 Mar 25, à 13:30, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :

...

Frédéric,

When I was writing the last email, my colleague launched a re-peering of the PG in activating state: the PG became active immediately but triggered a little bit of rebalancing of other PGs, not necessarily in the same pool. After this success, we decided to go for your approach, selected a not too critical pool and did a repeer command on all the pool PGs. This resulted in a huge rebalancing (5M objects, in progress, affecting many pools), basically a rebalancing similar (in size) to the unexpected one we have seen after the incident 2 days ago. Could it mean that the state of some OSD was improperly set/used after the restart of OSD servers after the incident and may have resulted in an inappropriate placement of PGs that is being currently fixed after the repeer command causes a reevaluation of the crush map ?

Cheers,

Michel

Le 27/03/2025 à 12:16, Michel Jouvin a écrit :

...

Frédéric,

Thanks for your answer. I checked the number of PG on osd.17: it is 164, very far from the hard limit (750, the default I think). So it doesn't seem to be the problem and may be the peering is a victim of the more general problem leading to many pools to be more or less inaccessible. What inaccessible means here is not entirely clear:

- We tested the ability to access the pool content with 'rados ls' as I said and we considered that a pool was inaccessible when the command was timing out after 10s (no explicit error). This happens also on empty pools.

- At the same time, on one such pool at least, we were able to successfully upload and download a large file with a S3 client (this pool is part of the data pool of a Swift RGW).

To be honest we have not checked all the logs yet! We concentrated mainly on the mon logs but we'll have a look to some OSD logs.

As for restarting daemons, I am not so reluctant to do it. I have the feeling that in the absence of any message related to inconsistencies, there is no real risk if we restart them one by one and check with ok-to-stop before doing it. What's your feeling? Is it worth restarting the 3 mon first (one by one)?

You mention as an alternative re-peering all PGs of one pool.I was not aware we could do it but I see that there is a 'ceph pg repeer' command. Anything else we should do before running the command? Does it make sense to try it on the PG stucked in activating+remapped state?

Best regards,

Michel

Le 27/03/2025 à 11:40, Frédéric Nass a écrit :

...

echo "`ceph config get osd.0 mon_max_pg_per_osd`*`ceph config get osd.0 osd_max_pg_per_osd_hard_ratio`" | bc

More likely the problem would just migrate. I suggest `ceph pg repair 32.7ef`. If the situation doesn’t improve within a few minutes, try `ceph osd down 100`

On Mar 27, 2025, at 6:40 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:

...

...

...

...

- A PG that cannot be reactivated with a remap operation that doesn't proceed: would stopping osd.17 help (so that osd.460 is reused)?