RGW Cascading Failures After Simultaneous Host Loss
Hi We recently experienced an incident in one of our production Ceph clusters and would greatly appreciate the community’s input. Our configuration uses Erasure Coding with a 4+2 profile for the main S3 data pools. During the incident, two storage hosts that contained several OSDs each became unavailable simultaneously. As a result, a large number of PGs entered an inactive state because the number of remaining fragments was insufficient for reconstruction under the EC 4+2 profile. This triggered a cascading failure in the RGW service layer: • Incoming S3 requests continued flowing into HAProxy. • RGW attempted to process requests that required I/O to inactive PGs. • I/O stalls caused excessive queue buildup and memory usage on RGWs. • Multiple RGW processes were terminated by the OOM Killer. • RGW daemons repeatedly restarted and were again selected by HAProxy. • The cycle continued, leading to severe service disruption. This behavior continued until the failed hosts were recovered and PGs regained a clean and active state. One interesting observation was that a separate set of RGW daemons dedicated only to internal consumers continued operating normally. Their traffic load was significantly lower, which prevented queue saturation and memory exhaustion. We are seeking guidance on two key questions: 1. In an EC 4+2 pool design, is such a cascading RGW failure considered inevitable if enough hosts fail to cause PG unavailability, or are there established best practices to prevent this behavior? 2. Based on our observation that internal RGWs remained healthy, is isolating customers by assigning separate RGWs to high-traffic and low-traffic groups considered a recommended architectural approach? Thank you in advance for any advice and shared experiences. Best regards, Ramin
Hi, first, you could have prevented inactive PGs by temporarily reducing min_size to 4 (default is k + 1). But don't leave min_size at k, that's too dangerous.
1. In an EC 4+2 pool design, is such a cascading RGW failure considered inevitable if enough hosts fail to cause PG unavailability, or are there established best practices to prevent this behavior?
I am not surprised about the RGW failure, although we haven't seen that happening on any cluster yet. How many OSD hosts do you have in total? You could add more nodes and use a different EC profile to be able to sustain the loss of two hosts without service interruption. But it really depends on your actual resiliency requirements. Now that you lost two and had a service disruption, it might be a good idea to reconsider.
2. Based on our observation that internal RGWs remained healthy, is isolating customers by assigning separate RGWs to high-traffic and low-traffic groups considered a recommended architectural approach?
I don't think it makes a difference, inactive PGs stall IO. Maybe there just wasn't enough client traffic to overload the internal RGWs, which looks as if they were "healthy"? But from a general perspective, it can make sense to dedicate RGWs to specific tasks. Regards, Eugen Zitat von Ramin Najjarbashi <ramin.najarbashi@gmail.com>:
Hi
We recently experienced an incident in one of our production Ceph clusters and would greatly appreciate the community’s input.
Our configuration uses Erasure Coding with a 4+2 profile for the main S3 data pools. During the incident, two storage hosts that contained several OSDs each became unavailable simultaneously. As a result, a large number of PGs entered an inactive state because the number of remaining fragments was insufficient for reconstruction under the EC 4+2 profile.
This triggered a cascading failure in the RGW service layer:
• Incoming S3 requests continued flowing into HAProxy. • RGW attempted to process requests that required I/O to inactive PGs. • I/O stalls caused excessive queue buildup and memory usage on RGWs. • Multiple RGW processes were terminated by the OOM Killer. • RGW daemons repeatedly restarted and were again selected by HAProxy. • The cycle continued, leading to severe service disruption.
This behavior continued until the failed hosts were recovered and PGs regained a clean and active state.
One interesting observation was that a separate set of RGW daemons dedicated only to internal consumers continued operating normally. Their traffic load was significantly lower, which prevented queue saturation and memory exhaustion.
We are seeking guidance on two key questions:
1. In an EC 4+2 pool design, is such a cascading RGW failure considered inevitable if enough hosts fail to cause PG unavailability, or are there established best practices to prevent this behavior? 2. Based on our observation that internal RGWs remained healthy, is isolating customers by assigning separate RGWs to high-traffic and low-traffic groups considered a recommended architectural approach?
Thank you in advance for any advice and shared experiences.
Best regards, Ramin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Eugen, Thank you very much for your detailed explanation and helpful guidance. Your point regarding temporarily lowering min_size to 4 in order to avoid inactive PGs is well-taken. We fully agree that keeping min_size at k would introduce unacceptable risk, so this would only be considered as a controlled short-term measure during failure events. Regarding resiliency, we currently have limited hardware capacity, which restricts our ability to either add more OSD nodes or adjust the EC profile immediately. However, based on this incident, we clearly recognize the need to revisit our failure domain design and EC configuration in the near future. Your clarification about RGW segmentation was also very useful. It makes sense that the internal RGWs only appeared healthy due to lower traffic, not due to architectural isolation. We will view that observation more carefully going forward. Thank you again for taking the time to share your experience. We truly appreciate the community support and will incorporate these recommendations into our next mitigation plan. Best regards, ًRamin On Tue, Oct 28, 2025 at 2:55 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
first, you could have prevented inactive PGs by temporarily reducing min_size to 4 (default is k + 1). But don't leave min_size at k, that's too dangerous.
1. In an EC 4+2 pool design, is such a cascading RGW failure considered inevitable if enough hosts fail to cause PG unavailability, or are there established best practices to prevent this behavior?
I am not surprised about the RGW failure, although we haven't seen that happening on any cluster yet. How many OSD hosts do you have in total? You could add more nodes and use a different EC profile to be able to sustain the loss of two hosts without service interruption. But it really depends on your actual resiliency requirements. Now that you lost two and had a service disruption, it might be a good idea to reconsider.
2. Based on our observation that internal RGWs remained healthy, is isolating customers by assigning separate RGWs to high-traffic and low-traffic groups considered a recommended architectural approach?
I don't think it makes a difference, inactive PGs stall IO. Maybe there just wasn't enough client traffic to overload the internal RGWs, which looks as if they were "healthy"? But from a general perspective, it can make sense to dedicate RGWs to specific tasks.
Regards, Eugen
Zitat von Ramin Najjarbashi <ramin.najarbashi@gmail.com>:
Hi
We recently experienced an incident in one of our production Ceph clusters and would greatly appreciate the community’s input.
Our configuration uses Erasure Coding with a 4+2 profile for the main S3 data pools. During the incident, two storage hosts that contained several OSDs each became unavailable simultaneously. As a result, a large number of PGs entered an inactive state because the number of remaining fragments was insufficient for reconstruction under the EC 4+2 profile.
This triggered a cascading failure in the RGW service layer:
• Incoming S3 requests continued flowing into HAProxy. • RGW attempted to process requests that required I/O to inactive PGs. • I/O stalls caused excessive queue buildup and memory usage on RGWs. • Multiple RGW processes were terminated by the OOM Killer. • RGW daemons repeatedly restarted and were again selected by HAProxy. • The cycle continued, leading to severe service disruption.
This behavior continued until the failed hosts were recovered and PGs regained a clean and active state.
One interesting observation was that a separate set of RGW daemons dedicated only to internal consumers continued operating normally. Their traffic load was significantly lower, which prevented queue saturation and memory exhaustion.
We are seeking guidance on two key questions:
1. In an EC 4+2 pool design, is such a cascading RGW failure considered inevitable if enough hosts fail to cause PG unavailability, or are there established best practices to prevent this behavior? 2. Based on our observation that internal RGWs remained healthy, is isolating customers by assigning separate RGWs to high-traffic and low-traffic groups considered a recommended architectural approach?
Thank you in advance for any advice and shared experiences.
Best regards, Ramin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Ramin Najjarbashi