CephFS limp mode when fullest OSD is between nearfull & backfillfull value
Good day We've been struggling with this issue since we've upgraded post 16.2.11 to 16.2.15 and now up to Reef 18.2.7. We didn't have this issue prior to upgrading from 16.2.11 to 16.2.15. The moment the fullest OSD % is between the nearful and backfillful value, CephFS goes into a limp mode. Meaning our Client IO goes from ~30 *GiB*/s to 100 *MiB*/s. Doesn't matter what the threshold values are set. If my nearful% is 75% and backfillfull 80%, with the highest OSD 75.01% it will limp. If my nearful% is 85% and backfillfull 90%, with the highest OSD 85.01% it will go into limp mode. The only way to get CephFS operational again, is to set both ratios to either far below or above the highest OSD value. e.g. if the fullest is 72% , I need to set it to ceph osd set-nearfull-ratio 0.76 & ceph osd set-backfillfull-ratio 0.81 When I deploy a brand new cluster from scratch on our testbed I get the same issue, Cephadm / Reef 18.2.4. Back in the days I was able to manipulate these settings with the following as well, but I think since Pacific they are no longer tuneable: ceph tell osd.1231 config show | grep -E 'osd_nearfull_ratio|osd_backfillfull_ratio|osd_full_ratio' "mon_osd_backfillfull_ratio": "0.900000", "mon_osd_full_ratio": "0.950000", "mon_osd_nearfull_ratio": "0.850000", ceph tell osd.$osd injectargs '--osd_nearfull_ratio=0.85' ceph tell osd.$osd injectargs '--osd_backfillfull_ratio=0.90' ceph tell osd.$osd injectargs '--osd_full_ratio=0.95' URL to the issue: https://tracker.ceph.com/issues/70129 Any ideas would be greatly appreciated. -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa
It’s the clients disabling write-back cache because it’s risky to have dirty data on a cluster getting full. Last time I checked it was hardcoded in kernel client so no way to disable this. Only remedy is increasing ratios like you stated. On Thu, 19 Jun 2025 at 09:14, Jeremi-Ernst Avenant <jeremi@idia.ac.za> wrote:
Good day
We've been struggling with this issue since we've upgraded post 16.2.11 to 16.2.15 and now up to Reef 18.2.7. We didn't have this issue prior to upgrading from 16.2.11 to 16.2.15.
The moment the fullest OSD % is between the nearful and backfillful value, CephFS goes into a limp mode. Meaning our Client IO goes from ~30 *GiB*/s to 100 *MiB*/s. Doesn't matter what the threshold values are set.
If my nearful% is 75% and backfillfull 80%, with the highest OSD 75.01% it will limp. If my nearful% is 85% and backfillfull 90%, with the highest OSD 85.01% it will go into limp mode.
The only way to get CephFS operational again, is to set both ratios to either far below or above the highest OSD value.
e.g. if the fullest is 72% , I need to set it to ceph osd set-nearfull-ratio 0.76 & ceph osd set-backfillfull-ratio 0.81
When I deploy a brand new cluster from scratch on our testbed I get the same issue, Cephadm / Reef 18.2.4.
Back in the days I was able to manipulate these settings with the following as well, but I think since Pacific they are no longer tuneable:
ceph tell osd.1231 config show | grep -E 'osd_nearfull_ratio|osd_backfillfull_ratio|osd_full_ratio' "mon_osd_backfillfull_ratio": "0.900000", "mon_osd_full_ratio": "0.950000", "mon_osd_nearfull_ratio": "0.850000",
ceph tell osd.$osd injectargs '--osd_nearfull_ratio=0.85' ceph tell osd.$osd injectargs '--osd_backfillfull_ratio=0.90' ceph tell osd.$osd injectargs '--osd_full_ratio=0.95'
URL to the issue: https://tracker.ceph.com/issues/70129
Any ideas would be greatly appreciated.
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi " It’s the clients disabling write-back cache because it’s risky to have dirty data on a cluster getting full. Last time I checked it was hardcoded in kernel client so no way to disable this. Only remedy is increasing ratios like you stated. " The thing is, why is cephfs running fine again, if the highest osd is above the backfillfull_ratio? e.g. Nearful set to 85% , backfillfull set to 90%, and my fullest disk 90.01%, Then everything is 100% again. Issue is when the fullest OSD is between the values of nearful_ratio & backfillfull_ratio. Even if nearfull is 50%, backfillfull_ratio 55% and fullest disk 53%. Then it still goes into limp mode. Regards On Thu, Jun 19, 2025 at 9:03 AM Jeremi-Ernst Avenant <jeremi@idia.ac.za> wrote:
Good day
We've been struggling with this issue since we've upgraded post 16.2.11 to 16.2.15 and now up to Reef 18.2.7. We didn't have this issue prior to upgrading from 16.2.11 to 16.2.15.
The moment the fullest OSD % is between the nearful and backfillful value, CephFS goes into a limp mode. Meaning our Client IO goes from ~30 *GiB*/s to 100 *MiB*/s. Doesn't matter what the threshold values are set.
If my nearful% is 75% and backfillfull 80%, with the highest OSD 75.01% it will limp. If my nearful% is 85% and backfillfull 90%, with the highest OSD 85.01% it will go into limp mode.
The only way to get CephFS operational again, is to set both ratios to either far below or above the highest OSD value.
e.g. if the fullest is 72% , I need to set it to ceph osd set-nearfull-ratio 0.76 & ceph osd set-backfillfull-ratio 0.81
When I deploy a brand new cluster from scratch on our testbed I get the same issue, Cephadm / Reef 18.2.4.
Back in the days I was able to manipulate these settings with the following as well, but I think since Pacific they are no longer tuneable:
ceph tell osd.1231 config show | grep -E 'osd_nearfull_ratio|osd_backfillfull_ratio|osd_full_ratio' "mon_osd_backfillfull_ratio": "0.900000", "mon_osd_full_ratio": "0.950000", "mon_osd_nearfull_ratio": "0.850000",
ceph tell osd.$osd injectargs '--osd_nearfull_ratio=0.85' ceph tell osd.$osd injectargs '--osd_backfillfull_ratio=0.90' ceph tell osd.$osd injectargs '--osd_full_ratio=0.95'
URL to the issue: https://tracker.ceph.com/issues/70129
Any ideas would be greatly appreciated.
--
*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town
Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa
-- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa
participants (2)
-
Can Özyurt
-
Jeremi-Ernst Avenant