- ceph-users - lists.ceph.io

Re: autoscale-status -no output
by Anthony D'Atri 11 Jul '26

11 Jul '26

There you go. default~MONERO is unusual, though. Please send: ceph df | head -10 How / why did you name a device class MONERO? By default OSDs will populate as "hdd" or "ssd" depending on the block device's `rotational` attribute. On Jul 11, 2026, at 11:00 AM, listy via ceph-users <ceph-users(a)ceph.io> wrote: -> $ ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "replicated_rule_MONERO", "type": 1, "steps": [ { "op": "take", "item": -16, "item_name": "default~MONERO" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] -> $ ceph osd crush class ls [ "MONERO" ] _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: autoscale-status -no output
by Anthony D'Atri 11 Jul '26

11 Jul '26

I documented this a few years ago, see the blue note here: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#viewing-… > Hi guys. > > I get no output from: > -> $ ceph osd pool autoscale-status > > I read docs and I thought I should have checked: > > -> $ ceph osd pool get .mgr crush_rule > crush_rule: replicated_rule > -> $ ceph osd pool get cephfs.MONERO.data crush_rule > crush_rule: replicated_rule_MONERO > -> $ ceph osd pool get cephfs.MONERO.meta crush_rule > crush_rule: replicated_rule_MONERO Please send `ceph osd crush rule dump` replicated_rule most likely looks like this: { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, which does not specify a CRUSH device class. Do you have both HDDs and SSDs? replicated_rule_MONERO probably looks like this: { "rule_id": 3, "rule_name": "replicated_hdd", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, Note the different in item_name: it specifies a device class via a shadow CRUSH tree. Which means that the rules in use are seen as spanning multiple CRUSH roots and the autoscaler "can't even". I put in an RFE for this last year. I've asked that the .mgr pool be created with a device class and that a health warning be raised if this situation detected. Depending on your constellation of rules, the best bet may be to use the CLI to change the CRUSH rule for the .mgr pool to replicated_rule_MONERO. Send the requested dump and I can say for sure. That said, I'd keep rule names lowercase, I don't *know* that mixed causes a problem, but I like to not tempt fate. > > All I have are above three pools - _.mgr_ created at/by deployment(in containers) > > I must be missing something else - what that might be? > I see no clear & obvious errors in the logs. > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: undersized - really ?
by Anthony D'Atri 11 Jul '26

11 Jul '26

> You have only 8 PGs in total? They take up several hundreds of GB, making one PG quite large. Agreed. We don't have information about whether the autoscaler is enabled, but the docs do have guidance about pg_num for small clusters. Fewer than 5 OSDs: set pg_num to 128. Between 5 and 10 OSDs: set pg_num to 512. Between 10 and 50 OSDs: set pg_num to 1024. That simplistically describes a single pool but you get the idea. A contributing factor is the very small OSDs. OSDs smaller than, say, half a GiB have a disproportionate fraction devoted to metadata and tend to not fill as one would like. >> 1 pg backfill_toofull; This is subtle: an OSD being over the backfill_toofull ratio is different from a PG being backfill_toofull, another case where we might have picked more distinct names but it's too late :-/ > That's why it's backfill_toofull, it can't store another 200 GB (or so) on that OSD because it would breach the (default) limit of 90%. You need to increase pg_num drastically to decrease your PG sizes. For reasons that I can't explain, this calculation in some circumstances effectively double-counts the data in a PG: once for (stale) data already on the destination, and once for the newer data on the source. PGs can be undersized because CRUSH can't place all shards/replicas because of the space / granularity dynamic. This will be especially pronounced if you're trying to do EC here. I'll guess that this is a sandbox cobbled together from what was laying around? If so I suggest sticking to replicated pools with size=2 to partly mitigate the limitations. > As a first shot I would aim for 100 PGs per OSD or so. For these small and non-uniform OSDs, I might go ever smaller

1 0

autoscale-status -no output
by listy 11 Jul '26

11 Jul '26

Hi guys. I get no output from: -> $ ceph osd pool autoscale-status I read docs and I thought I should have checked: -> $ ceph osd pool get .mgr crush_rule crush_rule: replicated_rule -> $ ceph osd pool get cephfs.MONERO.data crush_rule crush_rule: replicated_rule_MONERO -> $ ceph osd pool get cephfs.MONERO.meta crush_rule crush_rule: replicated_rule_MONERO All I have are above three pools - _.mgr_ created at/by deployment(in containers) I must be missing something else - what that might be? I see no clear & obvious errors in the logs. many thanks, L.

1 0

Re: undersized - really ?
by Eugen Block 11 Jul '26

11 Jul '26

You have only 8 PGs in total? They take up several hundreds of GB, making one PG quite large. That's why it's backfill_toofull, it can't store another 200 GB (or so) on that OSD because it would breach the (default) limit of 90%. You need to increase pg_num drastically to decrease your PG sizes. As a first shot I would aim for 100 PGs per OSD or so. Zitat von Sinan Polat via ceph-users <ceph-users(a)ceph.io>: > Hi, > > The cluster-wide utilization isn't the important metric here. > backfill_toofull means that the destination OSD selected for the backfill > is above the configured backfillfull_ratio, regardless of how much free > space the cluster has overall. > > Can you post the output of: > ceph pg 7.0 query > ceph osd dump | egrep 'nearfull_ratio|backfillfull_ratio|full_ratio' > > Kind regards, > Sinan > > Op za 11 jul 2026 om 09:41 schreef listy via ceph-users <ceph-users(a)ceph.io >> : > >> Hi guys. >> >> Is what ceph claims really the case? >> >> CEPH status is: HEALTH_WARN Low space hindering backfill >> (add storage if this doesn't resolve itself): 1 pg >> backfill_toofull; Degraded data redundancy: 69353/208545 >> objects degraded (33.256%), 1 pg degraded, 1 pg undersized >> [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add >> storage if this doesn't resolve itself): 1 pg backfill_toofull >> pg 7.0 is >> active+undersized+degraded+remapped+backfill_toofull, acting >> [3,2] >> [WRN] PG_DEGRADED: Degraded data redundancy: 69353/208545 >> objects degraded (33.256%), 1 pg degraded, 1 pg undersized >> pg 7.0 is stuck undersized for 7m, current state >> active+undersized+degraded+remapped+backfill_toofull, last >> acting [3,2] >> >> >> -> $ ceph osd df tree >> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA >> OMAP META AVAIL %USE VAR PGS STATUS TYPE >> NAME >> -1 1.61128 - 1.6 TiB 816 GiB 814 GiB >> 369 KiB 1.6 GiB 834 GiB 49.48 1.00 - root >> default >> -3 0.53709 - 550 GiB 272 GiB 271 GiB >> 147 KiB 540 MiB 278 GiB 49.44 1.00 - >> host podster1 >> 2 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB >> 96 KiB 499 MiB 128 GiB 67.96 1.37 3 up >> osd.2 >> 5 MONERO 0.14650 1.00000 150 GiB 49 MiB 8.1 MiB >> 51 KiB 41 MiB 150 GiB 0.03 0 0 up >> osd.5 >> -7 0.53709 - 550 GiB 272 GiB 271 GiB >> 171 KiB 532 MiB 278 GiB 49.43 1.00 - >> host podster2 >> 0 MONERO 0.14650 1.00000 150 GiB 60 MiB 8.5 MiB >> 92 KiB 51 MiB 150 GiB 0.04 0 1 up >> osd.0 >> 3 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB >> 79 KiB 481 MiB 128 GiB 67.96 1.37 2 up >> osd.3 >> -5 0.53709 - 550 GiB 273 GiB 271 GiB >> 51 KiB 543 MiB 277 GiB 49.58 1.00 - host >> podster3 >> 1 MONERO 0.14650 1.00000 150 GiB 447 MiB 8.5 MiB >> 16 KiB 38 MiB 150 GiB 0.29 0.01 1 up >> osd.1 >> 4 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB >> 35 KiB 504 MiB 128 GiB 68.06 1.38 1 up >> osd.4 >> TOTAL 1.6 TiB 816 GiB 814 GiB >> 372 KiB 1.6 GiB 834 GiB 49.48 >> >> MIN/MAX VAR: 0/1.38 STDDEV: 37.28 >> >> I cannot "decipher" this - can you help, explain what is >> (not)happening here - this "state" persists for a few days, >> I do not suppose it will heal itself. >> >> many thanks, L. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: undersized - really ?
by Sinan Polat 11 Jul '26

11 Jul '26

Hi, The cluster-wide utilization isn't the important metric here. backfill_toofull means that the destination OSD selected for the backfill is above the configured backfillfull_ratio, regardless of how much free space the cluster has overall. Can you post the output of: ceph pg 7.0 query ceph osd dump | egrep 'nearfull_ratio|backfillfull_ratio|full_ratio' Kind regards, Sinan Op za 11 jul 2026 om 09:41 schreef listy via ceph-users <ceph-users(a)ceph.io >: > Hi guys. > > Is what ceph claims really the case? > > CEPH status is: HEALTH_WARN Low space hindering backfill > (add storage if this doesn't resolve itself): 1 pg > backfill_toofull; Degraded data redundancy: 69353/208545 > objects degraded (33.256%), 1 pg degraded, 1 pg undersized > [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add > storage if this doesn't resolve itself): 1 pg backfill_toofull > pg 7.0 is > active+undersized+degraded+remapped+backfill_toofull, acting > [3,2] > [WRN] PG_DEGRADED: Degraded data redundancy: 69353/208545 > objects degraded (33.256%), 1 pg degraded, 1 pg undersized > pg 7.0 is stuck undersized for 7m, current state > active+undersized+degraded+remapped+backfill_toofull, last > acting [3,2] > > > -> $ ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA > OMAP META AVAIL %USE VAR PGS STATUS TYPE > NAME > -1 1.61128 - 1.6 TiB 816 GiB 814 GiB > 369 KiB 1.6 GiB 834 GiB 49.48 1.00 - root > default > -3 0.53709 - 550 GiB 272 GiB 271 GiB > 147 KiB 540 MiB 278 GiB 49.44 1.00 - > host podster1 > 2 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB > 96 KiB 499 MiB 128 GiB 67.96 1.37 3 up > osd.2 > 5 MONERO 0.14650 1.00000 150 GiB 49 MiB 8.1 MiB > 51 KiB 41 MiB 150 GiB 0.03 0 0 up > osd.5 > -7 0.53709 - 550 GiB 272 GiB 271 GiB > 171 KiB 532 MiB 278 GiB 49.43 1.00 - > host podster2 > 0 MONERO 0.14650 1.00000 150 GiB 60 MiB 8.5 MiB > 92 KiB 51 MiB 150 GiB 0.04 0 1 up > osd.0 > 3 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB > 79 KiB 481 MiB 128 GiB 67.96 1.37 2 up > osd.3 > -5 0.53709 - 550 GiB 273 GiB 271 GiB > 51 KiB 543 MiB 277 GiB 49.58 1.00 - host > podster3 > 1 MONERO 0.14650 1.00000 150 GiB 447 MiB 8.5 MiB > 16 KiB 38 MiB 150 GiB 0.29 0.01 1 up > osd.1 > 4 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB > 35 KiB 504 MiB 128 GiB 68.06 1.38 1 up > osd.4 > TOTAL 1.6 TiB 816 GiB 814 GiB > 372 KiB 1.6 GiB 834 GiB 49.48 > > MIN/MAX VAR: 0/1.38 STDDEV: 37.28 > > I cannot "decipher" this - can you help, explain what is > (not)happening here - this "state" persists for a few days, > I do not suppose it will heal itself. > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

undersized - really ?
by listy 11 Jul '26

11 Jul '26

Hi guys. Is what ceph claims really the case? CEPH status is: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull; Degraded data redundancy: 69353/208545 objects degraded (33.256%), 1 pg degraded, 1 pg undersized [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull pg 7.0 is active+undersized+degraded+remapped+backfill_toofull, acting [3,2] [WRN] PG_DEGRADED: Degraded data redundancy: 69353/208545 objects degraded (33.256%), 1 pg degraded, 1 pg undersized pg 7.0 is stuck undersized for 7m, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,2] -> $ ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1.61128 - 1.6 TiB 816 GiB 814 GiB 369 KiB 1.6 GiB 834 GiB 49.48 1.00 - root default -3 0.53709 - 550 GiB 272 GiB 271 GiB 147 KiB 540 MiB 278 GiB 49.44 1.00 - host podster1 2 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB 96 KiB 499 MiB 128 GiB 67.96 1.37 3 up osd.2 5 MONERO 0.14650 1.00000 150 GiB 49 MiB 8.1 MiB 51 KiB 41 MiB 150 GiB 0.03 0 0 up osd.5 -7 0.53709 - 550 GiB 272 GiB 271 GiB 171 KiB 532 MiB 278 GiB 49.43 1.00 - host podster2 0 MONERO 0.14650 1.00000 150 GiB 60 MiB 8.5 MiB 92 KiB 51 MiB 150 GiB 0.04 0 1 up osd.0 3 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB 79 KiB 481 MiB 128 GiB 67.96 1.37 2 up osd.3 -5 0.53709 - 550 GiB 273 GiB 271 GiB 51 KiB 543 MiB 277 GiB 49.58 1.00 - host podster3 1 MONERO 0.14650 1.00000 150 GiB 447 MiB 8.5 MiB 16 KiB 38 MiB 150 GiB 0.29 0.01 1 up osd.1 4 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB 35 KiB 504 MiB 128 GiB 68.06 1.38 1 up osd.4 TOTAL 1.6 TiB 816 GiB 814 GiB 372 KiB 1.6 GiB 834 GiB 49.48 MIN/MAX VAR: 0/1.38 STDDEV: 37.28 I cannot "decipher" this - can you help, explain what is (not)happening here - this "state" persists for a few days, I do not suppose it will heal itself. many thanks, L.

1 0

Ceph-flix! What's new on the Ceph YouTube Channel
by Anthony Middleton 10 Jul '26

10 Jul '26

Happy Friday, Ceph Friends! Who needs Netflix when you have the Ceph YouTube channel? Here are some recently added videos to enjoy over the weekend. July Ceph Tech Talk: For Whom the Doorbell Tolls, GPU-Initiated Key-Value Storage on Ceph <https://youtu.be/Z7zuATuBORw>, featuring Kyle Bader, Chief Architect, Data & AI, Ceph at IBM. Ceph Days London 2026 Playlist <https://www.youtube.com/watch?v=3GTet-knEoc&list=PLDhUTUI2hFE8>: all 16 talks with slides included Ceph Days Seattle 2026 Playlist <https://www.youtube.com/watch?v=bOy9-4KNziY&list=PLrBUGiINAakOCSyUapJL6ToBv…>: all 8 talks with slides included Enjoy your weekend. Feel free to message me with suggestions for a talk or blog post. I'm happy to support! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: squid 19.2.5 QE validation status
by Laura Flores 09 Jul '26

09 Jul '26

@Yuri Weinstein<mailto:yweinste@redhat.com> I added a bit more to Nitzan's summary. I have asked @rzarzynski to confirm there are no blockers for rados and upgrade. Upgrade and Rados suites are summarized here: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1925-httpstrackercephco… @Redouane Kachach<mailto:rkachach@ibm.com>, @Venky Shankar<mailto:Venky.Shankar@ibm.com>, and @Greg Farnum<mailto:gfarnum@ibm.com> please review the results for the upgrade suite (see link above)and confirm there are no CephFS or Orchestrator blockers. I have also written this comment on the tracker ticket: https://tracker.ceph.com/issues/76983#note-6 Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> Sent: Monday, June 29, 2026 9:30 AM To: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] squid 19.2.5 QE validation status Details of this release are summarized here: https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… Release Notes - TBD LRC upgrade - N/A Note: please reply to this email with comments/approvals and also add your comments to the tracker above. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Nitzan, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade: Laura, Patrick (see https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… , https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ceph_ceph_p… ) upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

Performance issue after tentacle upgrade
by Moti 09 Jul '26

09 Jul '26

Hello everyone! I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing as well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET and HEAD requests. There is no resource cap or significant increase in usage is visible. Neither fast ec nor any other new features are enabled yet, just did the upgrade and left it as is. For measuring latency I have a services that probes the cluster with consistent load so it's not a difference in load either. PG status history does not show any pg operation being heavily in process. Cluster has a hybrid build so OSDs have HDD for data and NVMe for db block. Both ec and replicated pools exist. The strange hint was: NVMe disks' iops went from something around 5-6K constant io per second to almost zero (less than a hundred per second, on all host/OSDs, with some non-frequent high peaks), while the HDD iops is not much changed. db blocks on NVMe are still attached to OSDs with no issue. I can't find a clue to explain this. Got the same results on 3 separate clusters (all went version 18 --> 20, same hybrid build and configs). Nothing in config dump changes (changed defaults) catches my attention to have caused this. Anyone had similar issues? Any tips and ideas on what to check?

1 0