July 2026 - ceph-users - lists.ceph.io

Re: undersized - really ?
by Anthony D'Atri 11 Jul '26

11 Jul '26

> You have only 8 PGs in total? They take up several hundreds of GB, making one PG quite large. Agreed. We don't have information about whether the autoscaler is enabled, but the docs do have guidance about pg_num for small clusters. Fewer than 5 OSDs: set pg_num to 128. Between 5 and 10 OSDs: set pg_num to 512. Between 10 and 50 OSDs: set pg_num to 1024. That simplistically describes a single pool but you get the idea. A contributing factor is the very small OSDs. OSDs smaller than, say, half a GiB have a disproportionate fraction devoted to metadata and tend to not fill as one would like. >> 1 pg backfill_toofull; This is subtle: an OSD being over the backfill_toofull ratio is different from a PG being backfill_toofull, another case where we might have picked more distinct names but it's too late :-/ > That's why it's backfill_toofull, it can't store another 200 GB (or so) on that OSD because it would breach the (default) limit of 90%. You need to increase pg_num drastically to decrease your PG sizes. For reasons that I can't explain, this calculation in some circumstances effectively double-counts the data in a PG: once for (stale) data already on the destination, and once for the newer data on the source. PGs can be undersized because CRUSH can't place all shards/replicas because of the space / granularity dynamic. This will be especially pronounced if you're trying to do EC here. I'll guess that this is a sandbox cobbled together from what was laying around? If so I suggest sticking to replicated pools with size=2 to partly mitigate the limitations. > As a first shot I would aim for 100 PGs per OSD or so. For these small and non-uniform OSDs, I might go ever smaller

1 0

autoscale-status -no output
by listy 11 Jul '26

11 Jul '26

Hi guys. I get no output from: -> $ ceph osd pool autoscale-status I read docs and I thought I should have checked: -> $ ceph osd pool get .mgr crush_rule crush_rule: replicated_rule -> $ ceph osd pool get cephfs.MONERO.data crush_rule crush_rule: replicated_rule_MONERO -> $ ceph osd pool get cephfs.MONERO.meta crush_rule crush_rule: replicated_rule_MONERO All I have are above three pools - _.mgr_ created at/by deployment(in containers) I must be missing something else - what that might be? I see no clear & obvious errors in the logs. many thanks, L.

1 0

Re: undersized - really ?
by Eugen Block 11 Jul '26

11 Jul '26

You have only 8 PGs in total? They take up several hundreds of GB, making one PG quite large. That's why it's backfill_toofull, it can't store another 200 GB (or so) on that OSD because it would breach the (default) limit of 90%. You need to increase pg_num drastically to decrease your PG sizes. As a first shot I would aim for 100 PGs per OSD or so. Zitat von Sinan Polat via ceph-users <ceph-users(a)ceph.io>: > Hi, > > The cluster-wide utilization isn't the important metric here. > backfill_toofull means that the destination OSD selected for the backfill > is above the configured backfillfull_ratio, regardless of how much free > space the cluster has overall. > > Can you post the output of: > ceph pg 7.0 query > ceph osd dump | egrep 'nearfull_ratio|backfillfull_ratio|full_ratio' > > Kind regards, > Sinan > > Op za 11 jul 2026 om 09:41 schreef listy via ceph-users <ceph-users(a)ceph.io >> : > >> Hi guys. >> >> Is what ceph claims really the case? >> >> CEPH status is: HEALTH_WARN Low space hindering backfill >> (add storage if this doesn't resolve itself): 1 pg >> backfill_toofull; Degraded data redundancy: 69353/208545 >> objects degraded (33.256%), 1 pg degraded, 1 pg undersized >> [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add >> storage if this doesn't resolve itself): 1 pg backfill_toofull >> pg 7.0 is >> active+undersized+degraded+remapped+backfill_toofull, acting >> [3,2] >> [WRN] PG_DEGRADED: Degraded data redundancy: 69353/208545 >> objects degraded (33.256%), 1 pg degraded, 1 pg undersized >> pg 7.0 is stuck undersized for 7m, current state >> active+undersized+degraded+remapped+backfill_toofull, last >> acting [3,2] >> >> >> -> $ ceph osd df tree >> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA >> OMAP META AVAIL %USE VAR PGS STATUS TYPE >> NAME >> -1 1.61128 - 1.6 TiB 816 GiB 814 GiB >> 369 KiB 1.6 GiB 834 GiB 49.48 1.00 - root >> default >> -3 0.53709 - 550 GiB 272 GiB 271 GiB >> 147 KiB 540 MiB 278 GiB 49.44 1.00 - >> host podster1 >> 2 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB >> 96 KiB 499 MiB 128 GiB 67.96 1.37 3 up >> osd.2 >> 5 MONERO 0.14650 1.00000 150 GiB 49 MiB 8.1 MiB >> 51 KiB 41 MiB 150 GiB 0.03 0 0 up >> osd.5 >> -7 0.53709 - 550 GiB 272 GiB 271 GiB >> 171 KiB 532 MiB 278 GiB 49.43 1.00 - >> host podster2 >> 0 MONERO 0.14650 1.00000 150 GiB 60 MiB 8.5 MiB >> 92 KiB 51 MiB 150 GiB 0.04 0 1 up >> osd.0 >> 3 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB >> 79 KiB 481 MiB 128 GiB 67.96 1.37 2 up >> osd.3 >> -5 0.53709 - 550 GiB 273 GiB 271 GiB >> 51 KiB 543 MiB 277 GiB 49.58 1.00 - host >> podster3 >> 1 MONERO 0.14650 1.00000 150 GiB 447 MiB 8.5 MiB >> 16 KiB 38 MiB 150 GiB 0.29 0.01 1 up >> osd.1 >> 4 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB >> 35 KiB 504 MiB 128 GiB 68.06 1.38 1 up >> osd.4 >> TOTAL 1.6 TiB 816 GiB 814 GiB >> 372 KiB 1.6 GiB 834 GiB 49.48 >> >> MIN/MAX VAR: 0/1.38 STDDEV: 37.28 >> >> I cannot "decipher" this - can you help, explain what is >> (not)happening here - this "state" persists for a few days, >> I do not suppose it will heal itself. >> >> many thanks, L. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: undersized - really ?
by Sinan Polat 11 Jul '26

11 Jul '26

Hi, The cluster-wide utilization isn't the important metric here. backfill_toofull means that the destination OSD selected for the backfill is above the configured backfillfull_ratio, regardless of how much free space the cluster has overall. Can you post the output of: ceph pg 7.0 query ceph osd dump | egrep 'nearfull_ratio|backfillfull_ratio|full_ratio' Kind regards, Sinan Op za 11 jul 2026 om 09:41 schreef listy via ceph-users <ceph-users(a)ceph.io >: > Hi guys. > > Is what ceph claims really the case? > > CEPH status is: HEALTH_WARN Low space hindering backfill > (add storage if this doesn't resolve itself): 1 pg > backfill_toofull; Degraded data redundancy: 69353/208545 > objects degraded (33.256%), 1 pg degraded, 1 pg undersized > [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add > storage if this doesn't resolve itself): 1 pg backfill_toofull > pg 7.0 is > active+undersized+degraded+remapped+backfill_toofull, acting > [3,2] > [WRN] PG_DEGRADED: Degraded data redundancy: 69353/208545 > objects degraded (33.256%), 1 pg degraded, 1 pg undersized > pg 7.0 is stuck undersized for 7m, current state > active+undersized+degraded+remapped+backfill_toofull, last > acting [3,2] > > > -> $ ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA > OMAP META AVAIL %USE VAR PGS STATUS TYPE > NAME > -1 1.61128 - 1.6 TiB 816 GiB 814 GiB > 369 KiB 1.6 GiB 834 GiB 49.48 1.00 - root > default > -3 0.53709 - 550 GiB 272 GiB 271 GiB > 147 KiB 540 MiB 278 GiB 49.44 1.00 - > host podster1 > 2 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB > 96 KiB 499 MiB 128 GiB 67.96 1.37 3 up > osd.2 > 5 MONERO 0.14650 1.00000 150 GiB 49 MiB 8.1 MiB > 51 KiB 41 MiB 150 GiB 0.03 0 0 up > osd.5 > -7 0.53709 - 550 GiB 272 GiB 271 GiB > 171 KiB 532 MiB 278 GiB 49.43 1.00 - > host podster2 > 0 MONERO 0.14650 1.00000 150 GiB 60 MiB 8.5 MiB > 92 KiB 51 MiB 150 GiB 0.04 0 1 up > osd.0 > 3 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB > 79 KiB 481 MiB 128 GiB 67.96 1.37 2 up > osd.3 > -5 0.53709 - 550 GiB 273 GiB 271 GiB > 51 KiB 543 MiB 277 GiB 49.58 1.00 - host > podster3 > 1 MONERO 0.14650 1.00000 150 GiB 447 MiB 8.5 MiB > 16 KiB 38 MiB 150 GiB 0.29 0.01 1 up > osd.1 > 4 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB > 35 KiB 504 MiB 128 GiB 68.06 1.38 1 up > osd.4 > TOTAL 1.6 TiB 816 GiB 814 GiB > 372 KiB 1.6 GiB 834 GiB 49.48 > > MIN/MAX VAR: 0/1.38 STDDEV: 37.28 > > I cannot "decipher" this - can you help, explain what is > (not)happening here - this "state" persists for a few days, > I do not suppose it will heal itself. > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

undersized - really ?
by listy 11 Jul '26

11 Jul '26

Hi guys. Is what ceph claims really the case? CEPH status is: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull; Degraded data redundancy: 69353/208545 objects degraded (33.256%), 1 pg degraded, 1 pg undersized [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull pg 7.0 is active+undersized+degraded+remapped+backfill_toofull, acting [3,2] [WRN] PG_DEGRADED: Degraded data redundancy: 69353/208545 objects degraded (33.256%), 1 pg degraded, 1 pg undersized pg 7.0 is stuck undersized for 7m, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,2] -> $ ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1.61128 - 1.6 TiB 816 GiB 814 GiB 369 KiB 1.6 GiB 834 GiB 49.48 1.00 - root default -3 0.53709 - 550 GiB 272 GiB 271 GiB 147 KiB 540 MiB 278 GiB 49.44 1.00 - host podster1 2 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB 96 KiB 499 MiB 128 GiB 67.96 1.37 3 up osd.2 5 MONERO 0.14650 1.00000 150 GiB 49 MiB 8.1 MiB 51 KiB 41 MiB 150 GiB 0.03 0 0 up osd.5 -7 0.53709 - 550 GiB 272 GiB 271 GiB 171 KiB 532 MiB 278 GiB 49.43 1.00 - host podster2 0 MONERO 0.14650 1.00000 150 GiB 60 MiB 8.5 MiB 92 KiB 51 MiB 150 GiB 0.04 0 1 up osd.0 3 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB 79 KiB 481 MiB 128 GiB 67.96 1.37 2 up osd.3 -5 0.53709 - 550 GiB 273 GiB 271 GiB 51 KiB 543 MiB 277 GiB 49.58 1.00 - host podster3 1 MONERO 0.14650 1.00000 150 GiB 447 MiB 8.5 MiB 16 KiB 38 MiB 150 GiB 0.29 0.01 1 up osd.1 4 MONERO 0.39059 1.00000 400 GiB 272 GiB 271 GiB 35 KiB 504 MiB 128 GiB 68.06 1.38 1 up osd.4 TOTAL 1.6 TiB 816 GiB 814 GiB 372 KiB 1.6 GiB 834 GiB 49.48 MIN/MAX VAR: 0/1.38 STDDEV: 37.28 I cannot "decipher" this - can you help, explain what is (not)happening here - this "state" persists for a few days, I do not suppose it will heal itself. many thanks, L.

1 0

Ceph-flix! What's new on the Ceph YouTube Channel
by Anthony Middleton 10 Jul '26

10 Jul '26

Happy Friday, Ceph Friends! Who needs Netflix when you have the Ceph YouTube channel? Here are some recently added videos to enjoy over the weekend. July Ceph Tech Talk: For Whom the Doorbell Tolls, GPU-Initiated Key-Value Storage on Ceph <https://youtu.be/Z7zuATuBORw>, featuring Kyle Bader, Chief Architect, Data & AI, Ceph at IBM. Ceph Days London 2026 Playlist <https://www.youtube.com/watch?v=3GTet-knEoc&list=PLDhUTUI2hFE8>: all 16 talks with slides included Ceph Days Seattle 2026 Playlist <https://www.youtube.com/watch?v=bOy9-4KNziY&list=PLrBUGiINAakOCSyUapJL6ToBv…>: all 8 talks with slides included Enjoy your weekend. Feel free to message me with suggestions for a talk or blog post. I'm happy to support! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: squid 19.2.5 QE validation status
by Laura Flores 09 Jul '26

09 Jul '26

@Yuri Weinstein<mailto:yweinste@redhat.com> I added a bit more to Nitzan's summary. I have asked @rzarzynski to confirm there are no blockers for rados and upgrade. Upgrade and Rados suites are summarized here: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1925-httpstrackercephco… @Redouane Kachach<mailto:rkachach@ibm.com>, @Venky Shankar<mailto:Venky.Shankar@ibm.com>, and @Greg Farnum<mailto:gfarnum@ibm.com> please review the results for the upgrade suite (see link above)and confirm there are no CephFS or Orchestrator blockers. I have also written this comment on the tracker ticket: https://tracker.ceph.com/issues/76983#note-6 Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> Sent: Monday, June 29, 2026 9:30 AM To: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] squid 19.2.5 QE validation status Details of this release are summarized here: https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… Release Notes - TBD LRC upgrade - N/A Note: please reply to this email with comments/approvals and also add your comments to the tracker above. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Nitzan, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade: Laura, Patrick (see https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… , https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ceph_ceph_p… ) upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

Performance issue after tentacle upgrade
by Moti 09 Jul '26

09 Jul '26

Hello everyone! I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing as well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET and HEAD requests. There is no resource cap or significant increase in usage is visible. Neither fast ec nor any other new features are enabled yet, just did the upgrade and left it as is. For measuring latency I have a services that probes the cluster with consistent load so it's not a difference in load either. PG status history does not show any pg operation being heavily in process. Cluster has a hybrid build so OSDs have HDD for data and NVMe for db block. Both ec and replicated pools exist. The strange hint was: NVMe disks' iops went from something around 5-6K constant io per second to almost zero (less than a hundred per second, on all host/OSDs, with some non-frequent high peaks), while the HDD iops is not much changed. db blocks on NVMe are still attached to OSDs with no issue. I can't find a clue to explain this. Got the same results on 3 separate clusters (all went version 18 --> 20, same hybrid build and configs). Nothing in config dump changes (changed defaults) catches my attention to have caused this. Anyone had similar issues? Any tips and ideas on what to check?

1 0

Re: squid 19.2.5 QE validation status
by Venky Shankar 09 Jul '26

09 Jul '26

Hi Yuri, fs approved. failures are https://tracker.ceph.com/projects/cephfs/wiki/Squid#Squid-v1925 On Wed, Jul 8, 2026 at 4:50 PM Venky Shankar via dev <dev(a)ceph.io> wrote: > > Hey Yuri, > > On Wed, Jul 8, 2026 at 1:16 AM Yuri Weinstein via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > Hi everyone, > > > > We are currently awaiting the fs review and approval from Venky. > > The run mostly looks good. There were some infra failures for which I > am reruning the failed tests > > https://pulpito.ceph.com/vshankar-2026-07-08_07:40:18-fs-squid-release-dist… > > Once the run finishes, I will post my approval. > > > > > Additionally, please review the Release Notes here: > > https://github.com/ceph/ceph/pull/69886 > > > > Best regards, > > Yuri Weinstein > > > > On Mon, Jun 29, 2026 at 7:26 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > > Details of this release are summarized here: > > > > > > https://tracker.ceph.com/issues/76983#note-2 > > > > > > Release Notes - TBD > > > LRC upgrade - N/A > > > > > > Note: please reply to this email with comments/approvals and also add your > > > comments to the tracker above. > > > > > > Seeking approvals/reviews for: > > > > > > smoke: PASSED > > > > > > rados: Radek, Laura, Nitzan, Afreen, Travis > > > rgw: Adam Emerson > > > fs: Venky > > > orch/ceph-volume: Redouane & Guillaume > > > rbd: Ilya > > > krbd: Ilya > > > > > > upgrade: Laura, Patrick (see https://tracker.ceph.com/issues/77569, > > > https://github.com/ceph/ceph/pull/69791) > > > > > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > > > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > > > > > perf-basic: PASSED > > > crimson-rados: N/A > > > powercycle: PASSED > > > > > > TIA > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > -- > Cheers, > Venky > -- Cheers, Venky

1 0

Re: Adding and removing monitors/Impact on consuming systems
by Eugen Block 08 Jul '26

08 Jul '26

Hi, if you haven't yet, check out the docs [0], [1]. It's recommended to first add new MONs, see if they work and then remove the old ones (as you already mentioned yourself). I recommend to turn on this cephadm health check (disabled by default): ceph config set mgr mgr/cephadm/config_checks_enabled true Especially "network_missing" can be valuable here, it will check if your additional hosts have correct network settings before you try to add daemons. You can turn off other checks if you don't need them. Regards, Eugen [0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a… [1] https://docs.ceph.com/en/latest/cephadm/services/mon/#moving-monitors-to-a-… Zitat von Edward R Huyer via ceph-users <ceph-users(a)ceph.io>: > I have a fairly run-of-the-mill cephadm managed/containerized 20.2.1 > cluster. Not hyperconverged or anything like that. It's currently > exporting RBD to a Proxmox VE 8.4 cluster (QEMU virtualization), and > CephFS to a couple RHEL 8.x servers via kernel mounts. > > I need to migrate the whole thing, including the monitors, to new > hardware. My question is, can I rely on the QEMU processes and > kernel mounts to correctly update their internal monitor maps as I > add new monitors and remove old monitors, or will I suddenly reach a > point where the consumers can't find the monitors they started up > with and lose their minds? (I know I'll also need to update the > static configs.) > > I'm pretty sure it should Just Work so long as I don't do something > dumb and lose quorum, but there's a big difference between "should" > and "actually does", and I haven't been able to find any writeups of > people actually doing it, so I'm asking here. Is this likely to > work? Is this known to not work for some reason (e.g., the kernel > mounts or QEMU processes never actually update their mon maps after > startup)? > > TIA! > > ----- > Edward Huyer > Golisano College of Computing and Information Sciences > Rochester Institute of Technology > Golisano 70-2373 > 152 Lomb Memorial Drive > Rochester, NY 14623 > 585-475-6651 > erhvks(a)rit.edu<mailto:erhvks@rit.edu> > > Obligatory Legalese: > The information transmitted, including attachments, is intended only > for the person(s) or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance > upon this information by persons or entities other than the intended > recipient is prohibited. If you received this in error, please > contact the sender and destroy any copies of this information. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Adding and removing monitors/Impact on consuming systems
by Edward R Huyer 08 Jul '26

08 Jul '26

Awesome, thank you! I knew it should work, but the explosion if it didn't would be big enough I wanted to see if someone could verify. Fortunately I have the hardware installed side by side, so I think I'll add all the new mons, then remove the old ones. Thanks again! > -----Original Message----- > From: Phillip Schichtel via ceph-users <ceph-users(a)ceph.io> > Sent: Wednesday, July 8, 2026 10:28 AM > To: Edward R Huyer <erhvks(a)rit.edu>; ceph-users(a)ceph.io > Subject: [ceph-users] Re: Adding and removing monitors/Impact on > consuming systems > > Hi Edward, > > I'm operating a fairly similar setup. I have done a similar migration to what you > described on a 19.x cluster: Migrated all hosts from an old HPE generation to a > newer one. I added one of the new hosts to the cluster, migrated a mon, mgr, > mds, rgw and osds, propagated the new IPs to firewalls and clients and > eventually removed the old host. I repeated that for the remaining hosts. Due > to a lack of rack space I had to do this host by host. > > The VMs and cephfs mounts did not experience issues during the migration > (both Debian and RHEL machines). > > ~ Phillip > > On Wed, 2026-07-08 at 13:58 +0000, Edward R Huyer via ceph-users wrote: > > I have a fairly run-of-the-mill cephadm managed/containerized 20.2.1 > > cluster. Not hyperconverged or anything like that. It's currently > > exporting RBD to a Proxmox VE 8.4 cluster (QEMU virtualization), and > > CephFS to a couple RHEL 8.x servers via kernel mounts. > > > > I need to migrate the whole thing, including the monitors, to new > > hardware. My question is, can I rely on the QEMU processes and kernel > > mounts to correctly update their internal monitor maps as I add new > > monitors and remove old monitors, or will I suddenly reach a point > > where the consumers can't find the monitors they started up with and > > lose their minds? (I know I'll also need to update the static > > configs.) > > > > I'm pretty sure it should Just Work so long as I don't do something > > dumb and lose quorum, but there's a big difference between "should" > > and "actually does", and I haven't been able to find any writeups of > > people actually doing it, so I'm asking here. Is this likely to work? > > Is this known to not work for some reason (e.g., the kernel mounts or > > QEMU processes never actually update their mon maps after startup)? > > > > TIA! > > > > ----- > > Edward Huyer > > Golisano College of Computing and Information Sciences Rochester > > Institute of Technology Golisano 70-2373 > > 152 Lomb Memorial Drive > > Rochester, NY 14623 > > 585-475-6651 > > erhvks(a)rit.edu<mailto:erhvks@rit.edu> > > > > Obligatory Legalese: > > The information transmitted, including attachments, is intended only > > for the person(s) or entity to which it is addressed and may contain > > confidential and/or privileged material. Any review, retransmission, > > dissemination or other use of, or taking of any action in reliance > > upon this information by persons or entities other than the intended > > recipient is prohibited. If you received this in error, please contact > > the sender and destroy any copies of this information. > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to > ceph-users-leave(a)ceph.io

1 0

Re: Adding and removing monitors/Impact on consuming systems
by Phillip Schichtel 08 Jul '26

08 Jul '26

Hi Edward, I'm operating a fairly similar setup. I have done a similar migration to what you described on a 19.x cluster: Migrated all hosts from an old HPE generation to a newer one. I added one of the new hosts to the cluster, migrated a mon, mgr, mds, rgw and osds, propagated the new IPs to firewalls and clients and eventually removed the old host. I repeated that for the remaining hosts. Due to a lack of rack space I had to do this host by host. The VMs and cephfs mounts did not experience issues during the migration (both Debian and RHEL machines). ~ Phillip On Wed, 2026-07-08 at 13:58 +0000, Edward R Huyer via ceph-users wrote: > I have a fairly run-of-the-mill cephadm managed/containerized 20.2.1 > cluster. Not hyperconverged or anything like that. It's currently > exporting RBD to a Proxmox VE 8.4 cluster (QEMU virtualization), and > CephFS to a couple RHEL 8.x servers via kernel mounts. > > I need to migrate the whole thing, including the monitors, to new > hardware. My question is, can I rely on the QEMU processes and > kernel mounts to correctly update their internal monitor maps as I > add new monitors and remove old monitors, or will I suddenly reach a > point where the consumers can't find the monitors they started up > with and lose their minds? (I know I'll also need to update the > static configs.) > > I'm pretty sure it should Just Work so long as I don't do something > dumb and lose quorum, but there's a big difference between "should" > and "actually does", and I haven't been able to find any writeups of > people actually doing it, so I'm asking here. Is this likely to > work? Is this known to not work for some reason (e.g., the kernel > mounts or QEMU processes never actually update their mon maps after > startup)? > > TIA! > > ----- > Edward Huyer > Golisano College of Computing and Information Sciences > Rochester Institute of Technology > Golisano 70-2373 > 152 Lomb Memorial Drive > Rochester, NY 14623 > 585-475-6651 > erhvks(a)rit.edu<mailto:erhvks@rit.edu> > > Obligatory Legalese: > The information transmitted, including attachments, is intended only > for the person(s) or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance > upon this information by persons or entities other than the intended > recipient is prohibited. If you received this in error, please > contact the sender and destroy any copies of this information. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Adding and removing monitors/Impact on consuming systems
by Edward R Huyer 08 Jul '26

08 Jul '26

I have a fairly run-of-the-mill cephadm managed/containerized 20.2.1 cluster. Not hyperconverged or anything like that. It's currently exporting RBD to a Proxmox VE 8.4 cluster (QEMU virtualization), and CephFS to a couple RHEL 8.x servers via kernel mounts. I need to migrate the whole thing, including the monitors, to new hardware. My question is, can I rely on the QEMU processes and kernel mounts to correctly update their internal monitor maps as I add new monitors and remove old monitors, or will I suddenly reach a point where the consumers can't find the monitors they started up with and lose their minds? (I know I'll also need to update the static configs.) I'm pretty sure it should Just Work so long as I don't do something dumb and lose quorum, but there's a big difference between "should" and "actually does", and I haven't been able to find any writeups of people actually doing it, so I'm asking here. Is this likely to work? Is this known to not work for some reason (e.g., the kernel mounts or QEMU processes never actually update their mon maps after startup)? TIA! ----- Edward Huyer Golisano College of Computing and Information Sciences Rochester Institute of Technology Golisano 70-2373 152 Lomb Memorial Drive Rochester, NY 14623 585-475-6651 erhvks(a)rit.edu<mailto:erhvks@rit.edu> Obligatory Legalese: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information.

1 0

Re: squid 19.2.5 QE validation status
by Venky Shankar 08 Jul '26

08 Jul '26

Hey Yuri, On Wed, Jul 8, 2026 at 1:16 AM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi everyone, > > We are currently awaiting the fs review and approval from Venky. The run mostly looks good. There were some infra failures for which I am reruning the failed tests https://pulpito.ceph.com/vshankar-2026-07-08_07:40:18-fs-squid-release-dist… Once the run finishes, I will post my approval. > > Additionally, please review the Release Notes here: > https://github.com/ceph/ceph/pull/69886 > > Best regards, > Yuri Weinstein > > On Mon, Jun 29, 2026 at 7:26 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/76983#note-2 > > > > Release Notes - TBD > > LRC upgrade - N/A > > > > Note: please reply to this email with comments/approvals and also add your > > comments to the tracker above. > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Nitzan, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Redouane & Guillaume > > rbd: Ilya > > krbd: Ilya > > > > upgrade: Laura, Patrick (see https://tracker.ceph.com/issues/77569, > > https://github.com/ceph/ceph/pull/69791) > > > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Cheers, Venky

1 0

squid 19.2.5 QE validation status
by Yuri Weinstein 07 Jul '26

07 Jul '26

Details of this release are summarized here: https://tracker.ceph.com/issues/76983#note-2 Release Notes - TBD LRC upgrade - N/A Note: please reply to this email with comments/approvals and also add your comments to the tracker above. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Nitzan, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade: Laura, Patrick (see https://tracker.ceph.com/issues/77569, https://github.com/ceph/ceph/pull/69791) upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 1

Ceph Tech Talk July Reminder
by Anthony Middleton 07 Jul '26

07 Jul '26

Hey Ceph Friends, Please join us tomorrow, July 8, at 10:00 AM PDT / 17:00 UTC for the next Ceph Tech Talk: "*For Whom the Doorbell Tolls: GPU-Initiated Key-Value Storage on Ceph*" Together, we will learn how rados-nkv enables AMD GPUs to retrieve key-value data directly from Ceph using the NVMe KV command set. Featuring Kyle Bader (IBM) and Stephen Bates (AMD). RSVP here: https://www.linkedin.com/events/7473007902295773184/?viewAsMember=true If you have a topic in mind for a future tech talk, reach out to me with details. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: umbrella 21.1.0 RC QE validation status
by Brad Hubbard 07 Jul '26

07 Jul '26

On Tue, Jul 7, 2026 at 1:03 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/76757 > > Release Notes - TBD > LRC upgrade - TBD > > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > Seeking approvals/reviews for: > > smoke: Radek, Laura, Nitzan > > rados: Radek, Laura, Notzan, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > > nvmeof: PASSED > > upgrade: Laura, Patrick > > upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: TBD ( > https://github.com/ceph/ceph/pull/69888) > upgrade-clients:client-upgrade-squid-umbrella-umbrella: TBD ( > https://github.com/ceph/ceph/pull/69887) > > perf-basic: PASSED > > crimson-rados: Matan > > powercycle: Brad > The problem does not appear to be in the powercycle test but seems to expose an issue in the LibRadosIoECPPParamCombination/LibRadosIoECPP tests. I created a tracker, https://tracker.ceph.com/issues/77970 and asked Alex for comment. TIA > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Cheers, Brad

1 0

CSC Minutes 2026/07/06
by Ernesto Puerta 06 Jul '26

06 Jul '26

Short meeting today: - [Dan] Waiting for the results of the ongoing motions (already closed). - [KyleB] Adopt rados-nkv <https://github.com/mmgaggle/rados-nkv> under the Ceph org (https://tracker.ceph.com/issues/77960) It'll be released as a container, potentially merging with the existing ceph-nvmeof container. - [Neha] Cephalocon working group <https://pad.ceph.com/p/cephalocon-working-group> (also <https://tracker.ceph.com/issues/77296>) starting today (interested parties are encouraged to join). - [Laura] Sepia Lab down due to storms (AdamK working on that; updates available at #sepia Slack channel). Kind Regards, Ernesto

1 0

Re: An issue during expanding an cluster - one osd getting close to nearfull
by Robert Lukan 05 Jul '26

05 Jul '26

Thank you Eugen and Anthony for your comments. Expanding a cluster with one host with the same amount of disks as in other nodes worked perfectly fine. We also (unwantingly) made a test with expanding with one disk(osd) and we have crossed the near-full state. Also thank you for sharing some more details, it is really informative to get those insights into more corner case scenarios. I wish that Ceph documentation would have a few more guidelines, something similar to what Clyso has put together. Regards, Robert On Wed, 1 Jul 2026 at 15:07, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > Hi, > > I would try to avoid adding a host with only few OSDs if other hosts > are equipped differently. Although I haven't checked in a while I > don't expect the behavior to have changed in recent versions. > > So here's my theory: If you add a new host with only one or two OSDs, > Ceph can assign many more PGs than average to the first up OSD in > order to comply with failure domain (probably host). Same goes for the > last OSD during shut down. We had a related case four years ago when > we tried to rebuild OSD nodes in a customer cluster. Since the > customer cluster was still on default values for mon_max_pg_per_osd > and osd_max_pg_per_osd_hard_ratio, the (too many) assigned PGs led to > inactive PGs: > > > maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750 > > When more OSDs were created on that host (ceph-volume does that > sequentially), the issue resolved eventually, except for some inactive > PGs, failing the MGR resolved those as well. Since you have around 360 > PGs per OSD in average, I assume you increased mon_max_pg_per_osd > (default 250), maybe also osd_max_pg_per_osd_hard_ratio (default 3), > you most likely dodged the "inactive PG" bullet. > > So my recommendation would be to rather add the hosts with all OSDs at > the same time but with crush weight 0. That way you have time to > inspect the outcome (for example with dedicated DB/WAL devices you can > check if they were created according to the specs). Then reweight the > OSDs to the correct value, some users do that gradually to prevent an > overload. > > Note the recent discussion about osd heartbeats when adding multiple > hosts at once: > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHVPS247M4E… > > Regards, > Eugen > > Zitat von Robert Lukan via ceph-users <ceph-users(a)ceph.io>: > > > Hi everyone, > > > > We are currently in the middle of migrating our infrastructure to a > > Proxmox/Ceph stack and ran into a concerning edge case while expanding > the > > cluster. I am hoping to get some advice on the safest way to handle this > > without risking a nearfull OSD. > > > > *The Topology & Situation:* > > > > *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs. > > > > *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and > > osd.41). > > > > *Cluster Average:* ~360 PGs per OSD, ~60% utilization. > > > > *Ceph Version:* Squid 19.2.3 > > > > The last time we were expanding the cluster, we had issues when expanding > > it with many OSDs at the same time(it was a memory problem on a host), so > > we decided to be more careful and add 2 disks at the time. > > > > During the backfill phase, we noticed osd.40 catching significantly more > > PGs than osd.41. We paused the migration by setting the nobackfill flag > to > > prevent a potential nearfull situation. > > > > *The Current State (ceph osd df):* > > > > osd.40: 423 PGs, 72.56% used, > > > > osd.41: 307 PGs, 51.87% used, > > > > We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we > > have paused the backfill. > > > > If we were to unset nobackfill, the incoming data for those 218 PGs would > > risk pushing osd.40 past the 85% nearfull threshold. However, our > > balancer refused to generate any optimization plans. It reports: > > "optimize_result": > > "Unable to find further optimization... or distribution is already > perfect" > > > > We "understand" why this is happening: the balancer is evaluating only > when > > backfilling is done. However, it is completely ignoring the transient > acting > > state, which was heavily overloading osd.40 > > > > *Our Temporary Workaround:* To prevent the potential issue, we have > > temporarily set the reweight of osd.40 to 0 (effectively draining it out) > > so we can safely lift the nobackfill flag and let the cluster settle > > without hitting capacity limits. > > > > *My Questions for the Community:* > > > > What is the standard best practice for handling this "small bucket" > > transient imbalance when adding a small number of OSDs to a new host? Or > is > > this just a bad idea in general ? > > > > Is there a way to force the built-in upmap balancer to respect the > > transient/acting capacity of an OSD so it doesn't allow a drive to go > > towards 85% during a migration? I understand we were not there, but being > > 12 % away was "close" enough. > > > > One of the ideas was to reweight the offending OSD, to remove a few PGs, > so > > it would not be so close to nearfull situation. But this approach is > > tedious and requires "baby-sitting". > > > > I guess this might be a good candidate for upmap remap script, but we > were > > not "brave" enough to run in it a production environment the first time. > > > > Thanks, > > > > Robert > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

New download.ceph.com package/repo signing key
by Dan Mick 04 Jul '26

04 Jul '26

Hi all. Recent releases (el10 in particular) started to complain about a SHA-1 algorithm in use for our release key (https://download.ceph.com/keys/release.asc) I have re-signed the portion of the key that was using the obsolete algorithm and the new key is available at the same location. You can tell the new key because it's 1660 bytes, and it has no "Version:" string in the text. Also, it will cleanly import into el10 rpm, whereas the original does not. This will fix el10 rpm's failure to verify the signature on .rpm files: error: Verifying a signature using certificate 08B73419AC32B4E966C1A330E84AC2C0460F3994 (Ceph.com (release key) <security(a)ceph.com>): 1. Certificate E84AC2C0460F3994 invalid: policy violation because: No binding signature at time 2025-10-03T17:18:24Z If you experience such errors, download the new key from https://tracker.ceph.com/issues/74861 and use rpm --import to store it in rpm's keyring. Details are in https://tracker.ceph.com/issues/74861. Feel free to email with questions. tl;dr: update your release.asc Ceph package-signing key.

1 0

Re: CephFS large amounts of dark data
by Alex Gorbachev 03 Jul '26

03 Jul '26

I don't deal with CephFS much, but on RBD I had an issue like that, and it was because the filesystem needed fstrim. -- Alex Gorbachev ISS/Storcium On Fri, Jul 3, 2026 at 3:48 AM Andrej Filipčič via ceph-users < ceph-users(a)ceph.io> wrote: > On 3. 7. 2026 02:05, Anthony D'Atri via ceph-users wrote: > > A couple of ideas: > > > > Do you have osd_recovery_sleep_* set? > > osd_recovery_sleep_hdd 0.100000 > > osd_delete_sleep_*? > not that one > > Do you have any pools in the middle of pg_num adjustments, such that > pg_num != pgp_num and a pg_num_target is reported? > no, that one is fine, we have autoscaling disabled. > > Best, > Andrej > > > >> On Jul 2, 2026, at 2:52 PM, Andras Pataki via ceph-users < > ceph-users(a)ceph.io> wrote: > >> > >> Hi, > >> > >> I have also seen this on a recent large scale/parallel test I can on a > test setup. After creating about a hundred million files, then removing > all of them - there were thousands of objects left in the cephfs data pool > corresponding to inodes that no longer existed. I.e. it looks like the MDS > is "leaking" data - does not correctly delete all objects corresponding to > files in the purge queue. We have never used any snapshots on this file > system, so it definitely isn't about snapshots holding data. Also, we > increased the purge queue settings (filer_max_purge_ops, mds_max_purge_* > and friends) and definitely waited long enough for all deletes to be > processed (the objects were there weeks after the file system was emptied). > This was on a squid 19.2.3 cluster. > >> In the tests that lead to this, both the file creation and deletions > were done in parallel using a few dozen clients - stressing the MDS for > sure. We had no crashes/problems on the cluster, we never had to do any > data recovery steps, i.e. the MDS appeared to work fine through the tests. > Also, this was a single MDS cluster, i.e. the problem isn't related to > subtrees moving across MDS's for example. The problem became obvious since > after removing all data from cephfs, we were expecting the data pools to be > empty, but they weren't. There were two pools, a primary triple replicated > one and an erasure coded one. Both of them had stray objects. > >> > >> I didn't pursue this further since I wasn't sure what useful > information I could gather for a bug report - but it is certainly a curious > observation that perhaps large, long living cephfs clusters might have > significant space tied up in these objects that should have been removed > but weren't. Short of a full scan of all objects and matching them to > inodes - it is hard to tell how much even. > >> > >> Andras > >> > >> > >>> On 7/2/26 12:17 PM, Andrej Filipčič via ceph-users wrote: > >>> > >>> So, to follow up on this, I did some further investigation. > >>> > >>> Checking for write amplification, I have copied 250TB of a mix of > small and large files (20M of them), and the stored space on EC pool > matched what was expected from actual data size to a few %. So EC overhead > was not really a factor. Also, after the removal of this data set, the > stored space was recovered as expected. > >>> > >>> The I checked the full dump of cephfs and compared it to list of all > objects in the EC pool as follows: > >>> > >>> rados -p cephfs_data_echdd ls > echdd.objectlist > >>> > >>> find /ceph/ -printf "%i %p\n" > cephfs.inodes > >>> > >>> this took several hours, while in the meantime, writing and removing > of data to cephfs was relatively low (few MB/s), so the impact of new > objects and files should have been minimal. > >>> > >>> Then I selected all the objects of the form > >>> 1007b28abae.00000000 > >>> 100cce97f6d.00000000 > >>> 10067733861.00000000 > >>> 100cc4646aa.00000000 > >>> 200044d5c07.00000000 > >>> ... > >>> > >>> and checked if they match the files in the cephfs.inodes list > >>> > >>> ~3M of *.00000000 objects do not have the corresponding inode in the > cephfs.inodes > >>> > >>> > >>> I did "rados stat" of ~70k of these objects (still running for all of > them), and almost all have timestamps from March and April these year. On > 4th of May I remember I have increased the mds purge queue values to speed > up the removal of files since OSDs were filling up too quickly, and since > then it seems the objects are not left uncleaned any more. > >>> > >>> With "rados getxattr objectid parent" I have checked several of them, > and they all belong to two very active projects which typically write files > with few GB/s all the time (and remove as well so space usage is not > increasing). The objects I have checked belonged to files that were > removed in cephfs, the projects have a separate file catalog which is > consistent with cephfs contents. > >>> > >>> So, I do not understand why so many objects were left unremoved in EC > pool in the period of 2 months, but at least, 3M uncleaned files explains > 1.5PB of dark data by quick estimate. > >>> > >>> We also scrubbed cephfs root and ~mdsdir several times and no > leftovers to remove were found. > >>> > >>> I hope the problem is gone now, but I would still like a good advice > on how to proceed with the cleanup. I see these options: > >>> > >>> 1) remove the unmatched objects directly from EC pool with "rados > rm". But this might have undesired side effects or corruption. > >>> > >>> 2) create a new EC pool and migrate all the data there by copying > files in filesystem, and then destroy the old pool. > >>> > >>> 3) create a new filesystem with new EC pools and migrate the files. > >>> > >>> 4) run some advanced MDS disaster recovery procedure > (cephfs-data-scan), but this requires offline FS and I do not want to > recover old files from existing unmatched objects. Anyway, cephfs seems to > be healthy now. > >>> > >>> Any good ideas? > >>> > >>> Best, > >>> Andrej > >>> > >>> > >>> On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote: > >>>>> ). > >>>>>>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with > 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used > consistent with 8+3 EC. > >>>>>>> The layout for most files: > >>>>>>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 > object_size=16777216 pool=cephfs_data_echdd" > >>>>>>> > >>>>>>> But there is 1.4PiB discrepancy between the pool and the filesystem > >>>> Do you have scrubs enabled? Which if any non-default config options > do you have set? Any undersized or degraded or backfilling PGs? > >>>> > >>>> Which Ceph release? Do you have a sizable fraction of small files? > If you’re running Squid or earlier or don’t have EC optimizations enabled, > even a tiny file will allocate a multiple of 11*16=176 KB. An 129KB file > will consume 352KB, etc. If I understand those layout options correctly. > >>>> > >>>> If that’s what’s going on, going to Tentacle with EC optimizations > would gain you some efficiency for files newly [re]written. You could also > migrate small files to a replicated pool. > >>>> > >>>> > >>>>>>> which I cannot explain and I suspect there are a lot of orphan > objects there. I have run mds scrub on / and ~mdsdir as well. There is some > mds damage on some old small files (~400 files), which I do not think it's > relevant here. > >>>>>>> > >>>>>> Hello, > >>>>>> We had a similar issue last year with a group of users that created > and removed files at a very high rate. > >>>>>> > >>>>>> Have you read > https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/cephfs/*cephfs-p… > ? > >>>>>> > >>>>>> We increased the purge rate parameters (very) aggressively to get > back to a comfortable situation (i.e. not a pool w/ near full warnings). > >>>>>> > >>>>>> > >>>>>> Loïc. > >>>>> -- > >>>>> _____________________________________________________________ > >>>>> prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si > >>>>> Department of Experimental High Energy Physics - F9 > >>>>> Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > >>>>> SI-1001 Ljubljana, Slovenia > >>>>> Tel.: +386-1-477-3674 Fax: +386-1-477-3166 > >>>>> ------------------------------------------------------------- > >>>>> _______________________________________________ > >>>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > -- > _____________________________________________________________ > prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si > Department of Experimental High Energy Physics - F9 > Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > SI-1001 Ljubljana, Slovenia > Tel.: +386-1-477-3674 Fax: +386-1-477-3166 > ------------------------------------------------------------- > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Grafana dashboards warnings after upgrade to Squid 19.2.4
by Anthony D'Atri 03 Jul '26

03 Jul '26

[ I've left off dev@ from this reply, as it doesn't seem necessarily like a dev issue ] Is it possible that your cluster is overriding the default container images, assuming that you're running containerized with cephadm? root@squid1:~# ceph config dump | grep container_image container_image XX mgr/cephadm/container_image_alertmanager YY mgr/cephadm/container_image_base ZZ mgr/cephadm/container_image_elasticsearch QQ mgr/cephadm/container_image_grafana .. mgr/cephadm/container_image_haproxy .. mgr/cephadm/container_image_jaeger_agent .. mgr/cephadm/container_image_jaeger_collector .. mgr/cephadm/container_image_jaeger_query .. mgr/cephadm/container_image_keepalived .. mgr/cephadm/container_image_loki .. mgr/cephadm/container_image_node_exporter .. mgr/cephadm/container_image_nvmeof .. mgr/cephadm/container_image_prometheus .. mgr/cephadm/container_image_promtail .. mgr/cephadm/container_image_samba .. mgr/cephadm/container_image_snmp_gateway .. > On Jul 1, 2026, at 6:53 AM, Sake via ceph-users <ceph-users(a)ceph.io> wrote: > > Probably something not going correctly, because I just upgraded to 19.2.4 and Grafana is using the grafana image (in the past this was ceph-grafana) version 12.3.1. > > > On 1 July 2026 11:13:02 CEST, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: >> Hi, >> >> I just upgraded our production cluster to 19.2.4 (from 19.2.3), there were no issues reported by our users, so all good. But I noticed that many graphs now show different warnings or strange labels: >> >> - "This panel requires Angular (deprecated)." on multiple graphs >> - "PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "ceph_pool_wr" (1:6)" also on multiple graphs for different metrics >> - "OSD Capacity Utilization" just shows: "RangeError: invalid array length" >> - The "Pool Overview" graph has column names like "Value #F", "Value #G" and so on. And the values within don't make sense either. >> >> I have uploaded screenshots: >> >> https://www.pasteboard.co/A34Zcy9cqHe5.png >> https://www.pasteboard.co/F_vkpeBjGGWH.png >> https://www.pasteboard.co/mkCtlBN3zS6v.png >> https://www.pasteboard.co/T_ByD_gCpslA.png >> >> I compared the prometheus and grafana versions we have currently running with a fresh 19.2.4 bootstrapped cluster and the versions are identical: >> >> ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' >> 10.4.0 >> >> ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' >> 2.51.0 >> >> Is there anything I'm missing? >> >> Thanks! >> Eugen >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: CephFS large amounts of dark data
by Andrej Filipčič 03 Jul '26

03 Jul '26

On 3. 7. 2026 02:05, Anthony D'Atri via ceph-users wrote: > A couple of ideas: > > Do you have osd_recovery_sleep_* set? osd_recovery_sleep_hdd 0.100000 > osd_delete_sleep_*? not that one > Do you have any pools in the middle of pg_num adjustments, such that pg_num != pgp_num and a pg_num_target is reported? no, that one is fine, we have autoscaling disabled. Best, Andrej > >> On Jul 2, 2026, at 2:52 PM, Andras Pataki via ceph-users <ceph-users(a)ceph.io> wrote: >> >> Hi, >> >> I have also seen this on a recent large scale/parallel test I can on a test setup. After creating about a hundred million files, then removing all of them - there were thousands of objects left in the cephfs data pool corresponding to inodes that no longer existed. I.e. it looks like the MDS is "leaking" data - does not correctly delete all objects corresponding to files in the purge queue. We have never used any snapshots on this file system, so it definitely isn't about snapshots holding data. Also, we increased the purge queue settings (filer_max_purge_ops, mds_max_purge_* and friends) and definitely waited long enough for all deletes to be processed (the objects were there weeks after the file system was emptied). This was on a squid 19.2.3 cluster. >> In the tests that lead to this, both the file creation and deletions were done in parallel using a few dozen clients - stressing the MDS for sure. We had no crashes/problems on the cluster, we never had to do any data recovery steps, i.e. the MDS appeared to work fine through the tests. Also, this was a single MDS cluster, i.e. the problem isn't related to subtrees moving across MDS's for example. The problem became obvious since after removing all data from cephfs, we were expecting the data pools to be empty, but they weren't. There were two pools, a primary triple replicated one and an erasure coded one. Both of them had stray objects. >> >> I didn't pursue this further since I wasn't sure what useful information I could gather for a bug report - but it is certainly a curious observation that perhaps large, long living cephfs clusters might have significant space tied up in these objects that should have been removed but weren't. Short of a full scan of all objects and matching them to inodes - it is hard to tell how much even. >> >> Andras >> >> >>> On 7/2/26 12:17 PM, Andrej Filipčič via ceph-users wrote: >>> >>> So, to follow up on this, I did some further investigation. >>> >>> Checking for write amplification, I have copied 250TB of a mix of small and large files (20M of them), and the stored space on EC pool matched what was expected from actual data size to a few %. So EC overhead was not really a factor. Also, after the removal of this data set, the stored space was recovered as expected. >>> >>> The I checked the full dump of cephfs and compared it to list of all objects in the EC pool as follows: >>> >>> rados -p cephfs_data_echdd ls > echdd.objectlist >>> >>> find /ceph/ -printf "%i %p\n" > cephfs.inodes >>> >>> this took several hours, while in the meantime, writing and removing of data to cephfs was relatively low (few MB/s), so the impact of new objects and files should have been minimal. >>> >>> Then I selected all the objects of the form >>> 1007b28abae.00000000 >>> 100cce97f6d.00000000 >>> 10067733861.00000000 >>> 100cc4646aa.00000000 >>> 200044d5c07.00000000 >>> ... >>> >>> and checked if they match the files in the cephfs.inodes list >>> >>> ~3M of *.00000000 objects do not have the corresponding inode in the cephfs.inodes >>> >>> >>> I did "rados stat" of ~70k of these objects (still running for all of them), and almost all have timestamps from March and April these year. On 4th of May I remember I have increased the mds purge queue values to speed up the removal of files since OSDs were filling up too quickly, and since then it seems the objects are not left uncleaned any more. >>> >>> With "rados getxattr objectid parent" I have checked several of them, and they all belong to two very active projects which typically write files with few GB/s all the time (and remove as well so space usage is not increasing). The objects I have checked belonged to files that were removed in cephfs, the projects have a separate file catalog which is consistent with cephfs contents. >>> >>> So, I do not understand why so many objects were left unremoved in EC pool in the period of 2 months, but at least, 3M uncleaned files explains 1.5PB of dark data by quick estimate. >>> >>> We also scrubbed cephfs root and ~mdsdir several times and no leftovers to remove were found. >>> >>> I hope the problem is gone now, but I would still like a good advice on how to proceed with the cleanup. I see these options: >>> >>> 1) remove the unmatched objects directly from EC pool with "rados rm". But this might have undesired side effects or corruption. >>> >>> 2) create a new EC pool and migrate all the data there by copying files in filesystem, and then destroy the old pool. >>> >>> 3) create a new filesystem with new EC pools and migrate the files. >>> >>> 4) run some advanced MDS disaster recovery procedure (cephfs-data-scan), but this requires offline FS and I do not want to recover old files from existing unmatched objects. Anyway, cephfs seems to be healthy now. >>> >>> Any good ideas? >>> >>> Best, >>> Andrej >>> >>> >>> On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote: >>>>> ). >>>>>>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3 EC. >>>>>>> The layout for most files: >>>>>>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216 pool=cephfs_data_echdd" >>>>>>> >>>>>>> But there is 1.4PiB discrepancy between the pool and the filesystem >>>> Do you have scrubs enabled? Which if any non-default config options do you have set? Any undersized or degraded or backfilling PGs? >>>> >>>> Which Ceph release? Do you have a sizable fraction of small files? If you’re running Squid or earlier or don’t have EC optimizations enabled, even a tiny file will allocate a multiple of 11*16=176 KB. An 129KB file will consume 352KB, etc. If I understand those layout options correctly. >>>> >>>> If that’s what’s going on, going to Tentacle with EC optimizations would gain you some efficiency for files newly [re]written. You could also migrate small files to a replicated pool. >>>> >>>> >>>>>>> which I cannot explain and I suspect there are a lot of orphan objects there. I have run mds scrub on / and ~mdsdir as well. There is some mds damage on some old small files (~400 files), which I do not think it's relevant here. >>>>>>> >>>>>> Hello, >>>>>> We had a similar issue last year with a group of users that created and removed files at a very high rate. >>>>>> >>>>>> Have you read https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/cephfs/*cephfs-p… ? >>>>>> >>>>>> We increased the purge rate parameters (very) aggressively to get back to a comfortable situation (i.e. not a pool w/ near full warnings). >>>>>> >>>>>> >>>>>> Loïc. >>>>> -- >>>>> _____________________________________________________________ >>>>> prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si >>>>> Department of Experimental High Energy Physics - F9 >>>>> Jozef Stefan Institute, Jamova 39, P.o.Box 3000 >>>>> SI-1001 Ljubljana, Slovenia >>>>> Tel.: +386-1-477-3674 Fax: +386-1-477-3166 >>>>> ------------------------------------------------------------- >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 0

Re: CephFS large amounts of dark data
by Gregory Farnum 03 Jul '26

03 Jul '26

Hmm, this is unfortunate news. I don’t think I’ve heard about anything like this before. Andrej, what Ceph version are you running, and has it changed during this time frame? I would imagine the most likely thing to be our “purge queue” is somehow some times not deleting objects that it marks as removed. But I’ve no idea how that could go wrong. As much information as we can get about stuff like mds restarts during the deletion, single or mult-mds, precise versions, and any pool or file layouts may help narrow things down. (For instance, it may be that the leak only happens if using multiple data pools, or if it was written on a particular version and a later buggy version parses something wrong.) -Greg On Thu, Jul 2, 2026 at 8:53 AM Andras Pataki via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > I have also seen this on a recent large scale/parallel test I can on a > test setup. After creating about a hundred million files, then removing > all of them - there were thousands of objects left in the cephfs data > pool corresponding to inodes that no longer existed. I.e. it looks like > the MDS is "leaking" data - does not correctly delete all objects > corresponding to files in the purge queue. We have never used any > snapshots on this file system, so it definitely isn't about snapshots > holding data. Also, we increased the purge queue settings > (filer_max_purge_ops, mds_max_purge_* and friends) and definitely waited > long enough for all deletes to be processed (the objects were there > weeks after the file system was emptied). This was on a squid 19.2.3 > cluster. > In the tests that lead to this, both the file creation and deletions > were done in parallel using a few dozen clients - stressing the MDS for > sure. We had no crashes/problems on the cluster, we never had to do any > data recovery steps, i.e. the MDS appeared to work fine through the > tests. Also, this was a single MDS cluster, i.e. the problem isn't > related to subtrees moving across MDS's for example. The problem became > obvious since after removing all data from cephfs, we were expecting the > data pools to be empty, but they weren't. There were two pools, a > primary triple replicated one and an erasure coded one. Both of them > had stray objects. > > I didn't pursue this further since I wasn't sure what useful information > I could gather for a bug report - but it is certainly a curious > observation that perhaps large, long living cephfs clusters might have > significant space tied up in these objects that should have been removed > but weren't. Short of a full scan of all objects and matching them to > inodes - it is hard to tell how much even. > > Andras > > > On 7/2/26 12:17 PM, Andrej Filipčič via ceph-users wrote: > > > > So, to follow up on this, I did some further investigation. > > > > Checking for write amplification, I have copied 250TB of a mix of > > small and large files (20M of them), and the stored space on EC pool > > matched what was expected from actual data size to a few %. So EC > > overhead was not really a factor. Also, after the removal of this data > > set, the stored space was recovered as expected. > > > > The I checked the full dump of cephfs and compared it to list of all > > objects in the EC pool as follows: > > > > rados -p cephfs_data_echdd ls > echdd.objectlist > > > > find /ceph/ -printf "%i %p\n" > cephfs.inodes > > > > this took several hours, while in the meantime, writing and removing > > of data to cephfs was relatively low (few MB/s), so the impact of new > > objects and files should have been minimal. > > > > Then I selected all the objects of the form > > 1007b28abae.00000000 > > 100cce97f6d.00000000 > > 10067733861.00000000 > > 100cc4646aa.00000000 > > 200044d5c07.00000000 > > ... > > > > and checked if they match the files in the cephfs.inodes list > > > > ~3M of *.00000000 objects do not have the corresponding inode in the > > cephfs.inodes > > > > > > I did "rados stat" of ~70k of these objects (still running for all of > > them), and almost all have timestamps from March and April these year. > > On 4th of May I remember I have increased the mds purge queue values > > to speed up the removal of files since OSDs were filling up too > > quickly, and since then it seems the objects are not left uncleaned > > any more. > > > > With "rados getxattr objectid parent" I have checked several of them, > > and they all belong to two very active projects which typically write > > files with few GB/s all the time (and remove as well so space usage is > > not increasing). The objects I have checked belonged to files that > > were removed in cephfs, the projects have a separate file catalog > > which is consistent with cephfs contents. > > > > So, I do not understand why so many objects were left unremoved in EC > > pool in the period of 2 months, but at least, 3M uncleaned files > > explains 1.5PB of dark data by quick estimate. > > > > We also scrubbed cephfs root and ~mdsdir several times and no > > leftovers to remove were found. > > > > I hope the problem is gone now, but I would still like a good advice > > on how to proceed with the cleanup. I see these options: > > > > 1) remove the unmatched objects directly from EC pool with "rados > > rm". But this might have undesired side effects or corruption. > > > > 2) create a new EC pool and migrate all the data there by copying > > files in filesystem, and then destroy the old pool. > > > > 3) create a new filesystem with new EC pools and migrate the files. > > > > 4) run some advanced MDS disaster recovery procedure > > (cephfs-data-scan), but this requires offline FS and I do not want to > > recover old files from existing unmatched objects. Anyway, cephfs > > seems to be healthy now. > > > > Any good ideas? > > > > Best, > > Andrej > > > > > > On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote: > >>> ). > >>>>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with > >>>>> 132M files and 139M rentries. The pool shows 7PiB stored and > >>>>> 9.7PiB used consistent with 8+3 EC. > >>>>> The layout for most files: > >>>>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 > >>>>> object_size=16777216 pool=cephfs_data_echdd" > >>>>> > >>>>> But there is 1.4PiB discrepancy between the pool and the filesystem > >> Do you have scrubs enabled? Which if any non-default config options > >> do you have set? Any undersized or degraded or backfilling PGs? > >> > >> Which Ceph release? Do you have a sizable fraction of small files? > >> If you’re running Squid or earlier or don’t have EC optimizations > >> enabled, even a tiny file will allocate a multiple of 11*16=176 KB. > >> An 129KB file will consume 352KB, etc. If I understand those layout > >> options correctly. > >> > >> If that’s what’s going on, going to Tentacle with EC optimizations > >> would gain you some efficiency for files newly [re]written. You > >> could also migrate small files to a replicated pool. > >> > >> > >>>>> which I cannot explain and I suspect there are a lot of orphan > >>>>> objects there. I have run mds scrub on / and ~mdsdir as well. > >>>>> There is some mds damage on some old small files (~400 files), > >>>>> which I do not think it's relevant here. > >>>>> > >>>> Hello, > >>>> We had a similar issue last year with a group of users that created > >>>> and removed files at a very high rate. > >>>> > >>>> Have you read > >>>> > https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/cephfs/*cephfs-p… > >>>> ? > >>>> > >>>> We increased the purge rate parameters (very) aggressively to get > >>>> back to a comfortable situation (i.e. not a pool w/ near full > >>>> warnings). > >>>> > >>>> > >>>> Loïc. > >>> > >>> -- > >>> _____________________________________________________________ > >>> prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si > >>> Department of Experimental High Energy Physics - F9 > >>> Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > >>> SI-1001 Ljubljana, Slovenia > >>> Tel.: +386-1-477-3674 Fax: +386-1-477-3166 > >>> ------------------------------------------------------------- > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Re: CephFS large amounts of dark data
by Anthony D'Atri 03 Jul '26

03 Jul '26

A couple of ideas: Do you have osd_recovery_sleep_* set? osd_delete_sleep_*? Do you have any pools in the middle of pg_num adjustments, such that pg_num != pgp_num and a pg_num_target is reported? > On Jul 2, 2026, at 2:52 PM, Andras Pataki via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I have also seen this on a recent large scale/parallel test I can on a test setup. After creating about a hundred million files, then removing all of them - there were thousands of objects left in the cephfs data pool corresponding to inodes that no longer existed. I.e. it looks like the MDS is "leaking" data - does not correctly delete all objects corresponding to files in the purge queue. We have never used any snapshots on this file system, so it definitely isn't about snapshots holding data. Also, we increased the purge queue settings (filer_max_purge_ops, mds_max_purge_* and friends) and definitely waited long enough for all deletes to be processed (the objects were there weeks after the file system was emptied). This was on a squid 19.2.3 cluster. > In the tests that lead to this, both the file creation and deletions were done in parallel using a few dozen clients - stressing the MDS for sure. We had no crashes/problems on the cluster, we never had to do any data recovery steps, i.e. the MDS appeared to work fine through the tests. Also, this was a single MDS cluster, i.e. the problem isn't related to subtrees moving across MDS's for example. The problem became obvious since after removing all data from cephfs, we were expecting the data pools to be empty, but they weren't. There were two pools, a primary triple replicated one and an erasure coded one. Both of them had stray objects. > > I didn't pursue this further since I wasn't sure what useful information I could gather for a bug report - but it is certainly a curious observation that perhaps large, long living cephfs clusters might have significant space tied up in these objects that should have been removed but weren't. Short of a full scan of all objects and matching them to inodes - it is hard to tell how much even. > > Andras > > >> On 7/2/26 12:17 PM, Andrej Filipčič via ceph-users wrote: >> >> So, to follow up on this, I did some further investigation. >> >> Checking for write amplification, I have copied 250TB of a mix of small and large files (20M of them), and the stored space on EC pool matched what was expected from actual data size to a few %. So EC overhead was not really a factor. Also, after the removal of this data set, the stored space was recovered as expected. >> >> The I checked the full dump of cephfs and compared it to list of all objects in the EC pool as follows: >> >> rados -p cephfs_data_echdd ls > echdd.objectlist >> >> find /ceph/ -printf "%i %p\n" > cephfs.inodes >> >> this took several hours, while in the meantime, writing and removing of data to cephfs was relatively low (few MB/s), so the impact of new objects and files should have been minimal. >> >> Then I selected all the objects of the form >> 1007b28abae.00000000 >> 100cce97f6d.00000000 >> 10067733861.00000000 >> 100cc4646aa.00000000 >> 200044d5c07.00000000 >> ... >> >> and checked if they match the files in the cephfs.inodes list >> >> ~3M of *.00000000 objects do not have the corresponding inode in the cephfs.inodes >> >> >> I did "rados stat" of ~70k of these objects (still running for all of them), and almost all have timestamps from March and April these year. On 4th of May I remember I have increased the mds purge queue values to speed up the removal of files since OSDs were filling up too quickly, and since then it seems the objects are not left uncleaned any more. >> >> With "rados getxattr objectid parent" I have checked several of them, and they all belong to two very active projects which typically write files with few GB/s all the time (and remove as well so space usage is not increasing). The objects I have checked belonged to files that were removed in cephfs, the projects have a separate file catalog which is consistent with cephfs contents. >> >> So, I do not understand why so many objects were left unremoved in EC pool in the period of 2 months, but at least, 3M uncleaned files explains 1.5PB of dark data by quick estimate. >> >> We also scrubbed cephfs root and ~mdsdir several times and no leftovers to remove were found. >> >> I hope the problem is gone now, but I would still like a good advice on how to proceed with the cleanup. I see these options: >> >> 1) remove the unmatched objects directly from EC pool with "rados rm". But this might have undesired side effects or corruption. >> >> 2) create a new EC pool and migrate all the data there by copying files in filesystem, and then destroy the old pool. >> >> 3) create a new filesystem with new EC pools and migrate the files. >> >> 4) run some advanced MDS disaster recovery procedure (cephfs-data-scan), but this requires offline FS and I do not want to recover old files from existing unmatched objects. Anyway, cephfs seems to be healthy now. >> >> Any good ideas? >> >> Best, >> Andrej >> >> >> On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote: >>>> ). >>>>>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3 EC. >>>>>> The layout for most files: >>>>>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216 pool=cephfs_data_echdd" >>>>>> >>>>>> But there is 1.4PiB discrepancy between the pool and the filesystem >>> Do you have scrubs enabled? Which if any non-default config options do you have set? Any undersized or degraded or backfilling PGs? >>> >>> Which Ceph release? Do you have a sizable fraction of small files? If you’re running Squid or earlier or don’t have EC optimizations enabled, even a tiny file will allocate a multiple of 11*16=176 KB. An 129KB file will consume 352KB, etc. If I understand those layout options correctly. >>> >>> If that’s what’s going on, going to Tentacle with EC optimizations would gain you some efficiency for files newly [re]written. You could also migrate small files to a replicated pool. >>> >>> >>>>>> which I cannot explain and I suspect there are a lot of orphan objects there. I have run mds scrub on / and ~mdsdir as well. There is some mds damage on some old small files (~400 files), which I do not think it's relevant here. >>>>>> >>>>> Hello, >>>>> We had a similar issue last year with a group of users that created and removed files at a very high rate. >>>>> >>>>> Have you read https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/cephfs/*cephfs-p… ? >>>>> >>>>> We increased the purge rate parameters (very) aggressively to get back to a comfortable situation (i.e. not a pool w/ near full warnings). >>>>> >>>>> >>>>> Loïc. >>>> >>>> -- >>>> _____________________________________________________________ >>>> prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si >>>> Department of Experimental High Energy Physics - F9 >>>> Jozef Stefan Institute, Jamova 39, P.o.Box 3000 >>>> SI-1001 Ljubljana, Slovenia >>>> Tel.: +386-1-477-3674 Fax: +386-1-477-3166 >>>> ------------------------------------------------------------- >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS large amounts of dark data
by Andras Pataki 02 Jul '26

02 Jul '26

Hi, I have also seen this on a recent large scale/parallel test I can on a test setup. After creating about a hundred million files, then removing all of them - there were thousands of objects left in the cephfs data pool corresponding to inodes that no longer existed. I.e. it looks like the MDS is "leaking" data - does not correctly delete all objects corresponding to files in the purge queue. We have never used any snapshots on this file system, so it definitely isn't about snapshots holding data. Also, we increased the purge queue settings (filer_max_purge_ops, mds_max_purge_* and friends) and definitely waited long enough for all deletes to be processed (the objects were there weeks after the file system was emptied). This was on a squid 19.2.3 cluster. In the tests that lead to this, both the file creation and deletions were done in parallel using a few dozen clients - stressing the MDS for sure. We had no crashes/problems on the cluster, we never had to do any data recovery steps, i.e. the MDS appeared to work fine through the tests. Also, this was a single MDS cluster, i.e. the problem isn't related to subtrees moving across MDS's for example. The problem became obvious since after removing all data from cephfs, we were expecting the data pools to be empty, but they weren't. There were two pools, a primary triple replicated one and an erasure coded one. Both of them had stray objects. I didn't pursue this further since I wasn't sure what useful information I could gather for a bug report - but it is certainly a curious observation that perhaps large, long living cephfs clusters might have significant space tied up in these objects that should have been removed but weren't. Short of a full scan of all objects and matching them to inodes - it is hard to tell how much even. Andras On 7/2/26 12:17 PM, Andrej Filipčič via ceph-users wrote: > > So, to follow up on this, I did some further investigation. > > Checking for write amplification, I have copied 250TB of a mix of > small and large files (20M of them), and the stored space on EC pool > matched what was expected from actual data size to a few %. So EC > overhead was not really a factor. Also, after the removal of this data > set, the stored space was recovered as expected. > > The I checked the full dump of cephfs and compared it to list of all > objects in the EC pool as follows: > > rados -p cephfs_data_echdd ls > echdd.objectlist > > find /ceph/ -printf "%i %p\n" > cephfs.inodes > > this took several hours, while in the meantime, writing and removing > of data to cephfs was relatively low (few MB/s), so the impact of new > objects and files should have been minimal. > > Then I selected all the objects of the form > 1007b28abae.00000000 > 100cce97f6d.00000000 > 10067733861.00000000 > 100cc4646aa.00000000 > 200044d5c07.00000000 > ... > > and checked if they match the files in the cephfs.inodes list > > ~3M of *.00000000 objects do not have the corresponding inode in the > cephfs.inodes > > > I did "rados stat" of ~70k of these objects (still running for all of > them), and almost all have timestamps from March and April these year. > On 4th of May I remember I have increased the mds purge queue values > to speed up the removal of files since OSDs were filling up too > quickly, and since then it seems the objects are not left uncleaned > any more. > > With "rados getxattr objectid parent" I have checked several of them, > and they all belong to two very active projects which typically write > files with few GB/s all the time (and remove as well so space usage is > not increasing). The objects I have checked belonged to files that > were removed in cephfs, the projects have a separate file catalog > which is consistent with cephfs contents. > > So, I do not understand why so many objects were left unremoved in EC > pool in the period of 2 months, but at least, 3M uncleaned files > explains 1.5PB of dark data by quick estimate. > > We also scrubbed cephfs root and ~mdsdir several times and no > leftovers to remove were found. > > I hope the problem is gone now, but I would still like a good advice > on how to proceed with the cleanup. I see these options: > > 1) remove the unmatched objects directly from EC pool with "rados > rm". But this might have undesired side effects or corruption. > > 2) create a new EC pool and migrate all the data there by copying > files in filesystem, and then destroy the old pool. > > 3) create a new filesystem with new EC pools and migrate the files. > > 4) run some advanced MDS disaster recovery procedure > (cephfs-data-scan), but this requires offline FS and I do not want to > recover old files from existing unmatched objects. Anyway, cephfs > seems to be healthy now. > > Any good ideas? > > Best, > Andrej > > > On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote: >>> ). >>>>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with >>>>> 132M files and 139M rentries. The pool shows 7PiB stored and >>>>> 9.7PiB used consistent with 8+3 EC. >>>>> The layout for most files: >>>>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 >>>>> object_size=16777216 pool=cephfs_data_echdd" >>>>> >>>>> But there is 1.4PiB discrepancy between the pool and the filesystem >> Do you have scrubs enabled? Which if any non-default config options >> do you have set? Any undersized or degraded or backfilling PGs? >> >> Which Ceph release? Do you have a sizable fraction of small files? >> If you’re running Squid or earlier or don’t have EC optimizations >> enabled, even a tiny file will allocate a multiple of 11*16=176 KB. >> An 129KB file will consume 352KB, etc. If I understand those layout >> options correctly. >> >> If that’s what’s going on, going to Tentacle with EC optimizations >> would gain you some efficiency for files newly [re]written. You >> could also migrate small files to a replicated pool. >> >> >>>>> which I cannot explain and I suspect there are a lot of orphan >>>>> objects there. I have run mds scrub on / and ~mdsdir as well. >>>>> There is some mds damage on some old small files (~400 files), >>>>> which I do not think it's relevant here. >>>>> >>>> Hello, >>>> We had a similar issue last year with a group of users that created >>>> and removed files at a very high rate. >>>> >>>> Have you read >>>> https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/cephfs/*cephfs-p… >>>> ? >>>> >>>> We increased the purge rate parameters (very) aggressively to get >>>> back to a comfortable situation (i.e. not a pool w/ near full >>>> warnings). >>>> >>>> >>>> Loïc. >>> >>> -- >>> _____________________________________________________________ >>> prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si >>> Department of Experimental High Energy Physics - F9 >>> Jozef Stefan Institute, Jamova 39, P.o.Box 3000 >>> SI-1001 Ljubljana, Slovenia >>> Tel.: +386-1-477-3674 Fax: +386-1-477-3166 >>> ------------------------------------------------------------- >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: CephFS large amounts of dark data
by Andrej Filipčič 02 Jul '26

02 Jul '26

So, to follow up on this, I did some further investigation. Checking for write amplification, I have copied 250TB of a mix of small and large files (20M of them), and the stored space on EC pool matched what was expected from actual data size to a few %. So EC overhead was not really a factor. Also, after the removal of this data set, the stored space was recovered as expected. The I checked the full dump of cephfs and compared it to list of all objects in the EC pool as follows: rados -p cephfs_data_echdd ls > echdd.objectlist find /ceph/ -printf "%i %p\n" > cephfs.inodes this took several hours, while in the meantime, writing and removing of data to cephfs was relatively low (few MB/s), so the impact of new objects and files should have been minimal. Then I selected all the objects of the form 1007b28abae.00000000 100cce97f6d.00000000 10067733861.00000000 100cc4646aa.00000000 200044d5c07.00000000 ... and checked if they match the files in the cephfs.inodes list ~3M of *.00000000 objects do not have the corresponding inode in the cephfs.inodes I did "rados stat" of ~70k of these objects (still running for all of them), and almost all have timestamps from March and April these year. On 4th of May I remember I have increased the mds purge queue values to speed up the removal of files since OSDs were filling up too quickly, and since then it seems the objects are not left uncleaned any more. With "rados getxattr objectid parent" I have checked several of them, and they all belong to two very active projects which typically write files with few GB/s all the time (and remove as well so space usage is not increasing). The objects I have checked belonged to files that were removed in cephfs, the projects have a separate file catalog which is consistent with cephfs contents. So, I do not understand why so many objects were left unremoved in EC pool in the period of 2 months, but at least, 3M uncleaned files explains 1.5PB of dark data by quick estimate. We also scrubbed cephfs root and ~mdsdir several times and no leftovers to remove were found. I hope the problem is gone now, but I would still like a good advice on how to proceed with the cleanup. I see these options: 1) remove the unmatched objects directly from EC pool with "rados rm". But this might have undesired side effects or corruption. 2) create a new EC pool and migrate all the data there by copying files in filesystem, and then destroy the old pool. 3) create a new filesystem with new EC pools and migrate the files. 4) run some advanced MDS disaster recovery procedure (cephfs-data-scan), but this requires offline FS and I do not want to recover old files from existing unmatched objects. Anyway, cephfs seems to be healthy now. Any good ideas? Best, Andrej On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote: >> ). >>>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3 EC. >>>> The layout for most files: >>>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216 pool=cephfs_data_echdd" >>>> >>>> But there is 1.4PiB discrepancy between the pool and the filesystem > Do you have scrubs enabled? Which if any non-default config options do you have set? Any undersized or degraded or backfilling PGs? > > Which Ceph release? Do you have a sizable fraction of small files? If you’re running Squid or earlier or don’t have EC optimizations enabled, even a tiny file will allocate a multiple of 11*16=176 KB. An 129KB file will consume 352KB, etc. If I understand those layout options correctly. > > If that’s what’s going on, going to Tentacle with EC optimizations would gain you some efficiency for files newly [re]written. You could also migrate small files to a replicated pool. > > >>>> which I cannot explain and I suspect there are a lot of orphan objects there. I have run mds scrub on / and ~mdsdir as well. There is some mds damage on some old small files (~400 files), which I do not think it's relevant here. >>>> >>> Hello, >>> We had a similar issue last year with a group of users that created and removed files at a very high rate. >>> >>> Have you read https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-withou… ? >>> >>> We increased the purge rate parameters (very) aggressively to get back to a comfortable situation (i.e. not a pool w/ near full warnings). >>> >>> >>> Loïc. >> >> -- >> _____________________________________________________________ >> prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si >> Department of Experimental High Energy Physics - F9 >> Jozef Stefan Institute, Jamova 39, P.o.Box 3000 >> SI-1001 Ljubljana, Slovenia >> Tel.: +386-1-477-3674 Fax: +386-1-477-3166 >> ------------------------------------------------------------- >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 0

Re: An issue during expanding an cluster - one osd getting close to nearfull
by Anthony D'Atri 01 Jul '26

01 Jul '26

With 5x existing hosts and Proxmox, I suspect that Robert's cluster is using RBD with 3x replication and is okay in this respect. There were a couple of bugs wrt uneven failure domain weights back around Firefly, but not recently. Is it possible that any of your OSDs have old-style override reweights? The REWEIGHT column of `ceph osd tree` ? That can confuse a cluster using pg-upmap. Please follow up with a copy of `ceph osd df tree` Another approach is https://www.clyso.com/us/adding-capacity-to-ceph-the-clyso-way/ upmap-remapped -- be sure to have the latest version -- is a terrific tool. With this strategy you avoid the thundering herd of backfill, and since the balancer will only move a smaller bite at a time, you avoid the dynamic of intermediate states filling up OSDs. A certain Ceph installation added 40 hosts (yes, you read that right) at once to a cluster and ran into this, though with a smaller upgrade I wouldn't expect to see the problem. With this approach most or all of the backfill gets cancelled before it starts, and the balancer moves it very incrementally. So if you start seeing a problem, you can just turn off the balancer while you investigate. You might also crank down how big a bite the balancer will take in one shot. ceph config set mgr target_max_misplaced_ratio .01 > On Jul 1, 2026, at 9:03 AM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I would try to avoid adding a host with only few OSDs if other hosts are equipped differently. Although I haven't checked in a while I don't expect the behavior to have changed in recent versions. > > So here's my theory: If you add a new host with only one or two OSDs, Ceph can assign many more PGs than average to the first up OSD in order to comply with failure domain (probably host). Same goes for the last OSD during shut down. We had a related case four years ago when we tried to rebuild OSD nodes in a customer cluster. Since the customer cluster was still on default values for mon_max_pg_per_osd and osd_max_pg_per_osd_hard_ratio, the (too many) assigned PGs led to inactive PGs: > >> maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750 > > When more OSDs were created on that host (ceph-volume does that sequentially), the issue resolved eventually, except for some inactive PGs, failing the MGR resolved those as well. Since you have around 360 PGs per OSD in average, I assume you increased mon_max_pg_per_osd (default 250), maybe also osd_max_pg_per_osd_hard_ratio (default 3), you most likely dodged the "inactive PG" bullet. > > So my recommendation would be to rather add the hosts with all OSDs at the same time but with crush weight 0. That way you have time to inspect the outcome (for example with dedicated DB/WAL devices you can check if they were created according to the specs). Then reweight the OSDs to the correct value, some users do that gradually to prevent an overload. > > Note the recent discussion about osd heartbeats when adding multiple hosts at once: > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHVPS247M4E… > > Regards, > Eugen > > Zitat von Robert Lukan via ceph-users <ceph-users(a)ceph.io>: > >> Hi everyone, >> >> We are currently in the middle of migrating our infrastructure to a >> Proxmox/Ceph stack and ran into a concerning edge case while expanding the >> cluster. I am hoping to get some advice on the safest way to handle this >> without risking a nearfull OSD. >> >> *The Topology & Situation:* >> >> *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs. >> >> *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and >> osd.41). >> >> *Cluster Average:* ~360 PGs per OSD, ~60% utilization. >> >> *Ceph Version:* Squid 19.2.3 >> >> The last time we were expanding the cluster, we had issues when expanding >> it with many OSDs at the same time(it was a memory problem on a host), so >> we decided to be more careful and add 2 disks at the time. >> >> During the backfill phase, we noticed osd.40 catching significantly more >> PGs than osd.41. We paused the migration by setting the nobackfill flag to >> prevent a potential nearfull situation. >> >> *The Current State (ceph osd df):* >> >> osd.40: 423 PGs, 72.56% used, >> >> osd.41: 307 PGs, 51.87% used, >> >> We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we >> have paused the backfill. >> >> If we were to unset nobackfill, the incoming data for those 218 PGs would >> risk pushing osd.40 past the 85% nearfull threshold. However, our >> balancer refused to generate any optimization plans. It reports: >> "optimize_result": >> "Unable to find further optimization... or distribution is already perfect" >> >> We "understand" why this is happening: the balancer is evaluating only when >> backfilling is done. However, it is completely ignoring the transient acting >> state, which was heavily overloading osd.40 >> >> *Our Temporary Workaround:* To prevent the potential issue, we have >> temporarily set the reweight of osd.40 to 0 (effectively draining it out) >> so we can safely lift the nobackfill flag and let the cluster settle >> without hitting capacity limits. >> >> *My Questions for the Community:* >> >> What is the standard best practice for handling this "small bucket" >> transient imbalance when adding a small number of OSDs to a new host? Or is >> this just a bad idea in general ? >> >> Is there a way to force the built-in upmap balancer to respect the >> transient/acting capacity of an OSD so it doesn't allow a drive to go >> towards 85% during a migration? I understand we were not there, but being >> 12 % away was "close" enough. >> >> One of the ideas was to reweight the offending OSD, to remove a few PGs, so >> it would not be so close to nearfull situation. But this approach is >> tedious and requires "baby-sitting". >> >> I guess this might be a good candidate for upmap remap script, but we were >> not "brave" enough to run in it a production environment the first time. >> >> Thanks, >> >> Robert >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: An issue during expanding an cluster - one osd getting close to nearfull
by Eugen Block 01 Jul '26

01 Jul '26

Hi, I would try to avoid adding a host with only few OSDs if other hosts are equipped differently. Although I haven't checked in a while I don't expect the behavior to have changed in recent versions. So here's my theory: If you add a new host with only one or two OSDs, Ceph can assign many more PGs than average to the first up OSD in order to comply with failure domain (probably host). Same goes for the last OSD during shut down. We had a related case four years ago when we tried to rebuild OSD nodes in a customer cluster. Since the customer cluster was still on default values for mon_max_pg_per_osd and osd_max_pg_per_osd_hard_ratio, the (too many) assigned PGs led to inactive PGs: > maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750 When more OSDs were created on that host (ceph-volume does that sequentially), the issue resolved eventually, except for some inactive PGs, failing the MGR resolved those as well. Since you have around 360 PGs per OSD in average, I assume you increased mon_max_pg_per_osd (default 250), maybe also osd_max_pg_per_osd_hard_ratio (default 3), you most likely dodged the "inactive PG" bullet. So my recommendation would be to rather add the hosts with all OSDs at the same time but with crush weight 0. That way you have time to inspect the outcome (for example with dedicated DB/WAL devices you can check if they were created according to the specs). Then reweight the OSDs to the correct value, some users do that gradually to prevent an overload. Note the recent discussion about osd heartbeats when adding multiple hosts at once: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHVPS247M4E… Regards, Eugen Zitat von Robert Lukan via ceph-users <ceph-users(a)ceph.io>: > Hi everyone, > > We are currently in the middle of migrating our infrastructure to a > Proxmox/Ceph stack and ran into a concerning edge case while expanding the > cluster. I am hoping to get some advice on the safest way to handle this > without risking a nearfull OSD. > > *The Topology & Situation:* > > *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs. > > *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and > osd.41). > > *Cluster Average:* ~360 PGs per OSD, ~60% utilization. > > *Ceph Version:* Squid 19.2.3 > > The last time we were expanding the cluster, we had issues when expanding > it with many OSDs at the same time(it was a memory problem on a host), so > we decided to be more careful and add 2 disks at the time. > > During the backfill phase, we noticed osd.40 catching significantly more > PGs than osd.41. We paused the migration by setting the nobackfill flag to > prevent a potential nearfull situation. > > *The Current State (ceph osd df):* > > osd.40: 423 PGs, 72.56% used, > > osd.41: 307 PGs, 51.87% used, > > We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we > have paused the backfill. > > If we were to unset nobackfill, the incoming data for those 218 PGs would > risk pushing osd.40 past the 85% nearfull threshold. However, our > balancer refused to generate any optimization plans. It reports: > "optimize_result": > "Unable to find further optimization... or distribution is already perfect" > > We "understand" why this is happening: the balancer is evaluating only when > backfilling is done. However, it is completely ignoring the transient acting > state, which was heavily overloading osd.40 > > *Our Temporary Workaround:* To prevent the potential issue, we have > temporarily set the reweight of osd.40 to 0 (effectively draining it out) > so we can safely lift the nobackfill flag and let the cluster settle > without hitting capacity limits. > > *My Questions for the Community:* > > What is the standard best practice for handling this "small bucket" > transient imbalance when adding a small number of OSDs to a new host? Or is > this just a bad idea in general ? > > Is there a way to force the built-in upmap balancer to respect the > transient/acting capacity of an OSD so it doesn't allow a drive to go > towards 85% during a migration? I understand we were not there, but being > 12 % away was "close" enough. > > One of the ideas was to reweight the offending OSD, to remove a few PGs, so > it would not be so close to nearfull situation. But this approach is > tedious and requires "baby-sitting". > > I guess this might be a good candidate for upmap remap script, but we were > not "brave" enough to run in it a production environment the first time. > > Thanks, > > Robert > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Grafana dashboards warnings after upgrade to Squid 19.2.4
by Sake 01 Jul '26

01 Jul '26

Probably something not going correctly, because I just upgraded to 19.2.4 and Grafana is using the grafana image (in the past this was ceph-grafana) version 12.3.1. On 1 July 2026 11:13:02 CEST, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: >Hi, > >I just upgraded our production cluster to 19.2.4 (from 19.2.3), there were no issues reported by our users, so all good. But I noticed that many graphs now show different warnings or strange labels: > >- "This panel requires Angular (deprecated)." on multiple graphs >- "PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "ceph_pool_wr" (1:6)" also on multiple graphs for different metrics >- "OSD Capacity Utilization" just shows: "RangeError: invalid array length" >- The "Pool Overview" graph has column names like "Value #F", "Value #G" and so on. And the values within don't make sense either. > >I have uploaded screenshots: > >https://www.pasteboard.co/A34Zcy9cqHe5.png >https://www.pasteboard.co/F_vkpeBjGGWH.png >https://www.pasteboard.co/mkCtlBN3zS6v.png >https://www.pasteboard.co/T_ByD_gCpslA.png > >I compared the prometheus and grafana versions we have currently running with a fresh 19.2.4 bootstrapped cluster and the versions are identical: > >ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' >10.4.0 > >ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' >2.51.0 > >Is there anything I'm missing? > >Thanks! >Eugen >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Grafana dashboards warnings after upgrade to Squid 19.2.4
by Chris Palmer 01 Jul '26

01 Jul '26

From Tentacle 20.2.1: ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' 12.3.1 ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' 3.6.0 I noticed a big improvement moving from squid to tentacle. A different set of things wrong, but a much smaller set! Chris On 01/07/2026 10:46, Daniel Brown via ceph-users wrote: > Not probably immediately related to your issue, but Grafana just release version 13.1 in June. That means that 12.4 becomes the earlier “supported” version. 11.x and 10.x are a couple years old and unsupported now. Less specially an issue here, as Ceph is using the OSS version, not the Enterprise version of Grafana, but might be time for the project to consider revving which Grafana image is used. > > > -- > Dan Brown > >> On Jul 1, 2026, at 05:13, Eugen Block via ceph-users<ceph-users(a)ceph.io> wrote: >> >> ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' >> 10.4.0 >> >> ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' >> 2.51.0 >> > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

Re: Grafana dashboards warnings after upgrade to Squid 19.2.4
by Daniel Brown 01 Jul '26

01 Jul '26

Not probably immediately related to your issue, but Grafana just release version 13.1 in June. That means that 12.4 becomes the earlier “supported” version. 11.x and 10.x are a couple years old and unsupported now. Less specially an issue here, as Ceph is using the OSS version, not the Enterprise version of Grafana, but might be time for the project to consider revving which Grafana image is used. -- Dan Brown > On Jul 1, 2026, at 05:13, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' > 10.4.0 > > ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' > 2.51.0 >

1 0

Grafana dashboards warnings after upgrade to Squid 19.2.4
by Eugen Block 01 Jul '26

01 Jul '26

Hi, I just upgraded our production cluster to 19.2.4 (from 19.2.3), there were no issues reported by our users, so all good. But I noticed that many graphs now show different warnings or strange labels: - "This panel requires Angular (deprecated)." on multiple graphs - "PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "ceph_pool_wr" (1:6)" also on multiple graphs for different metrics - "OSD Capacity Utilization" just shows: "RangeError: invalid array length" - The "Pool Overview" graph has column names like "Value #F", "Value #G" and so on. And the values within don't make sense either. I have uploaded screenshots: https://www.pasteboard.co/A34Zcy9cqHe5.png https://www.pasteboard.co/F_vkpeBjGGWH.png https://www.pasteboard.co/mkCtlBN3zS6v.png https://www.pasteboard.co/T_ByD_gCpslA.png I compared the prometheus and grafana versions we have currently running with a fresh 19.2.4 bootstrapped cluster and the versions are identical: ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' 10.4.0 ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' 2.51.0 Is there anything I'm missing? Thanks! Eugen

1 0