July 2026 - ceph-users - lists.ceph.io

Re: Performance issue after tentacle upgrade
by Moti 10 Aug '26

10 Aug '26

Hello Eugen, I agree that a performance issue this major would've gone noticed by now, but I did a load of investigation and still can't find the root cause, it might be a non-Ceph issue, but also possible to be a corner case? No there haven't been any OS upgrade, or another specific change. Did a ceph config dump (including defaults) before/after upgrade, checked all the changed ones, reverted the fishy ones, nothing changed. The previous 3 clusters that I was testing on were all bare-metal - all had multi-site between 2 zones; separation between ec and replica pool is handled with a lua script, no RBD only object store. I also checked on a VM-based cluster (again multi-site, but no hybrid build this time). Same issue there. However, when tested on a small single site cluster, that wasn't the case (so no visible latency increase there), but it just made things more confusing since in all of the other multi-site clusters, the RGWs that run sync threads are separate from the ones that clients connect too, and it's the client latency that I'm monitoring. I did notice this issue: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… but that's not much similar to our case since we don't see any jump in CPU or memory usage. Also checked pg ops (including scrubs), some clusters were running scrubs and stuff for a while after upgrade but that cleared soon enough, though the latency jump seems persistent still after 2 months. One correction about my observation on NVMe iops: that started dropping a little while "before" upgrade so it's most likely a separate issue, but latency jump case stays the same. As for other parameters, LC ops do not seem to have changed, there is a jump in GC max age but that's minor, cross site replication looks fine, no jump in osd compression. Any ideas what else might worth checking/testing? On Tue, 14 Jul 2026, 12:53 Eugen Block via ceph-users, <ceph-users(a)ceph.io> wrote: > Hi, > > I'm not sure if it could be related to summer vacation, but I feel > like there should have been several reports similar to yours by now. I > don't have upgraded any production cluster to 20 yet, so I can't > really deny or confirm anything. > Did you upgrade the OS as well? Maybe there were some relevant changes > there. Or some non-persistent configs were overwritten, either in the > OS or Ceph. > Was there more deep-scrubbing after the upgrade which could have > impacted the IO? > > Regards, > Eugen > > Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > > > Hello everyone! > > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing > as > > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET > and > > HEAD requests. > > > > There is no resource cap or significant increase in usage is visible. > > Neither fast ec nor any other new features are enabled yet, just did the > > upgrade and left it as is. For measuring latency I have a services that > > probes the cluster with consistent load so it's not a difference in load > > either. PG status history does not show any pg operation being heavily in > > process. > > > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db > block. > > Both ec and replicated pools exist. > > The strange hint was: NVMe disks' iops went from something around 5-6K > > constant io per second to almost zero (less than a hundred per second, on > > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > > not much changed. db blocks on NVMe are still attached to OSDs with no > > issue. > > > > I can't find a clue to explain this. Got the same results on 3 separate > > clusters (all went version 18 --> 20, same hybrid build and configs). > > Nothing in config dump changes (changed defaults) catches my attention to > > have caused this. > > > > Anyone had similar issues? Any tips and ideas on what to check? > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Many empty objects in CephFS data pool
by Eugen Block 31 Jul '26

31 Jul '26

Hi, are you using snapshots on CephFS as well? Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Hi, > > Am 31.07.26 um 9:11 AM schrieb Dario Graña: > >> here is an snippet from the documentation[0] that explains this behavior: >> /The data pool used to create the file system is the “default” data >> pool and the location for storing all inode backtrace information, >> used for hard link management and disaster recovery. For this >> reason, all inodes created in CephFS have at least one object in >> the default data pool. If erasure-coded pools are planned for the >> file system, it is usually better to use a replicated pool for the >> default data pool to improve small-object write and read >> performance for updating backtraces./ > > Thanks for the hint. I already suspected something like this. > > It is so strange as there are many more empty objects in the first > replicated data pool than there are "real" file objects in the EC > data pool. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Many empty objects in CephFS data pool
by Dario Graña 31 Jul '26

31 Jul '26

Hello, here is an snippet from the documentation[0] that explains this behavior: *The data pool used to create the file system is the “default” data pool and the location for storing all inode backtrace information, used for hard link management and disaster recovery. For this reason, all inodes created in CephFS have at least one object in the default data pool. If erasure-coded pools are planned for the file system, it is usually better to use a replicated pool for the default data pool to improve small-object write and read performance for updating backtraces.* [0] https://docs.ceph.com/en/latest/cephfs/createfs/ On Thu, Jul 30, 2026 at 12:04 PM Loïc Tortay via ceph-users < ceph-users(a)ceph.io> wrote: > On 30/07/2026 11:42, Robert Sander via ceph-users wrote: > > Hi, > > > > we have a CephFS with a replicated data pool and an additional EC data > > pool. All files store their data in the EC pool. > > > > But "ceph df" shows that the replicated data pool contains over 200 > > million objects with a total size of 700 bytes. So the vast majority are > > empty objects. > > > > How can this be? > > > Hello, > The replicated pool is the "default pool" and contains files metadata ? > (unlike the "metadata" pool) > > Like in this filesystem with ~1.4 billion files and 85 million directories: > POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL > cephfs_EC_data 3 8192 6.6 PiB 3.00G 8.3 PiB 79.90 1.7 PiB > cephfs_metadata 4 256 778 GiB 84.70M 2.3 TiB 3.38 22 TiB > cephfs_default 5 128 0 B 1.37G 0 B 0 22 TiB > > > Loïc. > -- > | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre | > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: v20.2.3 tentacle QE validation status
by Venky Shankar 31 Jul '26

31 Jul '26

Hey Yuri, On Mon, Jul 27, 2026 at 7:37 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > https://tracker.ceph.com/issues/77256#note-5 > > Release Notes - TBD > LRC upgrade - TBD > > PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky fs approved -- https://tracker.ceph.com/issues/77256#note-15 > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > krbd: Ilya > > nvmeof: Aviv, Vallari > > upgrade: Kamoltat, Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > Please reply with your approval or flag any concerns. -- Cheers, Venky

1 0

Ceph Developer Summit 2026
by Anthony Middleton 30 Jul '26

30 Jul '26

Hey Ceph friends, The Ceph Developer Summit begins next week! From August 4–13, the Ceph community comes together online to shape the next major release, Vampire, coming in Spring 2027. Nine sessions across two weeks will cover RADOS, RBD, RGW, CephFS, Dashboard, Cephadm, Telemetry/Ceph-MGR, NVMe-oF gateway, and Crimson/SeaStore — led by the engineers building each of these components. Whether you're a core contributor, an operator running Ceph in production, or just curious about what's next for the project, every session is open to join. Bring your questions, your feedback, and your ideas for where these features should go. Each session will be recorded and shared on our YouTube channel. Full schedule, speakers, and session links below: https://t.ly/CDS-Vampire Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: v20.2.3 tentacle QE validation status
by Kamoltat Sirivadhna 30 Jul '26

30 Jul '26

upgrade approved (Note that the many failures we saw was due to a known issue regarding stress split and mds upgrade) On Mon, Jul 27, 2026 at 10:09 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Details of this release are summarized here: > https://tracker.ceph.com/issues/77256#note-5 > > Release Notes - TBD > LRC upgrade - TBD > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > krbd: Ilya > > nvmeof: Aviv, Vallari > > upgrade: Kamoltat, Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > Please reply with your approval or flag any concerns. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Kamoltat (Junior) Sirivadhna (HE/HIM) Senior SoftWare Engineer - IBM Ceph Storage ksirivad(a)redhat.com T: (857) <(919)716-5348>253-8927

1 0

Using CephFS for encrypted Kubernetes volumes
by Stefan Möding 30 Jul '26

30 Jul '26

Hi, we are trying to use Ceph (Tentacle 20.2.2) as storage provider for multiple Kubernetes clusters. We are trying to implement the following security measures: * Each Kubernetes cluster should use a dedicated RADOS namespace for its objects. * The volumes should use fscrypt encryption. We can deploy pods successfully as long as we don’t try to enable fscrypt. With encryption enabled the pod shows the following logs: AttachVolume.Attach succeeded for volume "pvc-0923605d-3993-4d99-a4a3-d41749ead831" MountVolume.MountDevice failed for volume "pvc-0923605d-3993-4d99-a4a3-d41749ead831" : rpc error: code = Internal desc = failed to lock volume ID 64ab25ab-9d66-4508-be74-13c02825f40e: rados: ret=-1, Operation not permitted There is a closed issue that mentions a length limitation when creating a lock to initialize the encryption: https://github.com/ceph/ceph-csi/issues/5419 I also tried the mentioned workaround but that does not help. There has also been a discussion that the ’x’ permission is needed in the OSD capabilities, which we have set: mon_caps: 'allow r fsname=root' mgr_caps: 'allow rw' osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, allow rw namespace=k8s-test tag cephfs data=root' mds_caps: 'allow r fsname=root path=/volumes, allow rws fsname=root path=/volumes/k8s-test' Does anybody have a working setup with encryption and RADOS namespaces? -- Stefan

1 0

Re: Many empty objects in CephFS data pool
by Loïc Tortay 30 Jul '26

30 Jul '26

On 30/07/2026 11:42, Robert Sander via ceph-users wrote: > Hi, > > we have a CephFS with a replicated data pool and an additional EC data > pool. All files store their data in the EC pool. > > But "ceph df" shows that the replicated data pool contains over 200 > million objects with a total size of 700 bytes. So the vast majority are > empty objects. > > How can this be? > Hello, The replicated pool is the "default pool" and contains files metadata ? (unlike the "metadata" pool) Like in this filesystem with ~1.4 billion files and 85 million directories: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_EC_data 3 8192 6.6 PiB 3.00G 8.3 PiB 79.90 1.7 PiB cephfs_metadata 4 256 778 GiB 84.70M 2.3 TiB 3.38 22 TiB cephfs_default 5 128 0 B 1.37G 0 B 0 22 TiB Loïc. -- | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Re: Many empty objects in CephFS data pool
by Janne Johansson 30 Jul '26

30 Jul '26

> > > we have a CephFS with a replicated data pool and an additional EC data > pool. All files store their data in the EC pool. > > But "ceph df" shows that the replicated data pool contains over 200 > million objects with a total size of 700 bytes. So the vast majority are > empty objects. > > How can this be? > I don't do cephfs, but at least radosgw likes to stash information into zero-sized objects, so the sum of the sizes of those objects was small, but the data usage would of course be larger than zero on those rgw pools. Perhaps cephfs does the same? "ceph df detail" on one of our rgw clusters show this: POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR default.rgw.buckets.index 7 512 51 GiB 0 B 51 GiB 9.57k 154 GiB 0 B 154 GiB 0.02 225 TiB N/A N/A N/A 0 B 0 B the "DATA" column is zero, even though there are 9.57k objects in there, but OMAP is lots more, probably meaning it stashed information into the metadata of said zero-sized objs. If we stat one of the .dir.xxxx objects, it says: rados stat -p default.rgw.buckets.index ".dir.b9865dc1-e648-4479-abcd-ed812345678.144664284.33737.1.20" default.rgw.buckets.index/.dir.b9865dc1-e648-4479-abcd-ed87654321.144664284.33737.1.20 mtime 2026-07-30T09:48:05.000000+0000, size 0 -- May the most significant bit of your life be positive.

1 0

some details on "split brain" and 'min_size' vs. 'size' and 'k,m'
by pg＠ceph.list.sabi.co.UK 30 Jul '26

30 Jul '26

I have been slightly perplexed about some discussions around 'min_size' and had about that a discussion with a gooda LLM about some related technical details: https://chat.deepseek.com/share/u1jxte2mbbeetwr0sl tl;dr: why worry about details, be happy! :-)

1 0

Many empty objects in CephFS data pool
by Robert Sander 30 Jul '26

30 Jul '26

Hi, we have a CephFS with a replicated data pool and an additional EC data pool. All files store their data in the EC pool. But "ceph df" shows that the replicated data pool contains over 200 million objects with a total size of 700 bytes. So the vast majority are empty objects. How can this be? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

Re: PGs stuck remapped
by Maximilian Hill 30 Jul '26

30 Jul '26

Hi, Well, at least the PGs are empty (0 objects according to ceph pg dump). Declaring them lost might be a thing. It looks like it might not be that urgent, so I check on the managers tomorrow morning. On 2026-07-29 20:48, Eugen Block via ceph-users wrote: > Oh that’s not good. Can you fail the mgr and then see if anything changes? > > Zitat von Maximilian Hill <max(a)maxhill.eu>: > >> Hi, >> >> There are plenty hosts for backfilling. >> The cluster consists of 13 hosts configured like a stretch cluster but >> without enabling stretch mode. (6 osd nodes per "room" + 1 mon in a >> third "room", pools have size 4 (2 each "room")). >> >> Since I first wrote there has been a few changes: >> - 2 PGs show state "unknown" >> - output of ceph health detail: >> [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive >> pg 20.25 is stuck inactive for 12h, current state unknown, last >> acting [] >> pg 28.ca is stuck inactive for 12h, current state unknown, last >> acting [] >> >> Maybe that's part of the problem. It doesn't make much sense to me, >> that it appeared a few hours ago but ceph health detail shows 12h. >> >> Ceph pq 20.25 query, as well as 28.ca show both the following error: >> >> # ceph pg 20.25 query >> Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) >> Traceback (most recent call last): >> File "/usr/bin/ceph", line 1327, in <module> >> retval = main() >> File "/usr/bin/ceph", line 1247, in main >> sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') >> File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1006, >> in parse_json_funcsigs >> raise e >> File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1003, >> in parse_json_funcsigs >> overall = json.loads(s) >> File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads >> return _default_decoder.decode(s) >> File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode >> obj, end = self.raw_decode(s, idx=_w(s, 0).end()) >> File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode >> raise JSONDecodeError("Expecting value", s, err.value) from None >> json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) >> >> >> >> On 2026-07-29 19:00, Eugen Block via ceph-users wrote: >>> Hi, >>> >>> without more information I assume the affected PGs are from pools >>> that rely on the host you’re trying to drain. >>> How many hosts do you have, what’s the failure domain of the crush >>> rules in use? Can you share the crush rule dump and ‚ceph osd pool ls >>> detail‘? >>> Lets assume that you have 3 hosts with replica 3 with failure domain >>> host. Draining a host doesn’t allow any backfill because there aren’t >>> any hosts available to backfill to without breaking the crush rule. >>> Another reason could be mclock, but it usually showed very little >>> backfill IO and not none in previous reports. So I’m guessing it’s >>> the crush rule(s) and the number of hosts not suitable for recovery/ >>> backfill. >>> >>> Zitat von Maximilian Hill via ceph-users <ceph-users(a)ceph.io>: >>> >>>> Hello, >>>> >>>> I'm running a cluster on Ceph squid 19.2.3. >>>> Currently a drain (ceph orch host drain) was started for a node. >>>> >>>> Now I have misplaced objects and 605 PGs stuck in active+clean+remapped >>>> but no backfilling is active. >>>> >>>> Can someone give me some advice? >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 5

Re: PGs stuck remapped
by Maximilian Hill 29 Jul '26

29 Jul '26

Hi, There are plenty hosts for backfilling. The cluster consists of 13 hosts configured like a stretch cluster but without enabling stretch mode. (6 osd nodes per "room" + 1 mon in a third "room", pools have size 4 (2 each "room")). Since I first wrote there has been a few changes: - 2 PGs show state "unknown" - output of ceph health detail: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive pg 20.25 is stuck inactive for 12h, current state unknown, last acting [] pg 28.ca is stuck inactive for 12h, current state unknown, last acting [] Maybe that's part of the problem. It doesn't make much sense to me, that it appeared a few hours ago but ceph health detail shows 12h. Ceph pq 20.25 query, as well as 28.ca show both the following error: # ceph pg 20.25 query Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1247, in main sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1006, in parse_json_funcsigs raise e File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1003, in parse_json_funcsigs overall = json.loads(s) File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) On 2026-07-29 19:00, Eugen Block via ceph-users wrote: > Hi, > > without more information I assume the affected PGs are from pools that > rely on the host you’re trying to drain. > How many hosts do you have, what’s the failure domain of the crush rules > in use? Can you share the crush rule dump and ‚ceph osd pool ls detail‘? > Lets assume that you have 3 hosts with replica 3 with failure domain > host. Draining a host doesn’t allow any backfill because there aren’t > any hosts available to backfill to without breaking the crush rule. > Another reason could be mclock, but it usually showed very little > backfill IO and not none in previous reports. So I’m guessing it’s the > crush rule(s) and the number of hosts not suitable for recovery/backfill. > > Zitat von Maximilian Hill via ceph-users <ceph-users(a)ceph.io>: > >> Hello, >> >> I'm running a cluster on Ceph squid 19.2.3. >> Currently a drain (ceph orch host drain) was started for a node. >> >> Now I have misplaced objects and 605 PGs stuck in active+clean+remapped >> but no backfilling is active. >> >> Can someone give me some advice? >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: PGs stuck remapped
by Eugen Block 29 Jul '26

29 Jul '26

Hi, without more information I assume the affected PGs are from pools that rely on the host you’re trying to drain. How many hosts do you have, what’s the failure domain of the crush rules in use? Can you share the crush rule dump and ‚ceph osd pool ls detail‘? Lets assume that you have 3 hosts with replica 3 with failure domain host. Draining a host doesn’t allow any backfill because there aren’t any hosts available to backfill to without breaking the crush rule. Another reason could be mclock, but it usually showed very little backfill IO and not none in previous reports. So I’m guessing it’s the crush rule(s) and the number of hosts not suitable for recovery/backfill. Zitat von Maximilian Hill via ceph-users <ceph-users(a)ceph.io>: > Hello, > > I'm running a cluster on Ceph squid 19.2.3. > Currently a drain (ceph orch host drain) was started for a node. > > Now I have misplaced objects and 605 PGs stuck in active+clean+remapped > but no backfilling is active. > > Can someone give me some advice? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Anthony D'Atri 29 Jul '26

29 Jul '26

Might bumping vm.min_free_kbytes mitigate this dynamic? The default is kind of parsimonious; I routinely set to 4GB. > On Jul 29, 2026, at 3:14 AM, Dan van der Ster via ceph-users <ceph-users(a)ceph.io> wrote: > > On Wed, Jul 29, 2026 at 4:01 AM Gregory Farnum via ceph-users > <ceph-users(a)ceph.io> wrote: >>> On Tue, Jul 28, 2026 at 3:05 AM Janne Johansson via ceph-users >>> <ceph-users(a)ceph.io> wrote: >>> I think the idea was that if you mount cephfs on a box running OSDs or a >>> mon daemon, and it gets into some kind of trouble like running out of >>> memory or so, then both the client side and the "server" side might be >>> competing for the same resources that aren't there, making it worse. >> This is a classic problem in network filesystems: >> 1) The kernel runs out of memory and needs to flush out buffered >> writes to free up space >> 2) The client sends out a network message, which comes back in to the server >> 3) The server needs to allocate memory to handle the incoming network message >> 4) ...stuck >> RBD and CephFS are susceptible to this, if you mount the kernels on >> the same hosts as the OSDs. But the odds of hitting it are a lot lower >> on modern systems with their RAM capacities than in the past. > > It's been a couple of years, but I was still hitting this deadlock on > el8 kernels with CephFS / ceph-osd co-located alongside some HPC-type > MPI jobs. > It's like Greg wrote -- a large client read/write that comes back to > an OSD on the same host deadlocks the box. > > Cheers, Dan > > > >> -Greg >>> If you have separate OSD hosts and they are having issues, then cephfs >>> clients will be slow, but not much more than that, and if clients are >>> having resource dips, a separate cluster doesn't mind at all. I think that >>> is where the non-recommendation comes from, in case you do combine them, it >>> will be hard to get out of this situation, any cpu/ram given to the client >>> side would cause more IO to be requested, from the server side that doesn't >>> have it. In any case it might make it lots harder to diagnose which end >>> actually caused the issue if both suffer at the same time. >>> Den tis 28 juli 2026 kl 14:45 skrev Marek Szuba via ceph-users < >>> ceph-users(a)ceph.io>: >>>> Dear fellow Cephies, >>>> A few weeks ago I came across in Tentacle documentation on docs.ceph.org >>>> (and OF COURSE now that I'm trying to quote it, I cannot find it any >>>> more) a mention that CephFS should not be mounted on systems serving >>>> Ceph because of there being a risk of harmful interference in the >>>> kernel. The context suggested this pertained to one-node Ceph clusters, >>>> that said I do not believe that was explicitly stated. Therefore, I very >>>> much hope someone will be able to enlighten me whether: >>>> * this is indeed still the case with Tentacle, >>>> * it applies to one-node clusters only or all Ceph clusters, >>>> * it only applies to CephFS mounts using the kernel driver, or ceph-fuse >>>> as well. >>>> Thank you in advance! >>>> -- >>>> MS >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> -- >>> May the most significant bit of your life be positive. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

PGs stuck remapped
by Maximilian Hill 29 Jul '26

29 Jul '26

Hello, I'm running a cluster on Ceph squid 19.2.3. Currently a drain (ceph orch host drain) was started for a node. Now I have misplaced objects and 605 PGs stuck in active+clean+remapped but no backfilling is active. Can someone give me some advice?

1 0

Re: problem uploading dashboard certificates
by Simon Oosthoek 29 Jul '26

29 Jul '26

Hi Eugen, that worked! ceph config-key get mgr/dashboard/cephmon1/crt ceph config-key get mgr/dashboard/cephmon1/key ceph config-key rm mgr/dashboard/crt ceph config-key rm mgr/dashboard/key ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem >failed! ceph mgr module disable dashboard ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem >failed! ceph mgr module enable dashboard ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem ceph dashboard set-ssl-certificate-key -i cephmon.science.ru.nl.key Then it worked again :-) Cheers, /Simon On Mon, 27 Jul 2026 at 13:11, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > Hi, > > the dashboard cert and key files are stored in: > > ceph config-key get mgr/dashboard/crt > ceph config-key get mgr/dashboard/key > > Or something similar, maybe per host entries like (the "random" string > is cephadm specific and won't apply in your case, of course): > > ceph config-key get mgr/dashboard/host1.random/crt > ceph config-key get mgr/dashboard/host1.random/key > > You should be able to remove those faulty entries with > > ceph config-key rm mgr/dashboard/host1.random/crt > ceph config-key rm mgr/dashboard/host1.random/key > > Then fail the mgr and inject your correct cert/key files again. I > think that should work, if not, let us know. > > Regards, > Eugen > > Zitat von Simon Oosthoek via ceph-users <ceph-users(a)ceph.io>: > > > Sorry, forgot to add info about the installation; this is ceph 17.2.x on > > ubuntu 22.04.x, no cephadm and the cluster has about 360 OSDs of 12TB > HDD. > > > > On Mon, 27 Jul 2026 at 11:50, Simon Oosthoek via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Hi all, > >> we've had some issues when Harica revoked a bunch of our certificates, I > >> then accidentally uploaded mismatching key/pem files. When I wanted to > fix > >> that with the correct ones I got > >> > >> # ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem > >> Error EIO: Module 'dashboard' has experienced an error and cannot handle > >> commands: [('x509 certificate routines', '', 'key values mismatch')] > >> > >> Is there another way to fix this? > >> > >> Cheers, > >> > >> /Simon > >> -- > >> I'm using my gmail.com address, because the gmail.com dmarc policy is > >> "none", some mail servers will reject this (microsoft?) others will > instead > >> allow this when I send mail to a mailling list which has not yet been > >> configured to send mail "on behalf of" the sender, but rather do a kind > of > >> "forward". The latter situation causes dkim/dmarc failures and the dmarc > >> policy will be applied. see https://wiki.list.org/DEV/DMARC for more > >> details > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > > > > -- > > I'm using my gmail.com address, because the gmail.com dmarc policy is > > "none", some mail servers will reject this (microsoft?) others will > instead > > allow this when I send mail to a mailling list which has not yet been > > configured to send mail "on behalf of" the sender, but rather do a kind > of > > "forward". The latter situation causes dkim/dmarc failures and the dmarc > > policy will be applied. see https://wiki.list.org/DEV/DMARC for more > details > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

2 1

Re: v20.2.3 tentacle QE validation status
by Guillaume ABRIOUX 29 Jul '26

29 Jul '26

Hello Yuri, ceph-volume approved Regards, -- Guillaume Abrioux Software Engineer From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> Date: Monday, 27 July 2026 at 16:10 To: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] v20.2.3 tentacle QE validation status Details of this release are summarized here: https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… Release Notes - TBD LRC upgrade - TBD *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* Seeking approvals/reviews for: smoke: PASSED rados: Kamoltat, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv, Vallari upgrade: Kamoltat, Patrick upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED Please reply with your approval or flag any concerns. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io Unless otherwise stated above: Compagnie IBM France Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex RCS Nanterre 552 118 465 Forme Sociale : S.A.S. Capital Social : 664 614 175,50 € SIRET : 552 118 465 03644 - Code NAF 6203Z

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Dan van der Ster 29 Jul '26

29 Jul '26

On Wed, Jul 29, 2026 at 4:01 AM Gregory Farnum via ceph-users <ceph-users(a)ceph.io> wrote: > > On Tue, Jul 28, 2026 at 3:05 AM Janne Johansson via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > I think the idea was that if you mount cephfs on a box running OSDs or a > > mon daemon, and it gets into some kind of trouble like running out of > > memory or so, then both the client side and the "server" side might be > > competing for the same resources that aren't there, making it worse. > > This is a classic problem in network filesystems: > 1) The kernel runs out of memory and needs to flush out buffered > writes to free up space > 2) The client sends out a network message, which comes back in to the server > 3) The server needs to allocate memory to handle the incoming network message > 4) ...stuck > > RBD and CephFS are susceptible to this, if you mount the kernels on > the same hosts as the OSDs. But the odds of hitting it are a lot lower > on modern systems with their RAM capacities than in the past. It's been a couple of years, but I was still hitting this deadlock on el8 kernels with CephFS / ceph-osd co-located alongside some HPC-type MPI jobs. It's like Greg wrote -- a large client read/write that comes back to an OSD on the same host deadlocks the box. Cheers, Dan > -Greg > > > > > If you have separate OSD hosts and they are having issues, then cephfs > > clients will be slow, but not much more than that, and if clients are > > having resource dips, a separate cluster doesn't mind at all. I think that > > is where the non-recommendation comes from, in case you do combine them, it > > will be hard to get out of this situation, any cpu/ram given to the client > > side would cause more IO to be requested, from the server side that doesn't > > have it. In any case it might make it lots harder to diagnose which end > > actually caused the issue if both suffer at the same time. > > > > > > > > Den tis 28 juli 2026 kl 14:45 skrev Marek Szuba via ceph-users < > > ceph-users(a)ceph.io>: > > > > > Dear fellow Cephies, > > > > > > A few weeks ago I came across in Tentacle documentation on docs.ceph.org > > > (and OF COURSE now that I'm trying to quote it, I cannot find it any > > > more) a mention that CephFS should not be mounted on systems serving > > > Ceph because of there being a risk of harmful interference in the > > > kernel. The context suggested this pertained to one-node Ceph clusters, > > > that said I do not believe that was explicitly stated. Therefore, I very > > > much hope someone will be able to enlighten me whether: > > > > > > * this is indeed still the case with Tentacle, > > > > > > * it applies to one-node clusters only or all Ceph clusters, > > > > > > * it only applies to CephFS mounts using the kernel driver, or ceph-fuse > > > as well. > > > > > > Thank you in advance! > > > > > > -- > > > MS > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > -- > > May the most significant bit of your life be positive. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Stefan Möding 29 Jul '26

29 Jul '26

Marek Szuba via ceph-users <ceph-users(a)ceph.io> writes: > A few weeks ago I came across in Tentacle documentation on docs.ceph.org > (and OF COURSE now that I'm trying to quote it, I cannot find it any > more) a mention that CephFS should not be mounted on systems serving > Ceph because of there being a risk of harmful interference in the > kernel. The context suggested this pertained to one-node Ceph clusters, > that said I do not believe that was explicitly stated. Therefore, I very > much hope someone will be able to enlighten me whether: Maybe it was this page: https://docs.ceph.com/en/squid/rados/troubleshooting/troubleshooting-pg/ -- Stefan

1 0

Ceph Governing Board Meetings 2026-07-28
by Anthony Middleton 28 Jul '26

28 Jul '26

Hey Ceph friends, I hope you're doing well. Starting now, a portion of our Ceph Governing Board meetings will be recorded and made publicly available each month. This aims to increase transparency and encourage more community participation. You can watch the latest meeting and see the minutes attached below: https://www.youtube.com/watch?v=gPyu9KLVD64. As highlighted in the meeting, there are several committees you can get involved in, like Hardware, Ceph Days, Ceph Ambassadors, and Community Outreach. Feel free to reach out to me if you're interested in connecting with any of the committee leaders. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: v20.2.3 tentacle QE validation status
by Kamoltat Sirivadhna 28 Jul '26

28 Jul '26

RADOS Approved On Tue, Jul 28, 2026 at 1:00 PM Adam Emerson via ceph-users < ceph-users(a)ceph.io> wrote: > On 27/07/2026, Yuri Weinstein via dev wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/77256#note-5 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Kamoltat, Afreen, Travis > > > > rgw: Adam Emerson > > RGW Approved. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Kamoltat (Junior) Sirivadhna (HE/HIM) Senior SoftWare Engineer - IBM Ceph Storage ksirivad(a)redhat.com T: (857) <(919)716-5348>253-8927

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Gregory Farnum 28 Jul '26

28 Jul '26

On Tue, Jul 28, 2026 at 3:05 AM Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: > > I think the idea was that if you mount cephfs on a box running OSDs or a > mon daemon, and it gets into some kind of trouble like running out of > memory or so, then both the client side and the "server" side might be > competing for the same resources that aren't there, making it worse. This is a classic problem in network filesystems: 1) The kernel runs out of memory and needs to flush out buffered writes to free up space 2) The client sends out a network message, which comes back in to the server 3) The server needs to allocate memory to handle the incoming network message 4) ...stuck RBD and CephFS are susceptible to this, if you mount the kernels on the same hosts as the OSDs. But the odds of hitting it are a lot lower on modern systems with their RAM capacities than in the past. -Greg > > If you have separate OSD hosts and they are having issues, then cephfs > clients will be slow, but not much more than that, and if clients are > having resource dips, a separate cluster doesn't mind at all. I think that > is where the non-recommendation comes from, in case you do combine them, it > will be hard to get out of this situation, any cpu/ram given to the client > side would cause more IO to be requested, from the server side that doesn't > have it. In any case it might make it lots harder to diagnose which end > actually caused the issue if both suffer at the same time. > > > > Den tis 28 juli 2026 kl 14:45 skrev Marek Szuba via ceph-users < > ceph-users(a)ceph.io>: > > > Dear fellow Cephies, > > > > A few weeks ago I came across in Tentacle documentation on docs.ceph.org > > (and OF COURSE now that I'm trying to quote it, I cannot find it any > > more) a mention that CephFS should not be mounted on systems serving > > Ceph because of there being a risk of harmful interference in the > > kernel. The context suggested this pertained to one-node Ceph clusters, > > that said I do not believe that was explicitly stated. Therefore, I very > > much hope someone will be able to enlighten me whether: > > > > * this is indeed still the case with Tentacle, > > > > * it applies to one-node clusters only or all Ceph clusters, > > > > * it only applies to CephFS mounts using the kernel driver, or ceph-fuse > > as well. > > > > Thank you in advance! > > > > -- > > MS > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Malte Stroem 28 Jul '26

28 Jul '26

Hi, you can mount it on the Ceph nodes. You can mount it everywhere. Best, Malte On 7/28/26 14:48, Marek Szuba via ceph-users wrote: > Dear fellow Cephies, > > A few weeks ago I came across in Tentacle documentation on docs.ceph.org > (and OF COURSE now that I'm trying to quote it, I cannot find it any > more) a mention that CephFS should not be mounted on systems serving > Ceph because of there being a risk of harmful interference in the > kernel. The context suggested this pertained to one-node Ceph clusters, > that said I do not believe that was explicitly stated. Therefore, I very > much hope someone will be able to enlighten me whether: > > * this is indeed still the case with Tentacle, > > * it applies to one-node clusters only or all Ceph clusters, > > * it only applies to CephFS mounts using the kernel driver, or ceph-fuse > as well. > > Thank you in advance! >

1 0

Re: v20.2.3 tentacle QE validation status
by Adam Emerson 28 Jul '26

28 Jul '26

On 27/07/2026, Yuri Weinstein via dev wrote: > Details of this release are summarized here: > https://tracker.ceph.com/issues/77256#note-5 > > Release Notes - TBD > LRC upgrade - TBD > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Afreen, Travis > > rgw: Adam Emerson RGW Approved.

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Janne Johansson 28 Jul '26

28 Jul '26

I think the idea was that if you mount cephfs on a box running OSDs or a mon daemon, and it gets into some kind of trouble like running out of memory or so, then both the client side and the "server" side might be competing for the same resources that aren't there, making it worse. If you have separate OSD hosts and they are having issues, then cephfs clients will be slow, but not much more than that, and if clients are having resource dips, a separate cluster doesn't mind at all. I think that is where the non-recommendation comes from, in case you do combine them, it will be hard to get out of this situation, any cpu/ram given to the client side would cause more IO to be requested, from the server side that doesn't have it. In any case it might make it lots harder to diagnose which end actually caused the issue if both suffer at the same time. Den tis 28 juli 2026 kl 14:45 skrev Marek Szuba via ceph-users < ceph-users(a)ceph.io>: > Dear fellow Cephies, > > A few weeks ago I came across in Tentacle documentation on docs.ceph.org > (and OF COURSE now that I'm trying to quote it, I cannot find it any > more) a mention that CephFS should not be mounted on systems serving > Ceph because of there being a risk of harmful interference in the > kernel. The context suggested this pertained to one-node Ceph clusters, > that said I do not believe that was explicitly stated. Therefore, I very > much hope someone will be able to enlighten me whether: > > * this is indeed still the case with Tentacle, > > * it applies to one-node clusters only or all Ceph clusters, > > * it only applies to CephFS mounts using the kernel driver, or ceph-fuse > as well. > > Thank you in advance! > > -- > MS > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- May the most significant bit of your life be positive.

1 0

Mounting CephFS on Ceph nodes themselves
by Marek Szuba 28 Jul '26

28 Jul '26

Dear fellow Cephies, A few weeks ago I came across in Tentacle documentation on docs.ceph.org (and OF COURSE now that I'm trying to quote it, I cannot find it any more) a mention that CephFS should not be mounted on systems serving Ceph because of there being a risk of harmful interference in the kernel. The context suggested this pertained to one-node Ceph clusters, that said I do not believe that was explicitly stated. Therefore, I very much hope someone will be able to enlighten me whether: * this is indeed still the case with Tentacle, * it applies to one-node clusters only or all Ceph clusters, * it only applies to CephFS mounts using the kernel driver, or ceph-fuse as well. Thank you in advance! -- MS

1 0

Re: Ec pool min size
by GLE, Vivien 28 Jul '26

28 Jul '26

>That configuration is optimal for maximizing recovery times and >write latency: since each EC set will span 8 hosts of 9, every >time a host becames unavailable virtually all PGs will become >degraded, and even with all host available any write will >involve nearly all hosts. >This kind of optimal configuration is popular with those who >think that the original design assumption of Ceph of having >many, many more hosts and OSDs than the replication factor or >stripe width was wasteful and cowardly. :-) Isn't a bigger k+m a good things for performance if network and hardware not the bottleneck ? >Is there any particular reason why the failure domain is host >but pods with 3 hosts get shut down? I feel that it will give the same result as pods failure domain with the gain of loosing 3 hosts >Depends for what. But it is difficult to say without knowing the >expected data and workload profiles. Will be mostly bloc (virtual machine and kubernetes pods) >I might opine that in most cases it is as unimportant as having >'min_size=2' with 'size=3'. :-) >"Safely" and 'k=5' with 'min_size=5' is an interesting >association :-). Asking just in case but already knew the answer I guess >Perhaps 'k=2,m=2' or for the very brave 'k=4,m=2'. But again it >all depends on expected data and workload profiles. Just downsize the profile to 4+3 as this cluster is in test phase (I want to keep this failure host domain and I believe this is the best to do ?) ________________________________ De : Peter Grandi via ceph-users <ceph-users(a)ceph.io> Envoyé : lundi 27 juillet 2026 12:15:47 À : list Linux fs Ceph Objet : [ceph-users] Re: Ec pool min size > We have a 9 hosts (3 per pod) cluster with 1 pool in EC 5+3 > (host domain failure), That configuration is optimal for maximizing recovery times and write latency: since each EC set will span 8 hosts of 9, every time a host becames unavailable virtually all PGs will become degraded, and even with all host available any write will involve nearly all hosts. This kind of optimal configuration is popular with those who think that the original design assumption of Ceph of having many, many more hosts and OSDs than the replication factor or stripe width was wasteful and cowardly. :-) > I expected that shutting down 1 pod Is there any particular reason why the failure domain is host but pods with 3 hosts get shut down? > wouldn't affect our pg but I just discovered that the EC pool > is in min_size 6 while trying it. Is min_size 6 important in > this case ? Depends for what. But it is difficult to say without knowing the expected data and workload profiles. I might opine that in most cases it is as unimportant as having 'min_size=2' with 'size=3'. :-) > Could I safely lower it to 5 "Safely" and 'k=5' with 'min_size=5' is an interesting association :-). > or using another EC k+m would suit better? Perhaps 'k=2,m=2' or for the very brave 'k=4,m=2'. But again it all depends on expected data and workload profiles. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Ec pool min size
by Janne Johansson 28 Jul '26

28 Jul '26

> > > >That configuration is optimal for maximizing recovery times and > >write latency: since each EC set will span 8 hosts of 9, every > >time a host becames unavailable virtually all PGs will become > >degraded, and even with all host available any write will > >involve nearly all hosts. > > >This kind of optimal configuration is popular with those who > >think that the original design assumption of Ceph of having > >many, many more hosts and OSDs than the replication factor or > >stripe width was wasteful and cowardly. :-) > > > Isn't a bigger k+m a good things for performance if network and hardware > not the bottleneck ? > In some sense, but it also matters how many hosts you have, so it's more about "how close to degraded am I when everything is normal?" If you have 6+3 with only 10 hosts or 6+3 with 100 hosts matters a lot if one,two or three hosts accidentally goes down. So it's not only about if "6+3 is better than 4+2", but also how large part of your hosts this covers. -- May the most significant bit of your life be positive.

1 0

v20.2.3 tentacle QE validation status
by Yuri Weinstein 27 Jul '26

27 Jul '26

Details of this release are summarized here: https://tracker.ceph.com/issues/77256#note-5 Release Notes - TBD LRC upgrade - TBD *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* Seeking approvals/reviews for: smoke: PASSED rados: Kamoltat, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv, Vallari upgrade: Kamoltat, Patrick upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED Please reply with your approval or flag any concerns.

1 0

ingress service vrrp_interface_network bad effect
by Robert Sander 27 Jul '26

27 Jul '26

Hi, I am trying to deploy an ingress service with this spec: service_type: ingress service_id: rgw.default.default placement: label: ingress spec: backend_service: rgw.default.default virtual_ip: 10.166.166.205/24 frontend_port: 80 monitor_port: 8081 virtual_interface_networks: - 10.166.166.0/24 vrrp_interface_network: 10.104.130.0/24 The orchestrator tells me this: 2026-07-27T12:16:36.145458+0000 mgr.ceph103.xxazgi [INF] 10.166.166.205 is in 10.166.166.0/24 on ceph203 interface bondaccess.166 2026-07-27T12:16:36.145729+0000 mgr.ceph103.xxazgi [INF] 10.166.166.205 is in 10.166.166.0/24 on ceph204 interface bondaccess.166 2026-07-27T12:16:36.145809+0000 mgr.ceph103.xxazgi [INF] vrrp will be configured on ceph203 interface ['bondceph.1530'] (which is in subnet 10.104.130.0/24) 2026-07-27T12:16:36.147088+0000 mgr.ceph103.xxazgi [INF] Deploying daemon keepalived.rgw.default.default.ceph203.ztemnc on ceph203 2026-07-27T12:16:39.683591+0000 mgr.ceph103.xxazgi [INF] 10.166.166.205 is in 10.166.166.0/24 on ceph204 interface bondaccess.166 2026-07-27T12:16:39.683858+0000 mgr.ceph103.xxazgi [INF] 10.166.166.205 is in 10.166.166.0/24 on ceph203 interface bondaccess.166 2026-07-27T12:16:39.683958+0000 mgr.ceph103.xxazgi [INF] vrrp will be configured on ceph204 interface ['bondceph.1530'] (which is in subnet 10.104.130.0/24) 2026-07-27T12:16:39.684823+0000 mgr.ceph103.xxazgi [INF] Deploying daemon keepalived.rgw.default.default.ceph204.cpzqgk on ceph204 But the keepalived.conf contains this: interface bondceph.1530 unicast_src_ip 10.166.166.103 unicast_peer { 10.166.166.104 } which are the hosts' IPs in the network of the virtual IP. Why does the orchestrator not configure the IPs of the bondceph.1530 interface for VRRP? This results in both nodes having the virtual IP online as the backup keepalived does not get any VRRP packets from the first node. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: problem uploading dashboard certificates
by Eugen Block 27 Jul '26

27 Jul '26

Hi, the dashboard cert and key files are stored in: ceph config-key get mgr/dashboard/crt ceph config-key get mgr/dashboard/key Or something similar, maybe per host entries like (the "random" string is cephadm specific and won't apply in your case, of course): ceph config-key get mgr/dashboard/host1.random/crt ceph config-key get mgr/dashboard/host1.random/key You should be able to remove those faulty entries with ceph config-key rm mgr/dashboard/host1.random/crt ceph config-key rm mgr/dashboard/host1.random/key Then fail the mgr and inject your correct cert/key files again. I think that should work, if not, let us know. Regards, Eugen Zitat von Simon Oosthoek via ceph-users <ceph-users(a)ceph.io>: > Sorry, forgot to add info about the installation; this is ceph 17.2.x on > ubuntu 22.04.x, no cephadm and the cluster has about 360 OSDs of 12TB HDD. > > On Mon, 27 Jul 2026 at 11:50, Simon Oosthoek via ceph-users < > ceph-users(a)ceph.io> wrote: > >> Hi all, >> we've had some issues when Harica revoked a bunch of our certificates, I >> then accidentally uploaded mismatching key/pem files. When I wanted to fix >> that with the correct ones I got >> >> # ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem >> Error EIO: Module 'dashboard' has experienced an error and cannot handle >> commands: [('x509 certificate routines', '', 'key values mismatch')] >> >> Is there another way to fix this? >> >> Cheers, >> >> /Simon >> -- >> I'm using my gmail.com address, because the gmail.com dmarc policy is >> "none", some mail servers will reject this (microsoft?) others will instead >> allow this when I send mail to a mailling list which has not yet been >> configured to send mail "on behalf of" the sender, but rather do a kind of >> "forward". The latter situation causes dkim/dmarc failures and the dmarc >> policy will be applied. see https://wiki.list.org/DEV/DMARC for more >> details >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > > > -- > I'm using my gmail.com address, because the gmail.com dmarc policy is > "none", some mail servers will reject this (microsoft?) others will instead > allow this when I send mail to a mailling list which has not yet been > configured to send mail "on behalf of" the sender, but rather do a kind of > "forward". The latter situation causes dkim/dmarc failures and the dmarc > policy will be applied. see https://wiki.list.org/DEV/DMARC for more details > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Ec pool min size
by pg＠ceph.list.sabi.co.UK 27 Jul '26

27 Jul '26

> We have a 9 hosts (3 per pod) cluster with 1 pool in EC 5+3 > (host domain failure), That configuration is optimal for maximizing recovery times and write latency: since each EC set will span 8 hosts of 9, every time a host becames unavailable virtually all PGs will become degraded, and even with all host available any write will involve nearly all hosts. This kind of optimal configuration is popular with those who think that the original design assumption of Ceph of having many, many more hosts and OSDs than the replication factor or stripe width was wasteful and cowardly. :-) > I expected that shutting down 1 pod Is there any particular reason why the failure domain is host but pods with 3 hosts get shut down? > wouldn't affect our pg but I just discovered that the EC pool > is in min_size 6 while trying it. Is min_size 6 important in > this case ? Depends for what. But it is difficult to say without knowing the expected data and workload profiles. I might opine that in most cases it is as unimportant as having 'min_size=2' with 'size=3'. :-) > Could I safely lower it to 5 "Safely" and 'k=5' with 'min_size=5' is an interesting association :-). > or using another EC k+m would suit better? Perhaps 'k=2,m=2' or for the very brave 'k=4,m=2'. But again it all depends on expected data and workload profiles.

1 0

Re: problem uploading dashboard certificates
by Simon Oosthoek 27 Jul '26

27 Jul '26

Sorry, forgot to add info about the installation; this is ceph 17.2.x on ubuntu 22.04.x, no cephadm and the cluster has about 360 OSDs of 12TB HDD. On Mon, 27 Jul 2026 at 11:50, Simon Oosthoek via ceph-users < ceph-users(a)ceph.io> wrote: > Hi all, > we've had some issues when Harica revoked a bunch of our certificates, I > then accidentally uploaded mismatching key/pem files. When I wanted to fix > that with the correct ones I got > > # ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem > Error EIO: Module 'dashboard' has experienced an error and cannot handle > commands: [('x509 certificate routines', '', 'key values mismatch')] > > Is there another way to fix this? > > Cheers, > > /Simon > -- > I'm using my gmail.com address, because the gmail.com dmarc policy is > "none", some mail servers will reject this (microsoft?) others will instead > allow this when I send mail to a mailling list which has not yet been > configured to send mail "on behalf of" the sender, but rather do a kind of > "forward". The latter situation causes dkim/dmarc failures and the dmarc > policy will be applied. see https://wiki.list.org/DEV/DMARC for more > details > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

1 0

problem uploading dashboard certificates
by Simon Oosthoek 27 Jul '26

27 Jul '26

Hi all, we've had some issues when Harica revoked a bunch of our certificates, I then accidentally uploaded mismatching key/pem files. When I wanted to fix that with the correct ones I got # ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem Error EIO: Module 'dashboard' has experienced an error and cannot handle commands: [('x509 certificate routines', '', 'key values mismatch')] Is there another way to fix this? Cheers, /Simon -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

1 0

Re: subvolume in _nogroup
by Neeraj Pratap Singh 27 Jul '26

27 Jul '26

Hey Stephane, ---------- Forwarded message --------- From: Venky Shankar <vshankar(a)redhat.com> Date: Fri, Jul 24, 2026 at 11:19 PM Subject: Re: [ceph-users] subvolume in _nogroup To: Patrick Donnelly <pdonnell(a)redhat.com> Cc: Kotresh Hiremath Ravishankar <khiremat(a)redhat.com>, ceph-fs-team < ceph-fs-team(a)redhat.com>, Jos Collin <jcollin(a)redhat.com> It's possible the subvolume create command was invoked with an empty subvolume name. The subvolume group gets marked with ceph.dir.subvolume xattr. See: https://tracker.ceph.com/issues/68687 Workaround is to remove the xattr from the group directory. To check dump the group directory inode and check `is_subvolume`. Feel free to pass this along to the user, otherwise I will respond tomorrow. On Fri, Jul 24, 2026 at 10:56 PM Patrick Donnelly <pdonnell(a)redhat.com> wrote: > > Anyone have an idea on this issue and can help this poster? > > ---------- Forwarded message --------- > From: Stéphane Delmotte via ceph-users <ceph-users(a)ceph.io> > Date: Fri, Jul 24, 2026 at 9:35 AM > Subject: [ceph-users] subvolume in _nogroup > To: <ceph-users(a)ceph.io> > > > Hello, > > I find myself with a list of subvolumes in the|no_group|subvolume group, > which, if I understand correctly, is a special subvolume group. > > Actually, Manila created all these subvolumes without specifying a > subvolume group. > > Following an unexpected power outage and a restart, it no longer works. > > Manila can no longer create subvolumes, and I can no longer manipulate > my subvolumes. > > For example: > > content_copyCopy code > > |cephfssubvolumermcephfsbidulestef > ErrorEINVAL:invalidvaluespecifiedforceph.dir.subvolume| > > However, the state of my cluster is not critical. I only have: > > * 248 PGs not deep-scrubbed in time > * 517 PGs not scrubbed in time > > My Ceph version is: > > content_copyCopy code > > |cephversion19.3.0-4637-g42c1c3d4(42c1c3d42aaa0b6c9bcbccb51c8b397fb980aa9f) > squid (dev)| > > My CephFS mounts remain functional. > > If anyone has information about this, I’d appreciate it. Alternatively, > how can I continue creating subvolumes without specifying a subvolume group? > > Thanks, > > Stéphane > > > -- > Delmotte Stéphane > Laboratoire de Biométrie et Biologie Évolutive. UMR CNRS 5558 > Campus de La Doua - Université Claude Bernard - Lyon 1 > Bâtiment Grégoire Mendel - 1er étage > 16 rue Raphael Dubois > Villeurbanne 69100 > Tel : +33 06 82 87 93 59 > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > -- Cheers, Venky On Fri, Jul 24, 2026 at 7:05 PM Stéphane Delmotte via ceph-users < ceph-users(a)ceph.io> wrote: > Hello, > > I find myself with a list of subvolumes in the|no_group|subvolume group, > which, if I understand correctly, is a special subvolume group. > > Actually, Manila created all these subvolumes without specifying a > subvolume group. > > Following an unexpected power outage and a restart, it no longer works. > > Manila can no longer create subvolumes, and I can no longer manipulate > my subvolumes. > > For example: > > content_copyCopy code > > |cephfssubvolumermcephfsbidulestef > ErrorEINVAL:invalidvaluespecifiedforceph.dir.subvolume| > > However, the state of my cluster is not critical. I only have: > > * 248 PGs not deep-scrubbed in time > * 517 PGs not scrubbed in time > > My Ceph version is: > > content_copyCopy code > > |cephversion19.3.0-4637-g42c1c3d4(42c1c3d42aaa0b6c9bcbccb51c8b397fb980aa9f) > > squid (dev)| > > My CephFS mounts remain functional. > > If anyone has information about this, I’d appreciate it. Alternatively, > how can I continue creating subvolumes without specifying a subvolume > group? > > Thanks, > > Stéphane > > > -- > Delmotte Stéphane > Laboratoire de Biométrie et Biologie Évolutive. UMR CNRS 5558 > Campus de La Doua - Université Claude Bernard - Lyon 1 > Bâtiment Grégoire Mendel - 1er étage > 16 rue Raphael Dubois > Villeurbanne 69100 > Tel : +33 06 82 87 93 59 > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Unable to setup cloud sync to AWS eu-north-1 region
by Ieva Astašauskė 27 Jul '26

27 Jul '26

Hi, Ceph cluster: Multisite over 2 regions Per region: 6 OSD nodes, same nodes run RGW with rgw_run_sync_thread=true 1 dedicated RGW node serving client traffic (rgw_run_sync_thread=false) Trying to add a tier-type=cloud zone that would sync to AWS (based on https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/ and https://oneuptime.com/blog/post/2026-03-31-rook-cloud-sync-rgw-aws-s3/view ) Switched RGW on one OSD to sync that cloud zone, added IP address of that RGW node as zone endpoint sudo radosgw-admin zonegroup get { "id": "c1565bb4-acf9-4def-bfe1-7f189136c07b", "name": "poc-ms", "api_name": "poc-ms", "is_master": true, "endpoints": [ "http://ceph-cluster-poc1.ceph-radosgw.service.region1.consul:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "6e1e00de-a43d-4da9-994a-081041f6afb8", "zones": [ { "id": "3838f112-6804-4712-8af5-ab57a886e64c", "name": "poc2-region2", "endpoints": [ " http://ceph-cluster-poc2.ceph-radosgw.service.region2.consul:7480" ], "log_meta": false, "log_data": true, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "notification_v2", "resharding" ] }, { "id": "6e1e00de-a43d-4da9-994a-081041f6afb8", "name": "poc1-region1", "endpoints": [ " http://ceph-cluster-poc1.ceph-radosgw.service.region1.consul:7480" ], "log_meta": false, "log_data": true, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "notification_v2", "resharding" ] }, { "id": "8c988f68-975f-4c39-ab53-520f9fc6ee9e", "name": "poc-cloud-backup", "endpoints": [ "http://rgw_node_ip:7480" ], "log_meta": false, "log_data": true, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "cloud", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "notification_v2", "resharding" ] } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD", "STANDARD_IA" ] }, { "name": "ec-only", "tags": [], "storage_classes": [ "STANDARD", "STANDARD_IA" ] } ], "default_placement": "default-placement", "realm_id": "939ac8b7-8237-4a50-bea5-fed4b1dfd74d", "sync_policy": { "groups": [ { "id": "bucket-sync", "data_flow": { "symmetrical": [ { "id": "sym-region1-region2", "zones": [ "3838f112-6804-4712-8af5-ab57a886e64c", "6e1e00de-a43d-4da9-994a-081041f6afb8" ] } ], "directional": [ { "source_zone": "6e1e00de-a43d-4da9-994a-081041f6afb8", "dest_zone": "8c988f68-975f-4c39-ab53-520f9fc6ee9e" } ] }, "pipes": [ { "id": "pipe-all", "source": { "bucket": "*", "zones": [ "*" ] }, "dest": { "bucket": "*", "zones": [ "*" ] }, "params": { "source": { "filter": { "tags": [] } }, "dest": {}, "priority": 0, "mode": "system" } } ], "status": "enabled" } ] }, "enabled_features": [ "notification_v2", "resharding" ] } Verified that: - Cloud zone has correct AWS credentials (can PUT object directly to the bucket with same credentials) sudo radosgw-admin zone get --rgw-zone=poc-cloud-backup { "id": "8c988f68-975f-4c39-ab53-520f9fc6ee9e", "name": "poc-cloud-backup", "domain_root": "poc-cloud-backup.rgw.meta:root", "control_pool": "poc-cloud-backup.rgw.control", "dedup_pool": "poc-cloud-backup.rgw.dedup", "gc_pool": "poc-cloud-backup.rgw.log:gc", "lc_pool": "poc-cloud-backup.rgw.log:lc", "log_pool": "poc-cloud-backup.rgw.log", "intent_log_pool": "poc-cloud-backup.rgw.log:intent", "usage_log_pool": "poc-cloud-backup.rgw.log:usage", "roles_pool": "poc-cloud-backup.rgw.meta:roles", "reshard_pool": "poc-cloud-backup.rgw.log:reshard", "user_keys_pool": "poc-cloud-backup.rgw.meta:users.keys", "user_email_pool": "poc-cloud-backup.rgw.meta:users.email", "user_swift_pool": "poc-cloud-backup.rgw.meta:users.swift", "user_uid_pool": "poc-cloud-backup.rgw.meta:users.uid", "otp_pool": "poc-cloud-backup.rgw.otp", "notif_pool": "poc-cloud-backup.rgw.log:notif", "topics_pool": "poc-cloud-backup.rgw.meta:topics", "account_pool": "poc-cloud-backup.rgw.meta:accounts", "group_pool": "poc-cloud-backup.rgw.meta:groups", "bucket_logging_pool": "poc-cloud-backup.rgw.log:logging", "system_key": { "access_key": "XXX", "secret_key": "XXXXX" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "poc-cloud-backup.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "poc-cloud-backup.rgw.buckets.data-r3" }, "STANDARD_IA": { "data_pool": "poc-cloud-backup.rgw.buckets.data-ec42" } }, "data_extra_pool": "poc-cloud-backup.rgw.buckets.non-ec", "index_type": 0, "inline_data": true } }, { "key": "ec-only", "val": { "index_pool": "poc-cloud-backup.rgw.buckets.index", "storage_classes": { "STANDARD": {}, "STANDARD_IA": { "data_pool": "poc-cloud-backup.rgw.buckets.data-ec42" } }, "data_extra_pool": "poc-cloud-backup.rgw.buckets.non-ec", "index_type": 0, "inline_data": true } } ], "tier_config": { "connection": { "access_key": "XXX", "endpoint": "https://s3.eu-north-1.amazonaws.com", "host_style": "virtual", "region": "eu-north-1", "secret": "XXXXX" }, "target_path": "cloud-sync-test" }, "realm_id": "939ac8b7-8237-4a50-bea5-fed4b1dfd74d", "restore_pool": "poc-cloud-backup.rgw.log:restore" } Sync status seems OK sudo radosgw-admin bucket sync status --bucket=sync-test01 --rgw-zone=poc-cloud-backup realm 939ac8b7-8237-4a50-bea5-fed4b1dfd74d (default) zonegroup c1565bb4-acf9-4def-bfe1-7f189136c07b (poc-ms) zone 8c988f68-975f-4c39-ab53-520f9fc6ee9e (poc-cloud-backup) bucket :sync-test01[6e1e00de-a43d-4da9-994a-081041f6afb8.16963179.1 ]) current time 2026-07-24T13:48:45Z source zone 6e1e00de-a43d-4da9-994a-081041f6afb8 (poc1-region1) source bucket :sync-test01[6e1e00de-a43d-4da9-994a-081041f6afb8.16963179.1 ]) incremental sync on 11 shards bucket is caught up with source source zone 6e1e00de-a43d-4da9-994a-081041f6afb8 (poc1-region1) source bucket :sync-test01[6e1e00de-a43d-4da9-994a-081041f6afb8.16963179.1 ]) incremental sync on 11 shards bucket is caught up with source source zone 6e1e00de-a43d-4da9-994a-081041f6afb8 (poc1-region1) source bucket :sync-test01[6e1e00de-a43d-4da9-994a-081041f6afb8.16963179.1 ]) incremental sync on 11 shards bucket is caught up with source However, nothing appears on AWS And can’t find anything meaningful in RGW logs I keep adding new objects to the master zone every time I change some configs

1 0

Re: v21.1.0 Umbrella RC0 released
by Stefan Möding 27 Jul '26

27 Jul '26

Patrick Donnelly via ceph-users <ceph-users(a)ceph.io> writes: > Well that's unfortunate. I'm not going to make this become Ceph's > problem by retroactively supporting new Ubuntu releases on release > branches. It means multiplying our testing and dev workload when > maintaining a predictable cadence is already difficult. I'd rather > that we work on getting the new releases out on schedule which would > support the new Ubuntu release. I can understand that. I’m thinking about either migrating the cluster to Debian or using Podman. Thanks! -- Stefan

1 0

Re: v21.1.0 Umbrella RC0 released
by Patrick Donnelly 26 Jul '26

26 Jul '26

On Wed, Jul 22, 2026 at 10:33 AM Stefan Möding via ceph-users <ceph-users(a)ceph.io> wrote: > > Patrick Donnelly via ceph-users <ceph-users(a)ceph.io> writes: > > > Why is Canonical not updating/fixing these packages when the standard > > EOL for 24.04 is May 2029? > > The Qt5 packages are part of the Universe repo. This is not covered by > the standard (viz. free) 5 year. LTS covers only the Main repo. > > It requires an ESM contract to get support for the Universe repo. So > with the release of Ubuntu 26.04 the ongoing effort to keep the Universe > repo up to date has effectively been dropped. Well that's unfortunate. I'm not going to make this become Ceph's problem by retroactively supporting new Ubuntu releases on release branches. It means multiplying our testing and dev workload when maintaining a predictable cadence is already difficult. I'd rather that we work on getting the new releases out on schedule which would support the new Ubuntu release. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Andras Pataki 24 Jul '26

24 Jul '26

I'm very glad that we have a working theory of what is happening in the MDS - it seems like the client process experience, the logs and the theory on what the code is doing all match up. You mentioned that it takes some load/turnover on the MDS to trim the cache a specific way - this might be the reason I could not reproduce the issue on a test system. Our production MDS is quite busy with a couple thousand clients hammering on its cache constantly - so it probably does test most of the caching related code paths. When you have a patch that you feel is safe to try on a large production system, I can patch our MDS and test it on our workload. Thanks, Andras On 7/24/26 4:37 AM, Xiubo Li via ceph-users wrote: > Hi Andras, > > Thanks very much for your logs. This is very helpful. Following is > Dhairya's and my analysis: > > Your issue was most probably triggered by cases like: > > 1. COW deletes GSK: > items: snap dentry [2, 0x1c9] primary <-- preserved for snapshot > head dentry [0x1ca, CEPH_NOSNAP] NULL <-- COW correctly set first > > 2. Cache trims the COW head NULL dentry: > null + clean --> clear_complete = FALSE > --> STATE_COMPLETE preserved! <-- KEY! > --> items now only has snap dentry > > 3. A HEAD lookup comes in for GSK: > lookup("GSK", CEPH_NOSNAP) --> lower_bound(CEPH_NOSNAP, "GSK", h) > --> no head-level dentry found --> returns NULL > > MISS path: is_complete() = TRUE <-- still complete! > --> add_null_dentry("GSK") <-- default first=2! > --> creates [2, CEPH_NOSNAP] NULL > <------------------------------------ HERE ------------------ > > 4. Now items has: > snap dentry [2, 0x1c9] primary > NULL dentry [2, CEPH_NOSNAP] <-- first=2 covers ALL snapids > > 5. Cache trim --> trims the snap dentry: > primary --> clear_complete = TRUE --> STATE_COMPLETE cleared. This > could happen under workload pressure > > 6. Snap lookup comes in: > dir incomplete, only NULL dentry [2,head] left > 2 < 0x1c9 < CEPH_NOSNAP hit ENOENT ----------> in CDir: lookup() > it will just return the NULL dentry, which is [2, CPEH_NOSNAP] and > then the kclient cached it. > > So the root cause is in the MDS, and in kclient we still need to fix > kclient to prevent caching NULL dentry for this bug. > > Dhairya or I will fix the MDS side bug later. I have sent out the > kclient patches and will create a PR to fix the libcephfs later. > > Thanks & Regards > Xiubo > > Xiubo > > On Fri, 24 Jul 2026 at 13:07, Dhairya Parmar via dev <dev(a)ceph.io> wrote: >> Hey Andas, sorry to stress your MDS a bit and thanks a ton for the debug logs. This is really helpful. This is the most important bit in [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] and my suspicion is on [2,head] range. Let's see what I find. I'll keep you posted. BTW @Xiubo Li mentioned he isn't able to reproduce this locally. Would you be able to share the reproducer if possible? >> >> On Thu, Jul 23, 2026 at 10:19 PM Andras Pataki <apataki(a)flatironinstitute.org> wrote: >>> Hi Xiubo and Dhairya, >>> >>> I had a fruitful debugging session with my clients and MDS. The problem seems to be with the MDS, not with the client. >>> The incorrectly nonexistent files stay not existing after a client remount: >>> >>> [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >>> [root@cephmds611 ~]# systemctl restart mnt-ceph.mount >>> [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >>> ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log': No such file or directory >>> >>> In the MDS, the problem is when the 'GSK' directory is looked up as part of the path lookup. >>> client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK >>> >>> It seems like the MDS is looking in its cache, and instead of the snapshot it finds an entry for the live tree where there is no GSK directory. >>> Then the MDS returns ENOENT for the GSK directory in the snapshot (!) - so anything under GSK will be invisible to the client in the snapshot: >>> >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 >>> 400 caller_uid=0, caller_gid=0{0,}) v6 >>> >>> Here are the full logs: >>> >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 4 mds.0.server handle_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0,} >>> ) v6 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.5902540 get_session have 0x556382924000 client.265674856 v1:10.250.129.151:0/2806743593 state open >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.server oldest_client_tid=7 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_start request(client.265674856:7 nref=2 cr=0x555b0018b500) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server dispatch_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0 >>> ,}) v6 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server rdlock_path_pin_ref request(client.265674856:7 nref=2 cr=0x555b0018b500) #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache traverse: opening base ino 0x100131197c4 snap head path depth 5 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 0 '' snapid head >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snapdir >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 1 'sync-20260701-0904' snapid snapdir >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) resolve_snapname 'sync-20260701-0904' in [0,head] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) ? snap(1c9 0x100131197c4 'sync-20260701-0904' 2026-07-01T09:04:25.746531-0400) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snap sync-20260701-0904 -> 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 2 'bturzo' snapid 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) lookup (bturzo, '1c9') >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) hit -> (bturzo,head) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 3 'Projects' snapid 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) lookup (Projects, '1c9') >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) hit -> (Projects,head) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 4 'GSK' snapid 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 >>> 400 caller_uid=0, caller_gid=0{0,}) v6 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server apply_allocated_inos 0x0 / [] / 0x0 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server lat 0.575392 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist snapid 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.ino(0x101a2629a23) encode_inodestat snapid 1c9 to old_inode [1a7,1d3] n(v1256297 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003833=931566+72267) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snaps 1c9,1ec,1ee,1f0,1f2,1f4 (seq 1c9 cached_seq 1f5) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snap_info snaps 1c9,1ec,1ee,1f0,1f2,1f4 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snaps 1ec,1ee,1f0,1f2,1f4 (seq 1f5 cached_seq 1f5) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snap_info snaps 1ec,1ee,1f0,1f2,1f4 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.ino(0x101a2629a23) pfile 0 pauth 0 plink 0 pxattr 0 plocal 0 mtime 2026-06-08T14:02:36.442123-0400 ctime 2026-06-08T14:02:36.442123-0400 change_attr 196 valid=1 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.ino(0x101a2629a23) encode_inodestat caps pAsLsXsFs seq 0 mseq 0 xattrv 1 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added diri [inode 0x101a2629a23 [...1ea,head] /users/bturzo/Projects/ auth v22346457 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+41) n(v1256393 rc2026-07-17T12:32:38. >>> 513951-0400 b70376541394148 1003833=931566+72267) old_inodes=2 (inest lock) (iversion lock) caps={254564708=pAsLsXsFs/-@9,258742051=pAsLsXsFs/pAsLsXsFsxcral@3} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=0 wait >>> er=0 authpin=0 0x556f6adfe580] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added dir [dir 0x101a2629a23 /users/bturzo/Projects/ [2,head] auth v=26589887 cv=26589887/26589887 REP state=1074003968 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+4 >>> 1) n(v1256393 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003832=931566+72266) hs=45+1,ss=0+0 | child=1 dirty=0 waiter=0 authpin=0 0x555a65593a80] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.locker issue_client_lease no/null lease on [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.5902540 send_message_client client.265674856 v1:10.250.129.151:0/2806743593 client_reply(???:7 = -2 (2) No such file or directory) v1 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_finish request(client.265674856:7 nref=3 cr=0x555b0018b500) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache request_cleanup request(client.265674856:7 nref=3 cr=0x555b0018b500) >>> >>> The file system I'm reading for the copying is running ceph Reef 18.2.8, so this is a Reef MDS, in case the version/patches matter. >>> Thanks very much for looking into this - the help is appreciated! >>> >>> Andras >>> >>> >>> >>> On 7/22/26 8:28 PM, Xiubo Li wrote: >>> >>> Hi Andras, >>> >>> Thanks for your quick feedback. >>> >>> Sure. Yesterday Dhairya and I found something important in both the >>> MDS and kclient sides. Without the logs we will review the code to >>> ensure our investigation covers this case. We will provide feedback >>> later. >>> >>> Regards >>> Xiubo >>> >>> On Thu, 23 Jul 2026 at 00:14, Andras Pataki via ceph-users >>> <ceph-users(a)ceph.io> wrote: >>> >>> Hi Xiubo and Dhariya, >>> >>> The way I was able to reproduce the problem was to look at the error log >>> of my copy processes and take those files and check on them on the >>> various clients. Today, unfortunately, all those files seem to be >>> visible - and at the moment I don't have a way of reproducing the issue >>> from scratch. I'm trying to see what could be done to set up the >>> reproduction from scratch. If I manage to do that, I'll try to gather >>> some information on whether this is a client issue (i.e. flushing the >>> client caches changes anything) or whether it comes from the MDS. >>> Unfortunately the MDS in question is a very busy one, our main file >>> system MDS, so there are limits to what I can do to it. >>> >>> Andras >>> >>> >>> On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote: >>> >>> Hi Dhairya, >>> >>> Yeah, makes sense. >>> >>> Since I couldn't reproduce this locally, @Andras Pataki Could you help >>> reproduce it by enabling the debug logs as Dhairya mentioned? And if >>> possible please also try with the fuse client to see could you see >>> too. >>> >>> Thanks >>> Xiubo >>> >>> On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev(a)ceph.io> wrote: >>> >>> Hi Xiubo, >>> >>> I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds… . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT? >>> >>> >>> On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li(a)clyso.com> wrote: >>> >>> Hi Andras, Dhairya, >>> >>> I recall hitting a similar issue with this and the workaround involves >>> running the 'ls' command, but I can't recall exactly which command or >>> the tracker reference. This happened years ago. >>> >>> This morning I tried to reproduce this locally but failed. And I just >>> went through the mds and kclient code just found two suspect places >>> could cause this: >>> >>> MDS side: in https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/… >>> >>> MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: >>> >>> if (curdir->is_complete() || // (1) >>> (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) >>> return -ENOENT; >>> >>> is_complete() means all live entries are cached in memory. Shortcut >>> (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), >>> but shortcut (1) applies to all snapids — including snapshots. >>> >>> When a snapshot exists but the file was later deleted from the live >>> tree, the MDS eventually evicts the deleted file's dentry from the >>> in-memory cache (a null dentry, which trim_dentry() removes without >>> clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits >>> is_complete()=true with the dentry absent (ENOENT), without ever >>> checking the OSD where the snapshot data actually lives. >>> >>> READDIR takes a different code path that fetches from the OSD, which >>> is why ls -l fixes it. >>> >>> Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: >>> if (snapid == CEPH_NOSNAP && >>> (curdir->is_complete() || ...bloom...)) { >>> >>> @Andras >>> >>> BTW, to verify the above could you try to reproduce this by using the >>> ceph-fuse mount ? Locally I couldn't reproduce it by trying many times >>> and I am planing to add some debug code in MDS to ammulated it to >>> reproduce it later. >>> >>> ==== >>> >>> Kclient side: https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-li… >>> Plus a kclient defense: ceph_d_revalidate() unconditionally trusts >>> negative dentries in snapshot directories, caching a transient ENOENT >>> forever. Change valid = 1 to valid = inode ? 1 : 0. >>> >>> I just created on trakcer to follow this: https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-a… >>> >>> Thanks >>> Xiubo >>> >>> >>> >>> >>> On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users >>> <ceph-users(a)ceph.io> wrote: >>> >>> Hi Andras, >>> >>> This sounds like the client served the lookups from its cached negative? >>> Can you check if you see the lookup calls being made to MDS and maybe set >>> debug_mds=20 to check if the client did send the request but it was MDS >>> which returned ENOENT but since the `ls -l` succeeded I'm wondering if this >>> is actually something from MDS. Maybe try a sync and drop_caches to >>> validate that? >>> >>> *Dhairya Parmar* >>> >>> Software Engineer, CephFS >>> >>> >>> On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < >>> ceph-users(a)ceph.io> wrote: >>> >>> Hi ceph users/devs, >>> >>> We've been encountering a strange issue with snapshots as part of a >>> project to copy our large ceph file system to a backup in parallel >>> across nodes. We are taking a snapshot, and then doing a copy of the >>> snapshot, which we expect to be static (unchanging) - using several >>> clients in parallel - all kernel mounts. Some clients are unable to >>> open files and directories that are known to exist (were discovered >>> traversing the file system on perhaps another node). >>> After some playing around, I could reproduce the issue on a single node >>> with just 'ls' as follows: >>> >>> # This file doesn't seems to exist - and some number of parent directories >>> are also "missing": [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >>> No such file or directory # Until we get to this 'Projects' parent >>> directory, which seems to be there [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 >>> 1961 1961 46 Jun 8 14:02 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it >>> is still not there ... [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >>> No such file or directory # But ... let's see if the "tickle" the Projects >>> directory a bit ... >>> [root@cephmds603 ~]# ls -l >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null >>> >>> # Now all subdirectories and the file in question immediately becomes >>> visible: >>> [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >>> drwxrwsr-x 5 1961 1961 3 May 7 17:18 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >>> [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >>> drwxrwsr-x 12 1961 1961 10 May 11 16:08 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >>> [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >>> drwxrwsr-x 5 1961 1961 3 May 11 16:18 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >>> [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >>> -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >>> >>> # This is the kernel client 6.18.32 (relatively recent LTS kernel) with >>> ceph squid 19.2.3: >>> [root@cephmds603 ~]# uname -a >>> Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 >>> x86_64 x86_64 GNU/Linux >>> [root@cephmds603 ~]# ceph --version >>> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid >>> (stable) >>> [root@cephmds603 ~]# >>> >>> What is most interesting is that in the live tree (non-snapshotted tree) - >>> the Projects directory exists, but everything under it has been removed! >>> >>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ >>> drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ >>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK >>> ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or >>> directory >>> >>> Am I running into some known issue? This looks to be specific to some >>> clients - i.e. some clients see the directory and some don't at the same >>> point in time, i.e. this is perhaps a kclient problem? >>> Any ideas/suggestions would be appreciated, >>> >>> Andras >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

subvolume in _nogroup
by Stéphane Delmotte 24 Jul '26

24 Jul '26

Hello, I find myself with a list of subvolumes in the|no_group|subvolume group, which, if I understand correctly, is a special subvolume group. Actually, Manila created all these subvolumes without specifying a subvolume group. Following an unexpected power outage and a restart, it no longer works. Manila can no longer create subvolumes, and I can no longer manipulate my subvolumes. For example: content_copyCopy code |cephfssubvolumermcephfsbidulestef ErrorEINVAL:invalidvaluespecifiedforceph.dir.subvolume| However, the state of my cluster is not critical. I only have: * 248 PGs not deep-scrubbed in time * 517 PGs not scrubbed in time My Ceph version is: content_copyCopy code |cephversion19.3.0-4637-g42c1c3d4(42c1c3d42aaa0b6c9bcbccb51c8b397fb980aa9f) squid (dev)| My CephFS mounts remain functional. If anyone has information about this, I’d appreciate it. Alternatively, how can I continue creating subvolumes without specifying a subvolume group? Thanks, Stéphane -- Delmotte Stéphane Laboratoire de Biométrie et Biologie Évolutive. UMR CNRS 5558 Campus de La Doua - Université Claude Bernard - Lyon 1 Bâtiment Grégoire Mendel - 1er étage 16 rue Raphael Dubois Villeurbanne 69100 Tel : +33 06 82 87 93 59

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Redouane Kachach 24 Jul '26

24 Jul '26

Just to add: although assigning the _admin label to many hosts may be convenient, it also increases the security exposure. Cephadm distributes the highly privileged client.admin keyring to every _admin host, so an attacker who compromises one of those hosts and gains access to the keyring could obtain full administrative control of the cluster and cause significant damage. I would therefore recommend limiting the _admin label to a small, carefully controlled and hardened set of hosts that are explicitly used for cluster administration. On Fri, Jul 24, 2026 at 1:58 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > > > > On Jul 24, 2026, at 7:23 AM, Michel Le Cocq via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > The point I wanted to explore is the following: > > > > We have three nodes for the MON MGR and MDS roles. > > > > To bootstrap the cluster, we installed cephadm via apt only on the first > > node (M1). > > > > Then, we copied the ceph.pub key to the second node (M2) and the third > > node (M3). > > > > After that, we ran the following commands on M1: > > > > ceph orch host add m2 <IP> > > ceph orch host add m3 <IP> > > > > Note: cephadm is not installed on M2 and M3. > > Eugen covered this, but I'll try to help: > > The copy of cephadm that you installed on M1 was used *only* to bootstrap > the cluster. > You don't use that copy ever again. > The bootstrap process and "ceph orch host add" install *another*, > different copy of cephadm in a different location. > You could "apt delete cephadm" on M1 and nothing would change. > > This admittedly is a bit confusing. > > > Question: > > > > What happens if M1 fails completely? > > Nothing. > > > > > > In this scenario, we would need the file: > > /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} > > to be able to launch the cephadm shell and recover the cluster. > > That is installed on every cluster node. But you don't strictly need the > cephadm shell. > > What you do want to do is > > ceph orch host label add m1 _admin > ceph orch host label add m2 _admin > ceph orch host label add m3_admin > > and ensure that your orch services are set up to let mon, mgr, prometheus, > grafana etc. run on all three nodes. > > Once you have these additional hosts added to the cluster, M2 and M3 are > no different from M1. > > And assuming that your pools in this scenario are replicated size=3, which > is default, if M1 goes up in smoke you don't need to recover the cluster > because the cluster and all services will continue serving data. Your > redundancy will be degraded until you get a third host back in, but the > cluster itself will be fine. > > >>> > >>> Is it a good idee to do on host : > >>> > >>> # apt purge cephadm > > > That doesn't hurt, but it isn't necessary unless you want to run the > cephadm shell. > > >>> # chmod +x /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} > >>> # ln -s /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} /root/cephadm > > Nononono. The orchestrator goes to great lengths to free you from > micromanagement. > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Anthony D'Atri 24 Jul '26

24 Jul '26

> On Jul 24, 2026, at 7:23 AM, Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> wrote: > > The point I wanted to explore is the following: > > We have three nodes for the MON MGR and MDS roles. > > To bootstrap the cluster, we installed cephadm via apt only on the first > node (M1). > > Then, we copied the ceph.pub key to the second node (M2) and the third > node (M3). > > After that, we ran the following commands on M1: > > ceph orch host add m2 <IP> > ceph orch host add m3 <IP> > > Note: cephadm is not installed on M2 and M3. Eugen covered this, but I'll try to help: The copy of cephadm that you installed on M1 was used *only* to bootstrap the cluster. You don't use that copy ever again. The bootstrap process and "ceph orch host add" install *another*, different copy of cephadm in a different location. You could "apt delete cephadm" on M1 and nothing would change. This admittedly is a bit confusing. > Question: > > What happens if M1 fails completely? Nothing. > > In this scenario, we would need the file: > /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} > to be able to launch the cephadm shell and recover the cluster. That is installed on every cluster node. But you don't strictly need the cephadm shell. What you do want to do is ceph orch host label add m1 _admin ceph orch host label add m2 _admin ceph orch host label add m3_admin and ensure that your orch services are set up to let mon, mgr, prometheus, grafana etc. run on all three nodes. Once you have these additional hosts added to the cluster, M2 and M3 are no different from M1. And assuming that your pools in this scenario are replicated size=3, which is default, if M1 goes up in smoke you don't need to recover the cluster because the cluster and all services will continue serving data. Your redundancy will be degraded until you get a third host back in, but the cluster itself will be fine. >>> >>> Is it a good idee to do on host : >>> >>> # apt purge cephadm That doesn't hurt, but it isn't necessary unless you want to run the cephadm shell. >>> # chmod +x /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} >>> # ln -s /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} /root/cephadm Nononono. The orchestrator goes to great lengths to free you from micromanagement.

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 24 Jul '26

24 Jul '26

The point I wanted to explore is the following: We have three nodes for the MON MGR and MDS roles. To bootstrap the cluster, we installed cephadm via apt only on the first node (M1). Then, we copied the ceph.pub key to the second node (M2) and the third node (M3). After that, we ran the following commands on M1: ceph orch host add m2 <IP> ceph orch host add m3 <IP> Note: cephadm is not installed on M2 and M3. Question: What happens if M1 fails completely? In this scenario, we would need the file: /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} to be able to launch the cephadm shell and recover the cluster. -- M Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > There’s no need to do that. Making it executable would be only required if you > wanted to use that binary manually. The difference between the package and > container version are negligible in this case. We still have cephadm version > 16.2.13 installed on the hosts while the cluster is running 19.2.3. > You’re fine with this minor difference between versions. > > Zitat von Michel Le Cocq <lecocq(a)ipgp.fr>: > >> Hi, thanks for your answer. >> >> I effectively have on my host : >> /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} >> >> But it's not executable ! >> I did a chmod +x ... >> >> Then I can see that debian pkg version and the one in /var/lib are not >> the same ... >> >> For Debian : >> cephadm version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef >> (stable) >> >> For Ceph in /var/lib ... >> cephadm version 18.2.8 (efac5a54607c13fa50d4822e50242b86e6e446df) reef >> (stable) >> >> Is it a good idee to do on host : >> >> # apt purge cephadm >> # chmod +x /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} >> # ln -s /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} /root/cephadm >> >> Thanks. >> >> -- >> Michel >> >> Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: >> >>> Hi, responses inline... >>> >>> >>> Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: >>> >>>> Actually, what confused me was this: >>>> >>>> cephadm bootstrap --mon-ip <my-ip> >>>> >>>> Among other things, this command: >>>> - Generates a new SSH key for the Ceph cluster and adds it >>>> to the /root/.ssh/authorized_keys file. >>>> - Saves a copy of the public key to the /etc/ceph/ceph.pub file. >>>> >>>> You then need to add a node. >>>> >>>> ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> >>>> >>>> ceph orch host add <newhost> [<ip>] [<label1> ...] >>>> >>>> I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! >>> >>> That's stored in the config-key db: >>> >>> ceph config-key get mgr/cephadm/ssh_identity_key >>> >>>> >>>> -- >>>> M >>>> >>>> Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: >>>> >>>>> Hi again. >>>>> >>>>> I'm answering my own question. >>>>> >>>>> It seems I read the wrong version of the doc >>>>> ... sic! >>>>> >>>>> It should be enough to set the “_admin” flag on all nodes: >>>>> >>>>> and that we also need to install cephadm on the nodes we >>>>> want to use for administration, right!? >>> >>> Not necessarily, but it can be more convenient for debugging. The >>> orchestrator >>> stores a cephadm binary on each node to be able to manage them, it is stored >>> in >>> /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}. That's used to remotely >>> orchestrate >>> the cluster/nodes etc. The original bootstrap node requires a cephadm binary >>> either installed as a package or downloaded via curl as described in the >>> docs to >>> be able to bootstrap. After that you'll only need to execute cephadm >>> commands >>> locally on the nodes if you're debugging issues etc., otherwise the >>> orchestrator >>> will only use the file in /var/lib/ceph. >>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 24 Jul '26

24 Jul '26

Hi, thanks for your answer. I effectively have on my host : /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} But it's not executable ! I did a chmod +x ... Then I can see that debian pkg version and the one in /var/lib are not the same ... For Debian : cephadm version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) For Ceph in /var/lib ... cephadm version 18.2.8 (efac5a54607c13fa50d4822e50242b86e6e446df) reef (stable) Is it a good idee to do on host : # apt purge cephadm # chmod +x /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} # ln -s /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} /root/cephadm Thanks. -- Michel Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > Hi, responses inline... > > > Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > >> Actually, what confused me was this: >> >> cephadm bootstrap --mon-ip <my-ip> >> >> Among other things, this command: >> - Generates a new SSH key for the Ceph cluster and adds it >> to the /root/.ssh/authorized_keys file. >> - Saves a copy of the public key to the /etc/ceph/ceph.pub file. >> >> You then need to add a node. >> >> ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> >> >> ceph orch host add <newhost> [<ip>] [<label1> ...] >> >> I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! > > That's stored in the config-key db: > > ceph config-key get mgr/cephadm/ssh_identity_key > >> >> -- >> M >> >> Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: >> >>> Hi again. >>> >>> I'm answering my own question. >>> >>> It seems I read the wrong version of the doc >>> ... sic! >>> >>> It should be enough to set the “_admin” flag on all nodes: >>> >>> and that we also need to install cephadm on the nodes we >>> want to use for administration, right!? > > Not necessarily, but it can be more convenient for debugging. The orchestrator > stores a cephadm binary on each node to be able to manage them, it is stored in > /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}. That's used to remotely orchestrate > the cluster/nodes etc. The original bootstrap node requires a cephadm binary > either installed as a package or downloaded via curl as described in the docs to > be able to bootstrap. After that you'll only need to execute cephadm commands > locally on the nodes if you're debugging issues etc., otherwise the orchestrator > will only use the file in /var/lib/ceph. > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

July Ceph Science User Group Virtual Meeting
by Thomas Byrne - STFC UKRI 24 Jul '26

24 Jul '26

Hi all, We will be having a Ceph science/research/big cluster call next Tuesday (July 28th). If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc. We usually start each meeting with a brief 10-15 minutes talk given by a community member, but currently don't have one scheduled for this meeting. If you are interested in signing up to give a short talk for a meeting please let me know! It is a very informal talk. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20260728 Virtual event details: July 28th, 2026 14:00 UTC See the Ceph Community Calendar<https://ceph.io/en/community/meetups/> for the series information. (please double check the meeting time for your timezone!) Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph YouTube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Thanks, Tom

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Xiubo Li 24 Jul '26

24 Jul '26

Hi Andras, Thanks very much for your logs. This is very helpful. Following is Dhairya's and my analysis: Your issue was most probably triggered by cases like: 1. COW deletes GSK: items: snap dentry [2, 0x1c9] primary <-- preserved for snapshot head dentry [0x1ca, CEPH_NOSNAP] NULL <-- COW correctly set first 2. Cache trims the COW head NULL dentry: null + clean --> clear_complete = FALSE --> STATE_COMPLETE preserved! <-- KEY! --> items now only has snap dentry 3. A HEAD lookup comes in for GSK: lookup("GSK", CEPH_NOSNAP) --> lower_bound(CEPH_NOSNAP, "GSK", h) --> no head-level dentry found --> returns NULL MISS path: is_complete() = TRUE <-- still complete! --> add_null_dentry("GSK") <-- default first=2! --> creates [2, CEPH_NOSNAP] NULL <------------------------------------ HERE ------------------ 4. Now items has: snap dentry [2, 0x1c9] primary NULL dentry [2, CEPH_NOSNAP] <-- first=2 covers ALL snapids 5. Cache trim --> trims the snap dentry: primary --> clear_complete = TRUE --> STATE_COMPLETE cleared. This could happen under workload pressure 6. Snap lookup comes in: dir incomplete, only NULL dentry [2,head] left 2 < 0x1c9 < CEPH_NOSNAP hit ENOENT ----------> in CDir: lookup() it will just return the NULL dentry, which is [2, CPEH_NOSNAP] and then the kclient cached it. So the root cause is in the MDS, and in kclient we still need to fix kclient to prevent caching NULL dentry for this bug. Dhairya or I will fix the MDS side bug later. I have sent out the kclient patches and will create a PR to fix the libcephfs later. Thanks & Regards Xiubo Xiubo On Fri, 24 Jul 2026 at 13:07, Dhairya Parmar via dev <dev(a)ceph.io> wrote: > > Hey Andas, sorry to stress your MDS a bit and thanks a ton for the debug logs. This is really helpful. This is the most important bit in [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] and my suspicion is on [2,head] range. Let's see what I find. I'll keep you posted. BTW @Xiubo Li mentioned he isn't able to reproduce this locally. Would you be able to share the reproducer if possible? > > On Thu, Jul 23, 2026 at 10:19 PM Andras Pataki <apataki(a)flatironinstitute.org> wrote: >> >> Hi Xiubo and Dhairya, >> >> I had a fruitful debugging session with my clients and MDS. The problem seems to be with the MDS, not with the client. >> The incorrectly nonexistent files stay not existing after a client remount: >> >> [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >> [root@cephmds611 ~]# systemctl restart mnt-ceph.mount >> [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >> ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log': No such file or directory >> >> In the MDS, the problem is when the 'GSK' directory is looked up as part of the path lookup. >> client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK >> >> It seems like the MDS is looking in its cache, and instead of the snapshot it finds an entry for the live tree where there is no GSK directory. >> Then the MDS returns ENOENT for the GSK directory in the snapshot (!) - so anything under GSK will be invisible to the client in the snapshot: >> >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 >> 400 caller_uid=0, caller_gid=0{0,}) v6 >> >> Here are the full logs: >> >> 2026-07-23T10:34:53.281-0400 7ffff205a640 4 mds.0.server handle_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0,} >> ) v6 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.5902540 get_session have 0x556382924000 client.265674856 v1:10.250.129.151:0/2806743593 state open >> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.server oldest_client_tid=7 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_start request(client.265674856:7 nref=2 cr=0x555b0018b500) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server dispatch_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0 >> ,}) v6 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server rdlock_path_pin_ref request(client.265674856:7 nref=2 cr=0x555b0018b500) #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache traverse: opening base ino 0x100131197c4 snap head path depth 5 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 0 '' snapid head >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snapdir >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 1 'sync-20260701-0904' snapid snapdir >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) resolve_snapname 'sync-20260701-0904' in [0,head] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) ? snap(1c9 0x100131197c4 'sync-20260701-0904' 2026-07-01T09:04:25.746531-0400) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snap sync-20260701-0904 -> 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 2 'bturzo' snapid 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) lookup (bturzo, '1c9') >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) hit -> (bturzo,head) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 3 'Projects' snapid 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) lookup (Projects, '1c9') >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) hit -> (Projects,head) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 4 'GSK' snapid 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 >> 400 caller_uid=0, caller_gid=0{0,}) v6 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server apply_allocated_inos 0x0 / [] / 0x0 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server lat 0.575392 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist snapid 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.ino(0x101a2629a23) encode_inodestat snapid 1c9 to old_inode [1a7,1d3] n(v1256297 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003833=931566+72267) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snaps 1c9,1ec,1ee,1f0,1f2,1f4 (seq 1c9 cached_seq 1f5) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snap_info snaps 1c9,1ec,1ee,1f0,1f2,1f4 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snaps 1ec,1ee,1f0,1f2,1f4 (seq 1f5 cached_seq 1f5) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snap_info snaps 1ec,1ee,1f0,1f2,1f4 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.ino(0x101a2629a23) pfile 0 pauth 0 plink 0 pxattr 0 plocal 0 mtime 2026-06-08T14:02:36.442123-0400 ctime 2026-06-08T14:02:36.442123-0400 change_attr 196 valid=1 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.ino(0x101a2629a23) encode_inodestat caps pAsLsXsFs seq 0 mseq 0 xattrv 1 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added diri [inode 0x101a2629a23 [...1ea,head] /users/bturzo/Projects/ auth v22346457 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+41) n(v1256393 rc2026-07-17T12:32:38. >> 513951-0400 b70376541394148 1003833=931566+72267) old_inodes=2 (inest lock) (iversion lock) caps={254564708=pAsLsXsFs/-@9,258742051=pAsLsXsFs/pAsLsXsFsxcral@3} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=0 wait >> er=0 authpin=0 0x556f6adfe580] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added dir [dir 0x101a2629a23 /users/bturzo/Projects/ [2,head] auth v=26589887 cv=26589887/26589887 REP state=1074003968 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+4 >> 1) n(v1256393 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003832=931566+72266) hs=45+1,ss=0+0 | child=1 dirty=0 waiter=0 authpin=0 0x555a65593a80] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.locker issue_client_lease no/null lease on [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.5902540 send_message_client client.265674856 v1:10.250.129.151:0/2806743593 client_reply(???:7 = -2 (2) No such file or directory) v1 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_finish request(client.265674856:7 nref=3 cr=0x555b0018b500) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache request_cleanup request(client.265674856:7 nref=3 cr=0x555b0018b500) >> >> The file system I'm reading for the copying is running ceph Reef 18.2.8, so this is a Reef MDS, in case the version/patches matter. >> Thanks very much for looking into this - the help is appreciated! >> >> Andras >> >> >> >> On 7/22/26 8:28 PM, Xiubo Li wrote: >> >> Hi Andras, >> >> Thanks for your quick feedback. >> >> Sure. Yesterday Dhairya and I found something important in both the >> MDS and kclient sides. Without the logs we will review the code to >> ensure our investigation covers this case. We will provide feedback >> later. >> >> Regards >> Xiubo >> >> On Thu, 23 Jul 2026 at 00:14, Andras Pataki via ceph-users >> <ceph-users(a)ceph.io> wrote: >> >> Hi Xiubo and Dhariya, >> >> The way I was able to reproduce the problem was to look at the error log >> of my copy processes and take those files and check on them on the >> various clients. Today, unfortunately, all those files seem to be >> visible - and at the moment I don't have a way of reproducing the issue >> from scratch. I'm trying to see what could be done to set up the >> reproduction from scratch. If I manage to do that, I'll try to gather >> some information on whether this is a client issue (i.e. flushing the >> client caches changes anything) or whether it comes from the MDS. >> Unfortunately the MDS in question is a very busy one, our main file >> system MDS, so there are limits to what I can do to it. >> >> Andras >> >> >> On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote: >> >> Hi Dhairya, >> >> Yeah, makes sense. >> >> Since I couldn't reproduce this locally, @Andras Pataki Could you help >> reproduce it by enabling the debug logs as Dhairya mentioned? And if >> possible please also try with the fuse client to see could you see >> too. >> >> Thanks >> Xiubo >> >> On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev(a)ceph.io> wrote: >> >> Hi Xiubo, >> >> I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds… . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT? >> >> >> On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li(a)clyso.com> wrote: >> >> Hi Andras, Dhairya, >> >> I recall hitting a similar issue with this and the workaround involves >> running the 'ls' command, but I can't recall exactly which command or >> the tracker reference. This happened years ago. >> >> This morning I tried to reproduce this locally but failed. And I just >> went through the mds and kclient code just found two suspect places >> could cause this: >> >> MDS side: in https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/… >> >> MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: >> >> if (curdir->is_complete() || // (1) >> (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) >> return -ENOENT; >> >> is_complete() means all live entries are cached in memory. Shortcut >> (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), >> but shortcut (1) applies to all snapids — including snapshots. >> >> When a snapshot exists but the file was later deleted from the live >> tree, the MDS eventually evicts the deleted file's dentry from the >> in-memory cache (a null dentry, which trim_dentry() removes without >> clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits >> is_complete()=true with the dentry absent (ENOENT), without ever >> checking the OSD where the snapshot data actually lives. >> >> READDIR takes a different code path that fetches from the OSD, which >> is why ls -l fixes it. >> >> Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: >> if (snapid == CEPH_NOSNAP && >> (curdir->is_complete() || ...bloom...)) { >> >> @Andras >> >> BTW, to verify the above could you try to reproduce this by using the >> ceph-fuse mount ? Locally I couldn't reproduce it by trying many times >> and I am planing to add some debug code in MDS to ammulated it to >> reproduce it later. >> >> ==== >> >> Kclient side: https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-li… >> Plus a kclient defense: ceph_d_revalidate() unconditionally trusts >> negative dentries in snapshot directories, caching a transient ENOENT >> forever. Change valid = 1 to valid = inode ? 1 : 0. >> >> I just created on trakcer to follow this: https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-a… >> >> Thanks >> Xiubo >> >> >> >> >> On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users >> <ceph-users(a)ceph.io> wrote: >> >> Hi Andras, >> >> This sounds like the client served the lookups from its cached negative? >> Can you check if you see the lookup calls being made to MDS and maybe set >> debug_mds=20 to check if the client did send the request but it was MDS >> which returned ENOENT but since the `ls -l` succeeded I'm wondering if this >> is actually something from MDS. Maybe try a sync and drop_caches to >> validate that? >> >> *Dhairya Parmar* >> >> Software Engineer, CephFS >> >> >> On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < >> ceph-users(a)ceph.io> wrote: >> >> Hi ceph users/devs, >> >> We've been encountering a strange issue with snapshots as part of a >> project to copy our large ceph file system to a backup in parallel >> across nodes. We are taking a snapshot, and then doing a copy of the >> snapshot, which we expect to be static (unchanging) - using several >> clients in parallel - all kernel mounts. Some clients are unable to >> open files and directories that are known to exist (were discovered >> traversing the file system on perhaps another node). >> After some playing around, I could reproduce the issue on a single node >> with just 'ls' as follows: >> >> # This file doesn't seems to exist - and some number of parent directories >> are also "missing": [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >> No such file or directory # Until we get to this 'Projects' parent >> directory, which seems to be there [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 >> 1961 1961 46 Jun 8 14:02 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it >> is still not there ... [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >> No such file or directory # But ... let's see if the "tickle" the Projects >> directory a bit ... >> [root@cephmds603 ~]# ls -l >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null >> >> # Now all subdirectories and the file in question immediately becomes >> visible: >> [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >> drwxrwsr-x 5 1961 1961 3 May 7 17:18 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >> [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> drwxrwsr-x 12 1961 1961 10 May 11 16:08 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> drwxrwsr-x 5 1961 1961 3 May 11 16:18 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> >> # This is the kernel client 6.18.32 (relatively recent LTS kernel) with >> ceph squid 19.2.3: >> [root@cephmds603 ~]# uname -a >> Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 >> x86_64 x86_64 GNU/Linux >> [root@cephmds603 ~]# ceph --version >> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid >> (stable) >> [root@cephmds603 ~]# >> >> What is most interesting is that in the live tree (non-snapshotted tree) - >> the Projects directory exists, but everything under it has been removed! >> >> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ >> drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ >> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK >> ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or >> directory >> >> Am I running into some known issue? This looks to be specific to some >> clients - i.e. some clients see the directory and some don't at the same >> point in time, i.e. this is perhaps a kclient problem? >> Any ideas/suggestions would be appreciated, >> >> Andras >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >>

1 0

Re: Ubuntu 24.04 with Tentacle
by Redouane Kachach 24 Jul '26

24 Jul '26

Regarding using packages vs cephadm I'd just like to add that cephadm + Podman has supported critical loads in production for several years now. Operations such as upgrading to a new version are generally much easier when using cephadm. Best, Redo. On Fri, Jul 24, 2026 at 3:16 AM Szabo, Istvan (Agoda) via ceph-users < ceph-users(a)ceph.io> wrote: > Thank you very much, this makes me thinking again should I use cephadm or > should we still stay with the manual setup and package based update for a > heavy loaded critical block storage cluster :/ > > Get Outlook for Android<https://aka.ms/AAb9ysg> > ________________________________ > From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> > Sent: Thursday, 23 July 2026 18:33:59 > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Ubuntu 24.04 with Tentacle > > Email received from the internet. If in doubt, don't click any link nor > open any attachment ! > ________________________________ > > On 2026-07-23 07:06, Szabo, Istvan (Agoda) via ceph-users wrote: > > > Reading throught the Ceph documentation regarding different versions and > I see here > https://docs.ceph.com/en/latest/start/os-recommendations/#platforms and > "Upcoming" text. > > What does this mean exactly, which release is going to be "A" on Ubuntu > 24.04? > > Hi Istvan, > > Not really the answer you have asked for because I am not a Ceph > developer, we have however got some experience with running Tentacle on > Noble so perhaps it will come in handy to you. > > We have in recent months upgraded all of our Ceph servers, in both > production and development clusters, from Jammy to Noble. This has been > originally for two reasons: > > - for several reasons I shan't bore you with we currently use Podman > rather than Docker with cephadm, and it transpired that the Jammy > version of Podman does not support namespace settings required by > multi-container services like the cephadm-managed SMB gateways; > > - since the upgrade from Squid to Tentacle one of our cluster kept > seeing CPU lock-ups (and subsequent "slow BlueStore operations" > warnings) progressively appear for more and more OSDs, and having tried > to no avail a lot of different configuration tweaks we got to the point > where the remaining choices were "upgrade to a newer kernel" and "start > replacing HBAs". > > So far - touch wood! - all the upgraded systems have been operating > perfectly as far as Ceph is concerned. > > -- > MS > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ________________________________ > This message is confidential and is for the sole use of the intended > recipient(s). It may also be privileged or otherwise protected by copyright > or other legal rules. If you have received it by mistake please let us know > by reply email and delete it from your system. It is prohibited to copy > this message or disclose its content to anyone. Any confidentiality or > privilege is not waived or lost by any mistaken delivery or unauthorized > disclosure of the message. All messages sent to and from Agoda may be > monitored to ensure compliance with company policies, to protect the > company's interests and to remove potential malware. Electronic messages > may be intercepted, amended, lost or deleted, or contain viruses. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Xiubo Li 24 Jul '26

24 Jul '26

Hi Andras, Thanks for your quick feedback. Sure. Yesterday Dhairya and I found something important in both the MDS and kclient sides. Without the logs we will review the code to ensure our investigation covers this case. We will provide feedback later. Regards Xiubo On Thu, 23 Jul 2026 at 00:14, Andras Pataki via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Xiubo and Dhariya, > > The way I was able to reproduce the problem was to look at the error log > of my copy processes and take those files and check on them on the > various clients. Today, unfortunately, all those files seem to be > visible - and at the moment I don't have a way of reproducing the issue > from scratch. I'm trying to see what could be done to set up the > reproduction from scratch. If I manage to do that, I'll try to gather > some information on whether this is a client issue (i.e. flushing the > client caches changes anything) or whether it comes from the MDS. > Unfortunately the MDS in question is a very busy one, our main file > system MDS, so there are limits to what I can do to it. > > Andras > > > On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote: > > Hi Dhairya, > > > > Yeah, makes sense. > > > > Since I couldn't reproduce this locally, @Andras Pataki Could you help > > reproduce it by enabling the debug logs as Dhairya mentioned? And if > > possible please also try with the fuse client to see could you see > > too. > > > > Thanks > > Xiubo > > > > On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev(a)ceph.io> wrote: > >> Hi Xiubo, > >> > >> I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds… . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT? > >> > >> > >> On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li(a)clyso.com> wrote: > >>> Hi Andras, Dhairya, > >>> > >>> I recall hitting a similar issue with this and the workaround involves > >>> running the 'ls' command, but I can't recall exactly which command or > >>> the tracker reference. This happened years ago. > >>> > >>> This morning I tried to reproduce this locally but failed. And I just > >>> went through the mds and kclient code just found two suspect places > >>> could cause this: > >>> > >>> MDS side: in https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/… > >>> > >>> MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: > >>> > >>> if (curdir->is_complete() || // (1) > >>> (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) > >>> return -ENOENT; > >>> > >>> is_complete() means all live entries are cached in memory. Shortcut > >>> (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), > >>> but shortcut (1) applies to all snapids — including snapshots. > >>> > >>> When a snapshot exists but the file was later deleted from the live > >>> tree, the MDS eventually evicts the deleted file's dentry from the > >>> in-memory cache (a null dentry, which trim_dentry() removes without > >>> clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits > >>> is_complete()=true with the dentry absent (ENOENT), without ever > >>> checking the OSD where the snapshot data actually lives. > >>> > >>> READDIR takes a different code path that fetches from the OSD, which > >>> is why ls -l fixes it. > >>> > >>> Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: > >>> if (snapid == CEPH_NOSNAP && > >>> (curdir->is_complete() || ...bloom...)) { > >>> > >>> @Andras > >>> > >>> BTW, to verify the above could you try to reproduce this by using the > >>> ceph-fuse mount ? Locally I couldn't reproduce it by trying many times > >>> and I am planing to add some debug code in MDS to ammulated it to > >>> reproduce it later. > >>> > >>> ==== > >>> > >>> Kclient side: https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-li… > >>> Plus a kclient defense: ceph_d_revalidate() unconditionally trusts > >>> negative dentries in snapshot directories, caching a transient ENOENT > >>> forever. Change valid = 1 to valid = inode ? 1 : 0. > >>> > >>> I just created on trakcer to follow this: https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-a… > >>> > >>> Thanks > >>> Xiubo > >>> > >>> > >>> > >>> > >>> On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users > >>> <ceph-users(a)ceph.io> wrote: > >>>> Hi Andras, > >>>> > >>>> This sounds like the client served the lookups from its cached negative? > >>>> Can you check if you see the lookup calls being made to MDS and maybe set > >>>> debug_mds=20 to check if the client did send the request but it was MDS > >>>> which returned ENOENT but since the `ls -l` succeeded I'm wondering if this > >>>> is actually something from MDS. Maybe try a sync and drop_caches to > >>>> validate that? > >>>> > >>>> *Dhairya Parmar* > >>>> > >>>> Software Engineer, CephFS > >>>> > >>>> > >>>> On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < > >>>> ceph-users(a)ceph.io> wrote: > >>>> > >>>>> Hi ceph users/devs, > >>>>> > >>>>> We've been encountering a strange issue with snapshots as part of a > >>>>> project to copy our large ceph file system to a backup in parallel > >>>>> across nodes. We are taking a snapshot, and then doing a copy of the > >>>>> snapshot, which we expect to be static (unchanging) - using several > >>>>> clients in parallel - all kernel mounts. Some clients are unable to > >>>>> open files and directories that are known to exist (were discovered > >>>>> traversing the file system on perhaps another node). > >>>>> After some playing around, I could reproduce the issue on a single node > >>>>> with just 'ls' as follows: > >>>>> > >>>>> # This file doesn't seems to exist - and some number of parent directories > >>>>> are also "missing": [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > >>>>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > >>>>> No such file or directory # Until we get to this 'Projects' parent > >>>>> directory, which seems to be there [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 > >>>>> 1961 1961 46 Jun 8 14:02 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it > >>>>> is still not there ... [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > >>>>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > >>>>> No such file or directory # But ... let's see if the "tickle" the Projects > >>>>> directory a bit ... > >>>>> [root@cephmds603 ~]# ls -l > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null > >>>>> > >>>>> # Now all subdirectories and the file in question immediately becomes > >>>>> visible: > >>>>> [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > >>>>> drwxrwsr-x 5 1961 1961 3 May 7 17:18 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > >>>>> [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > >>>>> drwxrwsr-x 12 1961 1961 10 May 11 16:08 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > >>>>> [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > >>>>> drwxrwsr-x 5 1961 1961 3 May 11 16:18 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > >>>>> [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > >>>>> -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > >>>>> > >>>>> # This is the kernel client 6.18.32 (relatively recent LTS kernel) with > >>>>> ceph squid 19.2.3: > >>>>> [root@cephmds603 ~]# uname -a > >>>>> Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 > >>>>> x86_64 x86_64 GNU/Linux > >>>>> [root@cephmds603 ~]# ceph --version > >>>>> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > >>>>> (stable) > >>>>> [root@cephmds603 ~]# > >>>>> > >>>>> What is most interesting is that in the live tree (non-snapshotted tree) - > >>>>> the Projects directory exists, but everything under it has been removed! > >>>>> > >>>>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ > >>>>> drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ > >>>>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK > >>>>> ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or > >>>>> directory > >>>>> > >>>>> Am I running into some known issue? This looks to be specific to some > >>>>> clients - i.e. some clients see the directory and some don't at the same > >>>>> point in time, i.e. this is perhaps a kclient problem? > >>>>> Any ideas/suggestions would be appreciated, > >>>>> > >>>>> Andras > >>>>> _______________________________________________ > >>>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>>>> > >>>>> > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Re: Ubuntu 24.04 with Tentacle
by Szabo, Istvan (Agoda) 24 Jul '26

24 Jul '26

Thank you very much, this makes me thinking again should I use cephadm or should we still stay with the manual setup and package based update for a heavy loaded critical block storage cluster :/ Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> Sent: Thursday, 23 July 2026 18:33:59 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ubuntu 24.04 with Tentacle Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ On 2026-07-23 07:06, Szabo, Istvan (Agoda) via ceph-users wrote: > Reading throught the Ceph documentation regarding different versions and I see here https://docs.ceph.com/en/latest/start/os-recommendations/#platforms and "Upcoming" text. > What does this mean exactly, which release is going to be "A" on Ubuntu 24.04? Hi Istvan, Not really the answer you have asked for because I am not a Ceph developer, we have however got some experience with running Tentacle on Noble so perhaps it will come in handy to you. We have in recent months upgraded all of our Ceph servers, in both production and development clusters, from Jammy to Noble. This has been originally for two reasons: - for several reasons I shan't bore you with we currently use Podman rather than Docker with cephadm, and it transpired that the Jammy version of Podman does not support namespace settings required by multi-container services like the cephadm-managed SMB gateways; - since the upgrade from Squid to Tentacle one of our cluster kept seeing CPU lock-ups (and subsequent "slow BlueStore operations" warnings) progressively appear for more and more OSDs, and having tried to no avail a lot of different configuration tweaks we got to the point where the remaining choices were "upgrade to a newer kernel" and "start replacing HBAs". So far - touch wood! - all the upgraded systems have been operating perfectly as far as Ceph is concerned. -- MS _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Re: Ec pool min size
by Anthony D'Atri 23 Jul '26

23 Jul '26

> > the default min_size for an EC pool is always k + 1, except when you choose m = 1, then it will be k (I won’t go into details if m = 1 is a good choice or not) I will. It’s a bad idea for data you can’t easily reconstruct or write off. Indeed if you need continued availability if 3 hosts out of 9 are down, I would just do replica3. You could do a custom or MSR CRUSH rule but that’s a rabbit hole. With Umbrella you should be able to transparently migrate pools to different protection schemes … one it gets a lot of shakeout. > . You can temporarily reduce min_size to k, in a disaster recovery scenario it can help. But k + 1 is a safety switch, so if you reduce it, ensure to set it back after recovery. If you plan to take multiple hosts offline on a regular basis, I’d recommend to consider a different EC profile. Note that you can’t change the EC profile of a pool, you would need to create a new pool with the desired profile and copy the data over to the new pool. > > Regards > Eugen > > Zitat von "GLE, Vivien via ceph-users" <ceph-users(a)ceph.io>: > >> Hi, >> >> >> We have a 9 hosts (3 per pod) cluster with 1 pool in EC 5+3 (host domain failure), I expected that shutting down 1 pod wouldn't affect our pg but I just discovered that the EC pool is in min_size 6 while trying it. >> >> >> Is min_size 6 important in this case ? Could I safely lower it to 5 or using another EC k+m would suit better ? >> >> >> Thanks ! >> >> >> Vivien >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Ec pool min size
by Eugen Block 23 Jul '26

23 Jul '26

Hi, the default min_size for an EC pool is always k + 1, except when you choose m = 1, then it will be k (I won’t go into details if m = 1 is a good choice or not). You can temporarily reduce min_size to k, in a disaster recovery scenario it can help. But k + 1 is a safety switch, so if you reduce it, ensure to set it back after recovery. If you plan to take multiple hosts offline on a regular basis, I’d recommend to consider a different EC profile. Note that you can’t change the EC profile of a pool, you would need to create a new pool with the desired profile and copy the data over to the new pool. Regards Eugen Zitat von "GLE, Vivien via ceph-users" <ceph-users(a)ceph.io>: > Hi, > > > We have a 9 hosts (3 per pod) cluster with 1 pool in EC 5+3 (host > domain failure), I expected that shutting down 1 pod wouldn't affect > our pg but I just discovered that the EC pool is in min_size 6 while > trying it. > > > Is min_size 6 important in this case ? Could I safely lower it to 5 > or using another EC k+m would suit better ? > > > Thanks ! > > > Vivien > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ec pool min size
by GLE, Vivien 23 Jul '26

23 Jul '26

Hi, We have a 9 hosts (3 per pod) cluster with 1 pool in EC 5+3 (host domain failure), I expected that shutting down 1 pod wouldn't affect our pg but I just discovered that the EC pool is in min_size 6 while trying it. Is min_size 6 important in this case ? Could I safely lower it to 5 or using another EC k+m would suit better ? Thanks ! Vivien

1 0

Re: Ubuntu 24.04 with Tentacle
by Marek Szuba 23 Jul '26

23 Jul '26

On 2026-07-23 07:06, Szabo, Istvan (Agoda) via ceph-users wrote: > Reading throught the Ceph documentation regarding different versions and I see here https://docs.ceph.com/en/latest/start/os-recommendations/#platforms and "Upcoming" text. > What does this mean exactly, which release is going to be "A" on Ubuntu 24.04? Hi Istvan, Not really the answer you have asked for because I am not a Ceph developer, we have however got some experience with running Tentacle on Noble so perhaps it will come in handy to you. We have in recent months upgraded all of our Ceph servers, in both production and development clusters, from Jammy to Noble. This has been originally for two reasons: - for several reasons I shan't bore you with we currently use Podman rather than Docker with cephadm, and it transpired that the Jammy version of Podman does not support namespace settings required by multi-container services like the cephadm-managed SMB gateways; - since the upgrade from Squid to Tentacle one of our cluster kept seeing CPU lock-ups (and subsequent "slow BlueStore operations" warnings) progressively appear for more and more OSDs, and having tried to no avail a lot of different configuration tweaks we got to the point where the remaining choices were "upgrade to a newer kernel" and "start replacing HBAs". So far - touch wood! - all the upgraded systems have been operating perfectly as far as Ceph is concerned. -- MS

1 0

Guidance Required for Changing Ceph Squid Development Version to Stable Release
by Kathirvel T 23 Jul '26

23 Jul '26

Hello Ceph Community, We are looking for guidance regarding our existing Ceph cluster. Our cluster was deployed using ceph-ansible and is currently running the Squid development release (v19.3.0). We are planning to adopt the cluster into cephadm using the cephadm-adopt playbook provided by ceph-ansible. Before proceeding, we would like to understand the recommended approach for changing our Ceph version from the Squid development release (v19.3.0) to the stable Squid release (v19.2.3). Cluster Details Deployment: ceph-ansible Current Version: Squid Development (v19.3.0) Health: HEALTH_OK Hosts: 4 MONs: 3 MGRs: 1 Active, 2 Standby OSDs: 15 (15 Up / 15 In) RGW: 1 daemon Pools: 15 Objects: ~1.64 million Usage: 17 TiB used, 9.2 TiB / 26 TiB avail We have the following questions: Is it possible to change from the Squid development version (v19.3.0) to the stable Squid release (v19.2.3)? Is this a supported approach, or is there a recommended alternative? Should we change to the stable release before adopting the cluster into cephadm, or is it better to perform the cephadm adoption first and then update the version? Are there any known limitations, prerequisites, or best practices we should be aware of before proceeding? Any guidance, recommendations, or references to relevant documentation would be greatly appreciated. Thanks & Regards, Kathirvel T Associate R&D Engineer mailto:manoj.kumar@zybisys.com kathirvel.t(a)zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 0

Ubuntu 24.04 with Tentacle
by Szabo, Istvan (Agoda) 23 Jul '26

23 Jul '26

Hi Developers, Reading throught the Ceph documentation regarding different versions and I see here https://docs.ceph.com/en/latest/start/os-recommendations/#platforms and "Upcoming" text. What does this mean exactly, which release is going to be "A" on Ubuntu 24.04? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Re: No container image for 17.2.9 on quay.io?
by Patrick Donnelly 22 Jul '26

22 Jul '26

Hi Robert, On Wed, Jul 22, 2026 at 5:19 AM Robert Sander via ceph-users <ceph-users(a)ceph.io> wrote: > Why is there no container image for 17.2.9 available on quay.io? The answer is here: https://docs.ceph.com/en/latest/releases/quincy/#v17-2-9-quincy -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Andras Pataki 22 Jul '26

22 Jul '26

Hi Xiubo and Dhariya, The way I was able to reproduce the problem was to look at the error log of my copy processes and take those files and check on them on the various clients. Today, unfortunately, all those files seem to be visible - and at the moment I don't have a way of reproducing the issue from scratch. I'm trying to see what could be done to set up the reproduction from scratch. If I manage to do that, I'll try to gather some information on whether this is a client issue (i.e. flushing the client caches changes anything) or whether it comes from the MDS. Unfortunately the MDS in question is a very busy one, our main file system MDS, so there are limits to what I can do to it. Andras On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote: > Hi Dhairya, > > Yeah, makes sense. > > Since I couldn't reproduce this locally, @Andras Pataki Could you help > reproduce it by enabling the debug logs as Dhairya mentioned? And if > possible please also try with the fuse client to see could you see > too. > > Thanks > Xiubo > > On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev(a)ceph.io> wrote: >> Hi Xiubo, >> >> I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds… . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT? >> >> >> On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li(a)clyso.com> wrote: >>> Hi Andras, Dhairya, >>> >>> I recall hitting a similar issue with this and the workaround involves >>> running the 'ls' command, but I can't recall exactly which command or >>> the tracker reference. This happened years ago. >>> >>> This morning I tried to reproduce this locally but failed. And I just >>> went through the mds and kclient code just found two suspect places >>> could cause this: >>> >>> MDS side: in https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/… >>> >>> MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: >>> >>> if (curdir->is_complete() || // (1) >>> (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) >>> return -ENOENT; >>> >>> is_complete() means all live entries are cached in memory. Shortcut >>> (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), >>> but shortcut (1) applies to all snapids — including snapshots. >>> >>> When a snapshot exists but the file was later deleted from the live >>> tree, the MDS eventually evicts the deleted file's dentry from the >>> in-memory cache (a null dentry, which trim_dentry() removes without >>> clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits >>> is_complete()=true with the dentry absent (ENOENT), without ever >>> checking the OSD where the snapshot data actually lives. >>> >>> READDIR takes a different code path that fetches from the OSD, which >>> is why ls -l fixes it. >>> >>> Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: >>> if (snapid == CEPH_NOSNAP && >>> (curdir->is_complete() || ...bloom...)) { >>> >>> @Andras >>> >>> BTW, to verify the above could you try to reproduce this by using the >>> ceph-fuse mount ? Locally I couldn't reproduce it by trying many times >>> and I am planing to add some debug code in MDS to ammulated it to >>> reproduce it later. >>> >>> ==== >>> >>> Kclient side: https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-li… >>> Plus a kclient defense: ceph_d_revalidate() unconditionally trusts >>> negative dentries in snapshot directories, caching a transient ENOENT >>> forever. Change valid = 1 to valid = inode ? 1 : 0. >>> >>> I just created on trakcer to follow this: https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-a… >>> >>> Thanks >>> Xiubo >>> >>> >>> >>> >>> On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users >>> <ceph-users(a)ceph.io> wrote: >>>> Hi Andras, >>>> >>>> This sounds like the client served the lookups from its cached negative? >>>> Can you check if you see the lookup calls being made to MDS and maybe set >>>> debug_mds=20 to check if the client did send the request but it was MDS >>>> which returned ENOENT but since the `ls -l` succeeded I'm wondering if this >>>> is actually something from MDS. Maybe try a sync and drop_caches to >>>> validate that? >>>> >>>> *Dhairya Parmar* >>>> >>>> Software Engineer, CephFS >>>> >>>> >>>> On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < >>>> ceph-users(a)ceph.io> wrote: >>>> >>>>> Hi ceph users/devs, >>>>> >>>>> We've been encountering a strange issue with snapshots as part of a >>>>> project to copy our large ceph file system to a backup in parallel >>>>> across nodes. We are taking a snapshot, and then doing a copy of the >>>>> snapshot, which we expect to be static (unchanging) - using several >>>>> clients in parallel - all kernel mounts. Some clients are unable to >>>>> open files and directories that are known to exist (were discovered >>>>> traversing the file system on perhaps another node). >>>>> After some playing around, I could reproduce the issue on a single node >>>>> with just 'ls' as follows: >>>>> >>>>> # This file doesn't seems to exist - and some number of parent directories >>>>> are also "missing": [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >>>>> >>>>> ls: cannot access >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': >>>>> >>>>> No such file or directory [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io >>>>> >>>>> ls: cannot access >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': >>>>> >>>>> No such file or directory [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W >>>>> >>>>> ls: cannot access >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': >>>>> >>>>> No such file or directory [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores >>>>> >>>>> ls: cannot access >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': >>>>> >>>>> No such file or directory [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis >>>>> >>>>> ls: cannot access >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': >>>>> >>>>> No such file or directory [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output >>>>> >>>>> ls: cannot access >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': >>>>> >>>>> No such file or directory [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >>>>> >>>>> ls: cannot access >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': >>>>> >>>>> No such file or directory [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >>>>> ls: cannot access >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': >>>>> No such file or directory [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >>>>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >>>>> No such file or directory # Until we get to this 'Projects' parent >>>>> directory, which seems to be there [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 >>>>> 1961 1961 46 Jun 8 14:02 >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it >>>>> is still not there ... [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >>>>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >>>>> No such file or directory # But ... let's see if the "tickle" the Projects >>>>> directory a bit ... >>>>> [root@cephmds603 ~]# ls -l >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null >>>>> >>>>> # Now all subdirectories and the file in question immediately becomes >>>>> visible: >>>>> [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >>>>> drwxrwsr-x 5 1961 1961 3 May 7 17:18 >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >>>>> [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >>>>> drwxrwsr-x 12 1961 1961 10 May 11 16:08 >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >>>>> [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >>>>> drwxrwsr-x 5 1961 1961 3 May 11 16:18 >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >>>>> [root@cephmds603 ~]# ls -ld >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >>>>> -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >>>>> >>>>> # This is the kernel client 6.18.32 (relatively recent LTS kernel) with >>>>> ceph squid 19.2.3: >>>>> [root@cephmds603 ~]# uname -a >>>>> Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 >>>>> x86_64 x86_64 GNU/Linux >>>>> [root@cephmds603 ~]# ceph --version >>>>> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid >>>>> (stable) >>>>> [root@cephmds603 ~]# >>>>> >>>>> What is most interesting is that in the live tree (non-snapshotted tree) - >>>>> the Projects directory exists, but everything under it has been removed! >>>>> >>>>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ >>>>> drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ >>>>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK >>>>> ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or >>>>> directory >>>>> >>>>> Am I running into some known issue? This looks to be specific to some >>>>> clients - i.e. some clients see the directory and some don't at the same >>>>> point in time, i.e. this is perhaps a kclient problem? >>>>> Any ideas/suggestions would be appreciated, >>>>> >>>>> Andras >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: v21.1.0 Umbrella RC0 released
by Stefan Möding 22 Jul '26

22 Jul '26

Patrick Donnelly via ceph-users <ceph-users(a)ceph.io> writes: > Why is Canonical not updating/fixing these packages when the standard > EOL for 24.04 is May 2029? The Qt5 packages are part of the Universe repo. This is not covered by the standard (viz. free) 5 year. LTS covers only the Main repo. It requires an ESM contract to get support for the Universe repo. So with the release of Ubuntu 26.04 the ongoing effort to keep the Universe repo up to date has effectively been dropped. -- Stefan

1 0

Re: v21.1.0 Umbrella RC0 released
by Stefan Möding 22 Jul '26

22 Jul '26

Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> writes: > Are you prevented from deploying in containers? I thought it would be easier to use the provided packages and avoid the additional hassle with setting up and maintaining Docker. Also, if the images are build using Ubuntu 24.04 (I haven’t checked), the images would have the same issues. In addition we normally do not allow SSH logins between two production hosts, which seems to be a requirement for Cephadm. -- Stefan

1 0

Re: v21.1.0 Umbrella RC0 released
by Patrick Donnelly 22 Jul '26

22 Jul '26

On Wed, Jul 22, 2026 at 4:51 AM Stefan Möding via ceph-users <ceph-users(a)ceph.io> wrote: > > Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> writes: > > > This is the first release candidate for Umbrella. > > Ubuntu 22.04, 24.04, CentOS 9, Rocky Linux 10, Debian 12 (Bookworm) and > > Debian 13 (Trixie) were built for this RC0. > > Are there plans to also provide packages for Ubuntu 26.04 (released in > April)? The planned support is here: https://docs.ceph.com/en/latest/start/os-recommendations/#platforms > We are currently running Tentacle on Ubuntu 24.04. The Ceph code > depends on the Thrift c++ library which itself depends on multiple Qt 5 > modules. In Ubuntu 24.04 the included Qt 5 modules have know security > vulnerabilities and Canonical provides only fixes as part of an Expanded > Security Contract. That makes it difficult to run Ceph in an > environment where security compliance is a concern. Why is Canonical not updating/fixing these packages when the standard EOL for 24.04 is May 2029? In addition to the documentation improvements mentioned above, I am making an effort to have distribution package support predictable for future Ceph releases. I want to avoid situations where we retroactively add support for newer distributions to older Ceph releases. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: v21.1.0 Umbrella RC0 released
by Anthony D'Atri 22 Jul '26

22 Jul '26

Are you prevented from deploying in containers? > On Jul 22, 2026, at 4:51 AM, Stefan Möding via ceph-users <ceph-users(a)ceph.io> wrote: > > Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> writes: > >> This is the first release candidate for Umbrella. >> Ubuntu 22.04, 24.04, CentOS 9, Rocky Linux 10, Debian 12 (Bookworm) and >> Debian 13 (Trixie) were built for this RC0. > > Are there plans to also provide packages for Ubuntu 26.04 (released in > April)? > > We are currently running Tentacle on Ubuntu 24.04. The Ceph code > depends on the Thrift c++ library which itself depends on multiple Qt 5 > modules. In Ubuntu 24.04 the included Qt 5 modules have know security > vulnerabilities and Canonical provides only fixes as part of an Expanded > Security Contract. That makes it difficult to run Ceph in an > environment where security compliance is a concern. > > -- > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Anthony D'Atri 22 Jul '26

22 Jul '26

Excellent summary. In short, the bootstrap process is one time only. > On Jul 22, 2026, at 5:41 AM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, responses inline... > > > Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > >> Actually, what confused me was this: >> >> cephadm bootstrap --mon-ip <my-ip> >> >> Among other things, this command: >> - Generates a new SSH key for the Ceph cluster and adds it >> to the /root/.ssh/authorized_keys file. >> - Saves a copy of the public key to the /etc/ceph/ceph.pub file. >> >> You then need to add a node. >> >> ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> >> >> ceph orch host add <newhost> [<ip>] [<label1> ...] >> >> I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! > > That's stored in the config-key db: > > ceph config-key get mgr/cephadm/ssh_identity_key > >> >> -- >> M >> >> Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: >> >>> Hi again. >>> >>> I'm answering my own question. >>> >>> It seems I read the wrong version of the doc >>> ... sic! >>> >>> It should be enough to set the “_admin” flag on all nodes: >>> >>> and that we also need to install cephadm on the nodes we >>> want to use for administration, right!? > > Not necessarily, but it can be more convenient for debugging. The orchestrator stores a cephadm binary on each node to be able to manage them, it is stored in /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}. That's used to remotely orchestrate the cluster/nodes etc. The original bootstrap node requires a cephadm binary either installed as a package or downloaded via curl as described in the docs to be able to bootstrap. After that you'll only need to execute cephadm commands locally on the nodes if you're debugging issues etc., otherwise the orchestrator will only use the file in /var/lib/ceph. > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Xiubo Li 22 Jul '26

22 Jul '26

Hi Dhairya, Yeah, makes sense. Since I couldn't reproduce this locally, @Andras Pataki Could you help reproduce it by enabling the debug logs as Dhairya mentioned? And if possible please also try with the fuse client to see could you see too. Thanks Xiubo On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev(a)ceph.io> wrote: > > Hi Xiubo, > > I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://github.com/ceph/ceph/blob/squid/src/mds/Server.cc#L5029-L5041. Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT? > > > On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li(a)clyso.com> wrote: >> >> Hi Andras, Dhairya, >> >> I recall hitting a similar issue with this and the workaround involves >> running the 'ls' command, but I can't recall exactly which command or >> the tracker reference. This happened years ago. >> >> This morning I tried to reproduce this locally but failed. And I just >> went through the mds and kclient code just found two suspect places >> could cause this: >> >> MDS side: in https://github.com/ceph/ceph/blob/main/src/mds/MDCache.cc#L8693-L8696 >> >> MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: >> >> if (curdir->is_complete() || // (1) >> (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) >> return -ENOENT; >> >> is_complete() means all live entries are cached in memory. Shortcut >> (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), >> but shortcut (1) applies to all snapids — including snapshots. >> >> When a snapshot exists but the file was later deleted from the live >> tree, the MDS eventually evicts the deleted file's dentry from the >> in-memory cache (a null dentry, which trim_dentry() removes without >> clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits >> is_complete()=true with the dentry absent (ENOENT), without ever >> checking the OSD where the snapshot data actually lives. >> >> READDIR takes a different code path that fetches from the OSD, which >> is why ls -l fixes it. >> >> Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: >> if (snapid == CEPH_NOSNAP && >> (curdir->is_complete() || ...bloom...)) { >> >> @Andras >> >> BTW, to verify the above could you try to reproduce this by using the >> ceph-fuse mount ? Locally I couldn't reproduce it by trying many times >> and I am planing to add some debug code in MDS to ammulated it to >> reproduce it later. >> >> ==== >> >> Kclient side: https://github.com/ceph/ceph-client/blob/for-linus/fs/ceph/dir.c#L1984-L1987 >> Plus a kclient defense: ceph_d_revalidate() unconditionally trusts >> negative dentries in snapshot directories, caching a transient ENOENT >> forever. Change valid = 1 to valid = inode ? 1 : 0. >> >> I just created on trakcer to follow this: https://tracker.ceph.com/issues/78529 >> >> Thanks >> Xiubo >> >> >> >> >> On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users >> <ceph-users(a)ceph.io> wrote: >> > >> > Hi Andras, >> > >> > This sounds like the client served the lookups from its cached negative? >> > Can you check if you see the lookup calls being made to MDS and maybe set >> > debug_mds=20 to check if the client did send the request but it was MDS >> > which returned ENOENT but since the `ls -l` succeeded I'm wondering if this >> > is actually something from MDS. Maybe try a sync and drop_caches to >> > validate that? >> > >> > *Dhairya Parmar* >> > >> > Software Engineer, CephFS >> > >> > >> > On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < >> > ceph-users(a)ceph.io> wrote: >> > >> > > Hi ceph users/devs, >> > > >> > > We've been encountering a strange issue with snapshots as part of a >> > > project to copy our large ceph file system to a backup in parallel >> > > across nodes. We are taking a snapshot, and then doing a copy of the >> > > snapshot, which we expect to be static (unchanging) - using several >> > > clients in parallel - all kernel mounts. Some clients are unable to >> > > open files and directories that are known to exist (were discovered >> > > traversing the file system on perhaps another node). >> > > After some playing around, I could reproduce the issue on a single node >> > > with just 'ls' as follows: >> > > >> > > # This file doesn't seems to exist - and some number of parent directories >> > > are also "missing": [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >> > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >> > > No such file or directory # Until we get to this 'Projects' parent >> > > directory, which seems to be there [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 >> > > 1961 1961 46 Jun 8 14:02 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it >> > > is still not there ... [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >> > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >> > > No such file or directory # But ... let's see if the "tickle" the Projects >> > > directory a bit ... >> > > [root@cephmds603 ~]# ls -l >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null >> > > >> > > # Now all subdirectories and the file in question immediately becomes >> > > visible: >> > > [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >> > > drwxrwsr-x 5 1961 1961 3 May 7 17:18 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >> > > [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> > > drwxrwsr-x 12 1961 1961 10 May 11 16:08 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> > > [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> > > drwxrwsr-x 5 1961 1961 3 May 11 16:18 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> > > [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> > > -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> > > >> > > # This is the kernel client 6.18.32 (relatively recent LTS kernel) with >> > > ceph squid 19.2.3: >> > > [root@cephmds603 ~]# uname -a >> > > Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 >> > > x86_64 x86_64 GNU/Linux >> > > [root@cephmds603 ~]# ceph --version >> > > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid >> > > (stable) >> > > [root@cephmds603 ~]# >> > > >> > > What is most interesting is that in the live tree (non-snapshotted tree) - >> > > the Projects directory exists, but everything under it has been removed! >> > > >> > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ >> > > drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ >> > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK >> > > ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or >> > > directory >> > > >> > > Am I running into some known issue? This looks to be specific to some >> > > clients - i.e. some clients see the directory and some don't at the same >> > > point in time, i.e. this is perhaps a kclient problem? >> > > Any ideas/suggestions would be appreciated, >> > > >> > > Andras >> > > _______________________________________________ >> > > ceph-users mailing list -- ceph-users(a)ceph.io >> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > > >> > > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

Re: No container image for 17.2.9 on quay.io?
by Eugen Block 22 Jul '26

22 Jul '26

Hi, correct, there are no container images as announced on the list [0]: > Please be aware that this hotfix was finalized after its end-of-life > (EOL) declaration and subsequent removal from the build system. As a > result, container builds were not created, and the hotfix has only > been subject to limited testing. I concur that upgrading to Reef before adopting is the way to go here. Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YSIXQKUJU4… Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Hi, > > Why is there no container image for 17.2.9 available on quay.io? > > We are currently trying to adopt a Quincy 17.2.9 cluster into the > orchestrator and fail at the step where the image should be pulled. > > Do we need to upgrade this cluster manually to Reef before being > able to adopt it? > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

No container image for 17.2.9 on quay.io?
by Robert Sander 22 Jul '26

22 Jul '26

Hi, Why is there no container image for 17.2.9 available on quay.io? We are currently trying to adopt a Quincy 17.2.9 cluster into the orchestrator and fail at the step where the image should be pulled. Do we need to upgrade this cluster manually to Reef before being able to adopt it? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Xiubo Li 22 Jul '26

22 Jul '26

Hi Andras, Dhairya, I recall hitting a similar issue with this and the workaround involves running the 'ls' command, but I can't recall exactly which command or the tracker reference. This happened years ago. This morning I tried to reproduce this locally but failed. And I just went through the mds and kclient code just found two suspect places could cause this: MDS side: in https://github.com/ceph/ceph/blob/main/src/mds/MDCache.cc#L8693-L8696 MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: if (curdir->is_complete() || // (1) (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) return -ENOENT; is_complete() means all live entries are cached in memory. Shortcut (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), but shortcut (1) applies to all snapids — including snapshots. When a snapshot exists but the file was later deleted from the live tree, the MDS eventually evicts the deleted file's dentry from the in-memory cache (a null dentry, which trim_dentry() removes without clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits is_complete()=true with the dentry absent (ENOENT), without ever checking the OSD where the snapshot data actually lives. READDIR takes a different code path that fetches from the OSD, which is why ls -l fixes it. Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: if (snapid == CEPH_NOSNAP && (curdir->is_complete() || ...bloom...)) { @Andras BTW, to verify the above could you try to reproduce this by using the ceph-fuse mount ? Locally I couldn't reproduce it by trying many times and I am planing to add some debug code in MDS to ammulated it to reproduce it later. ==== Kclient side: https://github.com/ceph/ceph-client/blob/for-linus/fs/ceph/dir.c#L1984-L1987 Plus a kclient defense: ceph_d_revalidate() unconditionally trusts negative dentries in snapshot directories, caching a transient ENOENT forever. Change valid = 1 to valid = inode ? 1 : 0. I just created on trakcer to follow this: https://tracker.ceph.com/issues/78529 Thanks Xiubo On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Andras, > > This sounds like the client served the lookups from its cached negative? > Can you check if you see the lookup calls being made to MDS and maybe set > debug_mds=20 to check if the client did send the request but it was MDS > which returned ENOENT but since the `ls -l` succeeded I'm wondering if this > is actually something from MDS. Maybe try a sync and drop_caches to > validate that? > > *Dhairya Parmar* > > Software Engineer, CephFS > > > On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < > ceph-users(a)ceph.io> wrote: > > > Hi ceph users/devs, > > > > We've been encountering a strange issue with snapshots as part of a > > project to copy our large ceph file system to a backup in parallel > > across nodes. We are taking a snapshot, and then doing a copy of the > > snapshot, which we expect to be static (unchanging) - using several > > clients in parallel - all kernel mounts. Some clients are unable to > > open files and directories that are known to exist (were discovered > > traversing the file system on perhaps another node). > > After some playing around, I could reproduce the issue on a single node > > with just 'ls' as follows: > > > > # This file doesn't seems to exist - and some number of parent directories > > are also "missing": [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > > No such file or directory # Until we get to this 'Projects' parent > > directory, which seems to be there [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 > > 1961 1961 46 Jun 8 14:02 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it > > is still not there ... [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > > No such file or directory # But ... let's see if the "tickle" the Projects > > directory a bit ... > > [root@cephmds603 ~]# ls -l > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null > > > > # Now all subdirectories and the file in question immediately becomes > > visible: > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > > drwxrwsr-x 5 1961 1961 3 May 7 17:18 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > > drwxrwsr-x 12 1961 1961 10 May 11 16:08 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > drwxrwsr-x 5 1961 1961 3 May 11 16:18 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > > > # This is the kernel client 6.18.32 (relatively recent LTS kernel) with > > ceph squid 19.2.3: > > [root@cephmds603 ~]# uname -a > > Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 > > x86_64 x86_64 GNU/Linux > > [root@cephmds603 ~]# ceph --version > > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > > (stable) > > [root@cephmds603 ~]# > > > > What is most interesting is that in the live tree (non-snapshotted tree) - > > the Projects directory exists, but everything under it has been removed! > > > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ > > drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK > > ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or > > directory > > > > Am I running into some known issue? This looks to be specific to some > > clients - i.e. some clients see the directory and some don't at the same > > point in time, i.e. this is perhaps a kclient problem? > > Any ideas/suggestions would be appreciated, > > > > Andras > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: v21.1.0 Umbrella RC0 released
by Stefan Möding 22 Jul '26

22 Jul '26

Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> writes: > This is the first release candidate for Umbrella. > Ubuntu 22.04, 24.04, CentOS 9, Rocky Linux 10, Debian 12 (Bookworm) and > Debian 13 (Trixie) were built for this RC0. Are there plans to also provide packages for Ubuntu 26.04 (released in April)? We are currently running Tentacle on Ubuntu 24.04. The Ceph code depends on the Thrift c++ library which itself depends on multiple Qt 5 modules. In Ubuntu 24.04 the included Qt 5 modules have know security vulnerabilities and Canonical provides only fixes as part of an Expanded Security Contract. That makes it difficult to run Ceph in an environment where security compliance is a concern. -- Stefan

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Eugen Block 22 Jul '26

22 Jul '26

Hi, responses inline... Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > Actually, what confused me was this: > > cephadm bootstrap --mon-ip <my-ip> > > Among other things, this command: > - Generates a new SSH key for the Ceph cluster and adds it > to the /root/.ssh/authorized_keys file. > - Saves a copy of the public key to the /etc/ceph/ceph.pub file. > > You then need to add a node. > > ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> > > ceph orch host add <newhost> [<ip>] [<label1> ...] > > I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! That's stored in the config-key db: ceph config-key get mgr/cephadm/ssh_identity_key > > -- > M > > Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: > >> Hi again. >> >> I'm answering my own question. >> >> It seems I read the wrong version of the doc >> ... sic! >> >> It should be enough to set the “_admin” flag on all nodes: >> >> and that we also need to install cephadm on the nodes we >> want to use for administration, right!? Not necessarily, but it can be more convenient for debugging. The orchestrator stores a cephadm binary on each node to be able to manage them, it is stored in /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}. That's used to remotely orchestrate the cluster/nodes etc. The original bootstrap node requires a cephadm binary either installed as a package or downloaded via curl as described in the docs to be able to bootstrap. After that you'll only need to execute cephadm commands locally on the nodes if you're debugging issues etc., otherwise the orchestrator will only use the file in /var/lib/ceph. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 22 Jul '26

22 Jul '26

Actually, what confused me was this: cephadm bootstrap --mon-ip <my-ip> Among other things, this command: - Generates a new SSH key for the Ceph cluster and adds it to the /root/.ssh/authorized_keys file. - Saves a copy of the public key to the /etc/ceph/ceph.pub file. You then need to add a node. ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> ceph orch host add <newhost> [<ip>] [<label1> ...] I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! -- M Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: > Hi again. > > I'm answering my own question. > > It seems I read the wrong version of the doc > ... sic! > > It should be enough to set the “_admin” flag on all nodes: > > and that we also need to install cephadm on the nodes we > want to use for administration, right!?

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 22 Jul '26

22 Jul '26

Hi again. I'm answering my own question. It seems I read the wrong version of the doc ... sic! It should be enough to set the “_admin” flag on all nodes: and that we also need to install cephadm on the nodes we want to use for administration, right!? -- M Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: > Hi all, > > I am currently deploying a Ceph cluster with cephadm on 3 nodes with > MON, MGR, and MDS roles. OSDs will be added later. > > My goal is for these 3 nodes to be fully interchangeable, with the > ability to administer the cluster from any of them without risk if a > node fails. > > I have identified that the solution involves synchronizing > /var/lib/ceph/ (excluding the bootstrap file) across all nodes. > > However: > > - Is this the right approach to achieve this goal? > > - Which tool should I use for reliable bidirectional synchronization? > > - What are the potential risks (conflicts, corruption) and precautions > to take? > > Have you implemented such a configuration in production before? > > Thank you for your advice.

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Dhairya Parmar 22 Jul '26

22 Jul '26

Hi Andras, This sounds like the client served the lookups from its cached negative? Can you check if you see the lookup calls being made to MDS and maybe set debug_mds=20 to check if the client did send the request but it was MDS which returned ENOENT but since the `ls -l` succeeded I'm wondering if this is actually something from MDS. Maybe try a sync and drop_caches to validate that? *Dhairya Parmar* Software Engineer, CephFS On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < ceph-users(a)ceph.io> wrote: > Hi ceph users/devs, > > We've been encountering a strange issue with snapshots as part of a > project to copy our large ceph file system to a backup in parallel > across nodes. We are taking a snapshot, and then doing a copy of the > snapshot, which we expect to be static (unchanging) - using several > clients in parallel - all kernel mounts. Some clients are unable to > open files and directories that are known to exist (were discovered > traversing the file system on perhaps another node). > After some playing around, I could reproduce the issue on a single node > with just 'ls' as follows: > > # This file doesn't seems to exist - and some number of parent directories > are also "missing": [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > No such file or directory # Until we get to this 'Projects' parent > directory, which seems to be there [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 > 1961 1961 46 Jun 8 14:02 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it > is still not there ... [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > No such file or directory # But ... let's see if the "tickle" the Projects > directory a bit ... > [root@cephmds603 ~]# ls -l > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null > > # Now all subdirectories and the file in question immediately becomes > visible: > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > drwxrwsr-x 5 1961 1961 3 May 7 17:18 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > drwxrwsr-x 12 1961 1961 10 May 11 16:08 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > drwxrwsr-x 5 1961 1961 3 May 11 16:18 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > # This is the kernel client 6.18.32 (relatively recent LTS kernel) with > ceph squid 19.2.3: > [root@cephmds603 ~]# uname -a > Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 > x86_64 x86_64 GNU/Linux > [root@cephmds603 ~]# ceph --version > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > (stable) > [root@cephmds603 ~]# > > What is most interesting is that in the live tree (non-snapshotted tree) - > the Projects directory exists, but everything under it has been removed! > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ > drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK > ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or > directory > > Am I running into some known issue? This looks to be specific to some > clients - i.e. some clients see the directory and some don't at the same > point in time, i.e. this is perhaps a kclient problem? > Any ideas/suggestions would be appreciated, > > Andras > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Anthony D'Atri 22 Jul '26

22 Jul '26

Exactly. Use the orchestrator, Luke. Additionally I would set count:3 for the mgr service. It’s kind of a hassle to recover from no active mgr so I encourage running one alongside every mon. > On Jul 21, 2026, at 4:30 PM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of the admin keyring via orchestrator and the _admin label [0]. That enables you to manage the cluster from any admin node. Not sure what you mean by "bootstrap file" though... > > If you'd synced the /var/lib/ceph directory, you'd overwrite all the daemons but one, including keyring etc. So please don't do that, that would break your cluster. > > Regards, > Eugen > > [0] https://docs.ceph.com/en/latest/cephadm/operations/#enabling-management-of-… > > Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > >> Hi all, >> >> I am currently deploying a Ceph cluster with cephadm on 3 nodes with >> MON, MGR, and MDS roles. OSDs will be added later. >> >> My goal is for these 3 nodes to be fully interchangeable, with the >> ability to administer the cluster from any of them without risk if a >> node fails. >> >> I have identified that the solution involves synchronizing >> /var/lib/ceph/ (excluding the bootstrap file) across all nodes. >> >> However: >> >> - Is this the right approach to achieve this goal? >> >> - Which tool should I use for reliable bidirectional synchronization? >> >> - What are the potential risks (conflicts, corruption) and precautions >> to take? >> >> Have you implemented such a configuration in production before? >> >> Thank you for your advice. >> >> -- >> Michel >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Anthony D'Atri 22 Jul '26

22 Jul '26

I suggest gathering the kernel version on all clients, see if there’s a correlation.

1 0

CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Andras Pataki 21 Jul '26

21 Jul '26

Hi ceph users/devs, We've been encountering a strange issue with snapshots as part of a project to copy our large ceph file system to a backup in parallel across nodes. We are taking a snapshot, and then doing a copy of the snapshot, which we expect to be static (unchanging) - using several clients in parallel - all kernel mounts. Some clients are unable to open files and directories that are known to exist (were discovered traversing the file system on perhaps another node). After some playing around, I could reproduce the issue on a single node with just 'ls' as follows: # This file doesn't seems to exist - and some number of parent directories are also "missing": [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': No such file or directory # Until we get to this 'Projects' parent directory, which seems to be there [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 1961 1961 46 Jun 8 14:02 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it is still not there ... [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': No such file or directory # But ... let's see if the "tickle" the Projects directory a bit ... [root@cephmds603 ~]# ls -l /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null # Now all subdirectories and the file in question immediately becomes visible: [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK drwxrwsr-x 5 1961 1961 3 May 7 17:18 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM drwxrwsr-x 12 1961 1961 10 May 11 16:08 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen drwxrwsr-x 5 1961 1961 3 May 11 16:18 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log # This is the kernel client 6.18.32 (relatively recent LTS kernel) with ceph squid 19.2.3: [root@cephmds603 ~]# uname -a Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 x86_64 x86_64 GNU/Linux [root@cephmds603 ~]# ceph --version ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@cephmds603 ~]# What is most interesting is that in the live tree (non-snapshotted tree) - the Projects directory exists, but everything under it has been removed! [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or directory Am I running into some known issue? This looks to be specific to some clients - i.e. some clients see the directory and some don't at the same point in time, i.e. this is perhaps a kclient problem? Any ideas/suggestions would be appreciated, Andras

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Eugen Block 21 Jul '26

21 Jul '26

Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of the admin keyring via orchestrator and the _admin label [0]. That enables you to manage the cluster from any admin node. Not sure what you mean by "bootstrap file" though... If you'd synced the /var/lib/ceph directory, you'd overwrite all the daemons but one, including keyring etc. So please don't do that, that would break your cluster. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephadm/operations/#enabling-management-of-… Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > Hi all, > > I am currently deploying a Ceph cluster with cephadm on 3 nodes with > MON, MGR, and MDS roles. OSDs will be added later. > > My goal is for these 3 nodes to be fully interchangeable, with the > ability to administer the cluster from any of them without risk if a > node fails. > > I have identified that the solution involves synchronizing > /var/lib/ceph/ (excluding the bootstrap file) across all nodes. > > However: > > - Is this the right approach to achieve this goal? > > - Which tool should I use for reliable bidirectional synchronization? > > - What are the potential risks (conflicts, corruption) and precautions > to take? > > Have you implemented such a configuration in production before? > > Thank you for your advice. > > -- > Michel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph User / Dev Meetup - Wednesday, July 22, at 7:00 AM PDT
by Anthony Middleton 21 Jul '26

21 Jul '26

Hey Ceph Friends, Join us tomorrow, Wednesday, July 22, at 7:00 AM PDT / 14:00 UTC, for our July Ceph User + Dev Meetup. This Meetup is an opportunity for Ceph Users and Developers to discuss real-world deployment experiences and upcoming community initiatives. The following topics are on the agenda for this Meetup: - Discussing cloud transition and restore features that aid in integrating tape with Ceph and planning a proof of concept. - Inquire about improvements in the upcoming Umbrella release and ask for a release timeline. - Ongoing issues with SDs - Storage head nodes connected via multipath to a JBOD. RSVP and add your question or topic to the agenda here: https://lnkd.in/gRdjYAvF Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 21 Jul '26

21 Jul '26

Hi all, I am currently deploying a Ceph cluster with cephadm on 3 nodes with MON, MGR, and MDS roles. OSDs will be added later. My goal is for these 3 nodes to be fully interchangeable, with the ability to administer the cluster from any of them without risk if a node fails. I have identified that the solution involves synchronizing /var/lib/ceph/ (excluding the bootstrap file) across all nodes. However: - Is this the right approach to achieve this goal? - Which tool should I use for reliable bidirectional synchronization? - What are the potential risks (conflicts, corruption) and precautions to take? Have you implemented such a configuration in production before? Thank you for your advice. -- Michel

1 0

Re: Performance issue after tentacle upgrade
by Moti 21 Jul '26

21 Jul '26

Hello Frédéric Thanks for your response About iops; yes iops drop is good news, but this amount (far more than 50%) was a bit suspicious, and it shouldn't worsen latency. I was worried that slow disk might be doing what fast disk is supposed to do, but that's also not the case, so it looks like io just vanished to near zero. However this very well could be a separate problem/topic. About config defaults; thanks for that link, that's very handy. I checked the diff and tried to point out what might be problematic, these are the configs I tested changing, however did not get any better latency in response: - bluestore_elastic_shared_blob -> false - swapping osd_op_num_shards_hdd and osd_op_num_threads_per_shard that you mentioned - disabling ec partial reads - decreasing rgw thread pool size (I had this set to a higher value than default on reef) - decreasing rgw log trim interval I haven't tried changing envelope mode, I will, and thanks for the hint, but is it straight forward to upgrade wal back to envelope mode using bluestore tool if downgrade didn't seem to solve the issue? Also meanwhile since my original post I've been doing more tests, and the same case happened on other clusters as well (VMs with no separate fast/slow disks), the only time latency peak didn't happen, was when I tested on a single zone cluster, which is confusing since all my multi zone clusters use separate RGWs for running sync threads, and I'm seeing latency peak on client RGWs, not the sync ones. It might as well be another parameter that's different on the single zone one and I'm missing it, but that's my initial observation. Best regards Moti On Thu, 16 Jul 2026, 19:34 Frédéric Nass via ceph-users, <ceph-users(a)ceph.io> wrote: > Hi Moti, > > Anything suspicious on 'ceph osd perf' output? > > > NVMe disks' iops went from something around 5-6K constant io per second > to almost zero (less than a hundred per second, on all host/OSDs, with some > non-frequent high peaks), > > That would be pretty good news, actually. This would likely be the result > of the new 'bluefs_wal_envelope_mode' [1]. Its own description says it > "reduces by ~50% the amount of fdatasync syscalls" on the RocksDB WAL. :-) > I don't expect this to explain the performance drop at this time, though. > > I'd recommend that you check the new/old settings and default values > between both versions using our online Config Diff tool here [2]. It might > ring a bell. > > The tool highlights that the default values for 'osd_op_num_shards_hdd' and > 'osd_op_num_threads_per_shard_hdd' were swapped between both versions. I'm > not sure what the impact could be on your cluster in terms of performance, > but if you're using mClock, reverting this change or trying 'wpq' osd > scheduler instead might help. You can run 'ceph config set osd osd_op_queue > wpq' and restart all OSDs to figure out if mClock has anything to do with > what you observe. > > If swapping values or changing the osd scheduler to wpq doesn't help, you > could try disabling 'bluefs_wal_envelope_mode' and running > 'ceph-bluestore-tool --command downgrade-wal-to-v1' (make sure to test > this out on a single OSD first). > > Best regards, > > Frédéric > > [1] > > https://github.com/ceph/ceph/blob/708737d3ec0669a8e01265d9699d6f939debec13/… > [2] > > https://analyzer.clyso.com/#/analyzer/config-diff?source=v18.2.4&target=v20… > > > Frédéric Nass > > Senior Ceph Engineer > > Ceph Ambassador, France > > +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> > > frederic.nass(a)clyso.com > > www.clyso.com > > Hohenzollernstr. 27, 80801 Muni > <https://www.google.com/maps/search/Hohenzollernstr.+27,+80801+Muni?entry=gm…> > ch > > Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 > > > > Le jeu. 9 juil. 2026 à 15:33, Moti via ceph-users <ceph-users(a)ceph.io> a > écrit : > > > Hello everyone! > > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing > as > > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET > and > > HEAD requests. > > > > There is no resource cap or significant increase in usage is visible. > > Neither fast ec nor any other new features are enabled yet, just did the > > upgrade and left it as is. For measuring latency I have a services that > > probes the cluster with consistent load so it's not a difference in load > > either. PG status history does not show any pg operation being heavily in > > process. > > > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db > block. > > Both ec and replicated pools exist. > > The strange hint was: NVMe disks' iops went from something around 5-6K > > constant io per second to almost zero (less than a hundred per second, on > > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > > not much changed. db blocks on NVMe are still attached to OSDs with no > > issue. > > > > I can't find a clue to explain this. Got the same results on 3 separate > > clusters (all went version 18 --> 20, same hybrid build and configs). > > Nothing in config dump changes (changed defaults) catches my attention to > > have caused this. > > > > Anyone had similar issues? Any tips and ideas on what to check? > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

v21.1.0 Umbrella RC0 released
by Yuri Weinstein 20 Jul '26

20 Jul '26

This is the first release candidate for Umbrella. Ubuntu 22.04, 24.04, CentOS 9, Rocky Linux 10, Debian 12 (Bookworm) and Debian 13 (Trixie) were built for this RC0. Please note: these release notes are a draft and incomplete. Feature highlights: - RADOS/OSD: New Optimized ("Fast") Erasure Coding I/O path (direct/sync reads, OMAP support, deep scrub); mon RocksDB backup/restore; stretch cluster improvements; mClock scheduler refactor; continued Crimson/Seastore EC and perf work. - RGW: Full object dedup (tech preview); bucket logging; cloud restore from Glacier/tape; AES-256-GCM encryption + security fixes; expanded IAM/Account and STS/OIDC support. - Dashboard: Large-scale migration to IBM Carbon design; expanded NVMe/TCP management; SMB cluster support; OAuth2 SSO; Angular 19. - cephadm/Orchestrator: New unified certmgr for TLS cert management; SMB/Samba managed service; expanded NVMe-oF gateway spec; new hardware monitoring (Atollon BMCs); NFS-Ganesha active-active mode. - CephFS/MDS: FSCrypt client-side encryption; multi-threaded cephfs-mirror; new python bindings; subvolume improvements. - ceph-volume: Migrated to dataclasses + udev-based device detection (perf/reliability); seastore OSD support. - Build/Packaging/QA: Ubuntu 24.04 (Noble) and Rocky Linux 10 support added; the Ceph container's base OS image has changed to rockylinux:10; CMake 3.22.1+/CMake-4; C++23. - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.0.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: f7eb5f7c8097648828a6b31f4590c3ee3661c95a

1 0

Re: Survey: Is anyone using 32-bit builds of the CephFS kernel client?
by Xiubo Li 17 Jul '26

17 Jul '26

Just forward this since I wasn't in the ceph-users mail list and it was reject yesterday. Thanks Xiubo Li On Thu, 16 Jul 2026 at 16:53, Xiubo Li <xiubo.li(a)clyso.com> wrote: > > Hi everyone, > > We are considering a performance improvement for the CephFS kernel > client (kclient) that would be significantly simpler and cleaner to > implement as 64-bit-only. Supporting 32-bit in this case adds > non-trivial complexity and ongoing maintenance burden, so we'd like to > understand whether continuing 32-bit support is still necessary. > > Before making any decision, we want to hear from the community: > > - Are you currently using, or planning to use, 32-bit (arm/armhf, > i386/i686) builds of the CephFS kernel client? > - If yes, is this for production workloads? Or legacy/testing purposes? > - What is your platform (distribution, kernel version, architecture)? > > If there are no active 32-bit users, we would proceed with a > 64-bit-only implementation and clearly document the deprecation in the > release notes with appropriate warnings. > > If 32-bit support is still needed, we will explore alternative > approaches — for example, keeping the current code path for 32-bit > while the new fast path is 64-bit-only, or finding a way to cleanly > abstract the difference. > > Please reply by August 6, 2026 so we can factor your feedback into the plan. > > Thanks! > Xiubo Li

1 0

Re: Performance issue after tentacle upgrade
by Frédéric Nass 16 Jul '26

16 Jul '26

Hi Moti, Anything suspicious on 'ceph osd perf' output? > NVMe disks' iops went from something around 5-6K constant io per second to almost zero (less than a hundred per second, on all host/OSDs, with some non-frequent high peaks), That would be pretty good news, actually. This would likely be the result of the new 'bluefs_wal_envelope_mode' [1]. Its own description says it "reduces by ~50% the amount of fdatasync syscalls" on the RocksDB WAL. :-) I don't expect this to explain the performance drop at this time, though. I'd recommend that you check the new/old settings and default values between both versions using our online Config Diff tool here [2]. It might ring a bell. The tool highlights that the default values for 'osd_op_num_shards_hdd' and 'osd_op_num_threads_per_shard_hdd' were swapped between both versions. I'm not sure what the impact could be on your cluster in terms of performance, but if you're using mClock, reverting this change or trying 'wpq' osd scheduler instead might help. You can run 'ceph config set osd osd_op_queue wpq' and restart all OSDs to figure out if mClock has anything to do with what you observe. If swapping values or changing the osd scheduler to wpq doesn't help, you could try disabling 'bluefs_wal_envelope_mode' and running 'ceph-bluestore-tool --command downgrade-wal-to-v1' (make sure to test this out on a single OSD first). Best regards, Frédéric [1] https://github.com/ceph/ceph/blob/708737d3ec0669a8e01265d9699d6f939debec13/… [2] https://analyzer.clyso.com/#/analyzer/config-diff?source=v18.2.4&target=v20… Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> frederic.nass(a)clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Le jeu. 9 juil. 2026 à 15:33, Moti via ceph-users <ceph-users(a)ceph.io> a écrit : > Hello everyone! > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing as > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET and > HEAD requests. > > There is no resource cap or significant increase in usage is visible. > Neither fast ec nor any other new features are enabled yet, just did the > upgrade and left it as is. For measuring latency I have a services that > probes the cluster with consistent load so it's not a difference in load > either. PG status history does not show any pg operation being heavily in > process. > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db block. > Both ec and replicated pools exist. > The strange hint was: NVMe disks' iops went from something around 5-6K > constant io per second to almost zero (less than a hundred per second, on > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > not much changed. db blocks on NVMe are still attached to OSDs with no > issue. > > I can't find a clue to explain this. Got the same results on 3 separate > clusters (all went version 18 --> 20, same hybrid build and configs). > Nothing in config dump changes (changed defaults) catches my attention to > have caused this. > > Anyone had similar issues? Any tips and ideas on what to check? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: umbrella 21.1.0 RC QE validation status
by Venky Shankar 14 Jul '26

14 Jul '26

Hi Yuri, On Tue, Jul 14, 2026 at 1:46 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Waiting for approvals/reviews for: > > rados: Travis > > rgw: Adam Emerson > fs: Venky fs approved - failures are https://tracker.ceph.com/projects/cephfs/wiki/Umbrella#Umbrella-v2110-rc1 > orch/ceph-volume: Adam King & Guillaume > > upgrade: Kamoltat, Patrick > > powercycle: Brad (I saw your tracker, but pls approve this) > > On Mon, Jul 6, 2026 at 8:00 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/76757 >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! >> >> Seeking approvals/reviews for: >> >> smoke: Radek, Laura, Nitzan >> >> rados: Radek, Laura, Notzan, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya >> >> nvmeof: PASSED >> >> upgrade: Laura, Patrick >> >> upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: TBD (https://github.com/ceph/ceph/pull/69888) >> upgrade-clients:client-upgrade-squid-umbrella-umbrella: TBD (https://github.com/ceph/ceph/pull/69887) >> >> perf-basic: PASSED >> >> crimson-rados: Matan >> >> powercycle: Brad >> >> TIA -- Cheers, Venky

1 0

v19.2.5 Squid released
by Yuri Weinstein 14 Jul '26

14 Jul '26

This is the fifth backport release in the Squid series. We recommend all users update to this release. ## Notable Changes MDS (Metadata Server) --------------------- - Fixed a segmentation fault relating to MDSContext completion and request queuing. OSD (Object Storage Daemon) --------------------------- - Rollback & Vector Fixes: Corrected rollback logic for partial write object information (OI) and optimized Erasure Coding (EC) by ensuring Twiddle creates a full-sized vector. RGW (RADOS Gateway) ------------------- - TLS 1.3 Ciphersuites: Introduced the ssl_ciphersuites configuration option for the Beast front-end to support TLS 1.3 cipher customizations. - S3 PutObject ACLs Fix: Fixed a bug where PutObject requests were erroneously rejected under BlockPublicAcls due to an incorrect comparison return value from canned_acl.compare(). - Multisite Sync Robustness: Resolved an infinite loop scenario in RGWBucketFullSyncCR when a source bucket was deleted mid-sync by introducing a mechanism to clear stateful data in reused bucket_list_result objects. - SNS Policy Evaluation: Updated ListTopics to use the account root ARN for policy evaluation when invoked by a non-root account user, preventing unexpected implicit denies from identity-based policies. - Presigned URLs: Fixed an internal server error occurring during the authentication process of presigned URLs. - Prerequest Hook: Restructured the order of execution so that the prerequest hook now runs after the authorization process has completed. - Object Tagging: Added handling for plain-text object tags during execution of RGWObjTags::decode(). - Beast Frontend Stability: Applied a strand executor to the Beast timeout timer to avoid concurrent socket access and potential race conditions. - Bucket Statistics: Added a fix for bucket stats execution when the underlying bucket index does not exist. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v19-2-5-squid-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.5.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: abc7aa7f2701e5d46878fd5e6bb7e2955f1a395a

1 0

Re: Performance issue after tentacle upgrade
by Eugen Block 14 Jul '26

14 Jul '26

Hi, I'm not sure if it could be related to summer vacation, but I feel like there should have been several reports similar to yours by now. I don't have upgraded any production cluster to 20 yet, so I can't really deny or confirm anything. Did you upgrade the OS as well? Maybe there were some relevant changes there. Or some non-persistent configs were overwritten, either in the OS or Ceph. Was there more deep-scrubbing after the upgrade which could have impacted the IO? Regards, Eugen Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > Hello everyone! > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing as > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET and > HEAD requests. > > There is no resource cap or significant increase in usage is visible. > Neither fast ec nor any other new features are enabled yet, just did the > upgrade and left it as is. For measuring latency I have a services that > probes the cluster with consistent load so it's not a difference in load > either. PG status history does not show any pg operation being heavily in > process. > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db block. > Both ec and replicated pools exist. > The strange hint was: NVMe disks' iops went from something around 5-6K > constant io per second to almost zero (less than a hundred per second, on > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > not much changed. db blocks on NVMe are still attached to OSDs with no > issue. > > I can't find a clue to explain this. Got the same results on 3 separate > clusters (all went version 18 --> 20, same hybrid build and configs). > Nothing in config dump changes (changed defaults) catches my attention to > have caused this. > > Anyone had similar issues? Any tips and ideas on what to check? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: umbrella 21.1.0 RC QE validation status
by Brad Hubbard 13 Jul '26

13 Jul '26

On Tue, Jul 14, 2026 at 6:18 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Waiting for approvals/reviews for: > > rados: Travis > > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > > upgrade: Kamoltat, Patrick > > powercycle: Brad (I saw your tracker, but pls approve this) > Approved. > > On Mon, Jul 6, 2026 at 8:00 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/76757 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > > Seeking approvals/reviews for: > > > > smoke: Radek, Laura, Nitzan > > > > rados: Radek, Laura, Notzan, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya > > > > nvmeof: PASSED > > > > upgrade: Laura, Patrick > > > > upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: TBD ( > > https://github.com/ceph/ceph/pull/69888) > > upgrade-clients:client-upgrade-squid-umbrella-umbrella: TBD ( > > https://github.com/ceph/ceph/pull/69887) > > > > perf-basic: PASSED > > > > crimson-rados: Matan > > > > powercycle: Brad > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Cheers, Brad

1 0

Re: umbrella 21.1.0 RC QE validation status
by Adam Emerson 13 Jul '26

13 Jul '26

On 06/07/2026, Yuri Weinstein via ceph-users wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/76757 > > Release Notes - TBD > LRC upgrade - TBD > > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > Seeking approvals/reviews for: > > smoke: Radek, Laura, Nitzan > > rados: Radek, Laura, Notzan, Afreen, Travis > rgw: Adam Emerson RGW Approved.

1 0

Re: Argonaut: an open-source terminal UI for Ceph administration
by Sinan Polat 13 Jul '26

13 Jul '26

Hi Thomas, Thanks for taking the time to write all that out, it is genuinely useful. Let me deal with the AI thing first since you raised it. I did lean on AI a lot while building this and I want to be clear I was never trying to hide that. I have been open about it elsewhere (I said as much on Reddit) and I am actually working on a blog post about how the whole thing came together, so it's not something I'm shy about. It wasn't "prompt and ship." I acted as the architect and reviewer, I made every design, UX, and safety decision, set the hard constraints (strict separation between UI, logic, and a single isolated Ceph transport; every destructive action confirms and previews the exact ceph command before it runs; version-aware JSON decoding in one place), reviewed the changes, and maybe importantly I myself validated it against real Reef, Squid, and Tentacle clusters (the decoding layer is covered by golden fixtures captured from live clusters), plus a full audit pass before release. On the single commit: that is a fair read and I owe an explanation. The project does have a real, iterative development history, it lives in a private dev repo. The public repo is a clean snapshot because some early commits contained infrastructure credentials I didn't want to publish. I can see how one squashed commit looks like an opaque code-dump. That was absolutely not the intent. And your point about not handing cluster access to a tool you haven't read, I'd do exactly the same :-) It authenticates the same way the ceph CLI does, with your keyring and your permissions, and there is a mock mode (--client mock) that runs on fake in-memory data if you just want to click around without touching a real cluster. About your feedback, which I really appreciate: The connected-clients overview is a great idea! The same goes for a view with the real recovery ETA. The hung-but-still-marked-up OSDs sound nasty! I will have to think about how you'd even detect that reliably. Perhaps you can share that part of your script? Unrelated note, based on other feedback about the name the project is now called Siphon. Repo here: https://github.com/cinpol/siphon Thanks again for the honest feedback :) Cheers, Sinan Op ma 13 jul 2026 om 16:58 schreef Thomas Schneider via ceph-users < ceph-users(a)ceph.io>: > Because some users won't trust tools that have limited or no human > oversight. > Having a repository that is written by "AI" with one single large commit > calls into question > * the maintainability of the code > * the understanding of the author of the code > * the longevity of the project > * the safety > > If the software was read only it _could_ be ok, but I would never give a > vibe coded piece of software that I didn't read completely access to my > cluster. > If a human maintains a piece of software I can look at the commit > history, understand the design, be reasonable sure to follow the code > and could in theory follow the development. LLM code-dumps that span > thousands of lines and aren't even completely shown in the github diff > view are a nightmare. > > Oh and I find it intransparent and questionable if someone isn't open > about using AI. If a project is transparent about it's usage I'm more > willing to give them the benefit of the doubt as if they don't mention > it at all. > > Kind regards, > Thomas > > On 2026-07-13 16:43, Anthony D'Atri wrote: > > Why does that matter? > > > >> On Jul 13, 2026, at 10:41 AM, Thomas Schneider via ceph-users > >> <ceph-users(a)ceph.io> wrote: > >> Looking at the README and "commit history" this kind of screams vibe > >> coded which turns me off from even trying to use it tbh. If you used > >> LLMs you should be transparent about it. > > -- > Thomas Schneider > IT.SERVICES > Wissenschaftliche Informationsversorgung > Ruhr-Universität Bochum | 44780 Bochum > Telefon: +49 234 32 23939 > http://www.it-services.rub.de/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

umbrella 21.1.0 RC QE validation status
by Yuri Weinstein 13 Jul '26

13 Jul '26

Details of this release are summarized here: https://tracker.ceph.com/issues/76757 Release Notes - TBD LRC upgrade - TBD *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* Seeking approvals/reviews for: smoke: Radek, Laura, Nitzan rados: Radek, Laura, Notzan, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: PASSED upgrade: Laura, Patrick upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: TBD ( https://github.com/ceph/ceph/pull/69888) upgrade-clients:client-upgrade-squid-umbrella-umbrella: TBD ( https://github.com/ceph/ceph/pull/69887) perf-basic: PASSED crimson-rados: Matan powercycle: Brad TIA

1 1

Re: Argonaut: an open-source terminal UI for Ceph administration
by Sinan Polat 13 Jul '26

13 Jul '26

Thanks for the feedback and good catch on the release history. The name is actually a deliberate nod to Ceph's first stable release. It felt fitting for a tool aimed at Ceph operators to point back to where Ceph started. That said, I take your point about the potential for confusion, I am genuinely open to revisiting it. For now, though, I would like to keep the focus on the tool itself and whether it is actually useful. If you (or anyone on the list) get a chance to try it I would really value feedback on the functionality: which day-to-day operations you want it to handle and anything that feels missing or wrong. Thanks again for taking the time to look. Sinan Op za 11 jul 2026 om 17:58 schreef Eugen Block via ceph-users < ceph-users(a)ceph.io>: > The only comment I have (at the moment) is the naming. Argonaut was > the first stable Ceph release: > > https://docs.ceph.com/en/latest/releases/argonaut/ > > So that might be a bit misleading here, at least in combination with > Ceph. There have been several discussions about naming on this list as > well (for example the Ceph Nautilus release vs. file browser on gnome > desktop). > > Just my 2 cents. > > Zitat von Sinan Polat via ceph-users <ceph-users(a)ceph.io>: > > > Hi everyone, > > > > I have been working on a new open-source project called Argonaut. I would > > love to get feedback from the Ceph community. > > > > Argonaut is a terminal UI for administering Ceph clusters. The > inspiration > > comes from tools like k9s and lazydocker: a keyboard-driven interface for > > common operational tasks while remaining transparent about what is > > happening under the hood. > > > > It is still an early release and I am particularly interested in feedback > > from operators who manage production clusters. > > > > Some questions I would like to ask you: > > - Which operational workflows consume the most time today? > > - Which information do you find difficult to discover quickly? > > - Which features would make a TUI genuinely useful in your daily > workflow? > > > > The project is available here: https://github.com/cinpol/argonaut > > > > Any feedback, feature requests, bug reports, or criticism would be > greatly > > appreciated. > > > > Kind regards > > Sinan Polat > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 2

Re: Argonaut: an open-source terminal UI for Ceph administration
by Thomas Schneider 13 Jul '26

13 Jul '26

Because some users won't trust tools that have limited or no human oversight. Having a repository that is written by "AI" with one single large commit calls into question * the maintainability of the code * the understanding of the author of the code * the longevity of the project * the safety If the software was read only it _could_ be ok, but I would never give a vibe coded piece of software that I didn't read completely access to my cluster. If a human maintains a piece of software I can look at the commit history, understand the design, be reasonable sure to follow the code and could in theory follow the development. LLM code-dumps that span thousands of lines and aren't even completely shown in the github diff view are a nightmare. Oh and I find it intransparent and questionable if someone isn't open about using AI. If a project is transparent about it's usage I'm more willing to give them the benefit of the doubt as if they don't mention it at all. Kind regards, Thomas On 2026-07-13 16:43, Anthony D'Atri wrote: > Why does that matter? > >> On Jul 13, 2026, at 10:41 AM, Thomas Schneider via ceph-users >> <ceph-users(a)ceph.io> wrote: >> Looking at the README and "commit history" this kind of screams vibe >> coded which turns me off from even trying to use it tbh. If you used >> LLMs you should be transparent about it. -- Thomas Schneider IT.SERVICES Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 Bochum Telefon: +49 234 32 23939 http://www.it-services.rub.de/

1 0

Re: Argonaut: an open-source terminal UI for Ceph administration
by Thomas Schneider 13 Jul '26

13 Jul '26

> I have been working on a new open-source project called Argonaut. I > would > love to get feedback from the Ceph community. Looking at the README and "commit history" this kind of screams vibe coded which turns me off from even trying to use it tbh. If you used LLMs you should be transparent about it. > Argonaut is a terminal UI for administering Ceph clusters. The > inspiration > comes from tools like k9s and lazydocker: a keyboard-driven interface > for > common operational tasks while remaining transparent about what is > happening under the hood. Sounds great in concept! Especially being transparent about what happens. > Some questions I would like to ask you: > - Which operational workflows consume the most time today? > - Which information do you find difficult to discover quickly? > - Which features would make a TUI genuinely useful in your daily > workflow? Most admins probably have (shell) scripts for most actions so answering is kind of hit or miss. Some actions we have scripts for that take some time. Many of the things listed below are possible/easier over the dashboard. - Listing connected clients RBD, RGW, CephFS, ... every client connected is listed through a different interface. If I quickly want to check who is using what service right now I have to use my scripts. - Actual ETA for rebalance/recovery What it says on the tin. We got a script that checks how many objects need to be moved, are moved per second and depending on the remaining count map it to a curve to guesstimate an ETA - Diagnosing OSD problems This is probably a me problem, but: Sometimes OSDs hang intransparently (they are marked as up, react to status and even are able to be benched, but don't work and block the cluster) would be nice to test them using a tui instead of my script - Bucket usage/User usage by bucket we have automated usage statistics by user/bucket - it's important that the TUI supports different zone/realms Dumping everything to a json file or something like that would be a plus - managing data movement When doing maintenance, adding hosts, replacing OSDs and so on we often manage the data movement using scripts/by hand to minimize data movement. Doing this using a tui would be interesting. - better PG overview This is not something that normally takes up much time, but mostly because we do it seldom. We got a multi site cluster which (without extra patches) limits our usage of the autoscaler (it doesn't work at all without our patches) being able to browse pools by pg_num/size with some usage info Like I said got a heap of tools of my own and am using scripts like https://github.com/TheJJ/ceph-balancer as well. For Ceph newcomers something that would do something like the https://analyzer.clyso.com/ could be neat. Ceph warns you if you do really stupid things, but some tool that tells you if you did slightly stupid things or could optimize your setup could be neat. Anyway most of that should be relatively specific to my cluster and I probably forgot things I do by muscle memory, but I hope I at least gave you some inspiration. If anything what I said is wrong or could be done easily with the tools available I would be glad if someone pointed me in the right direction as well. Kind regards Thomas On 2026-07-11 17:32, Sinan Polat via ceph-users wrote: > Hi everyone, > > I have been working on a new open-source project called Argonaut. I > would > love to get feedback from the Ceph community. > > Argonaut is a terminal UI for administering Ceph clusters. The > inspiration > comes from tools like k9s and lazydocker: a keyboard-driven interface > for > common operational tasks while remaining transparent about what is > happening under the hood. > > It is still an early release and I am particularly interested in > feedback > from operators who manage production clusters. > > Some questions I would like to ask you: > - Which operational workflows consume the most time today? > - Which information do you find difficult to discover quickly? > - Which features would make a TUI genuinely useful in your daily > workflow? > > The project is available here: https://github.com/cinpol/argonaut > > Any feedback, feature requests, bug reports, or criticism would be > greatly > appreciated. > > Kind regards > Sinan Polat > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Thomas Schneider IT.SERVICES Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 Bochum Telefon: +49 234 32 23939 http://www.it-services.rub.de/

1 0

Re: Argonaut: an open-source terminal UI for Ceph administration
by Anthony D'Atri 12 Jul '26

12 Jul '26

> The name is actually a deliberate nod to Ceph's first stable release. It > felt fitting for a tool aimed at Ceph operators to point back to where Ceph > started. That said, I take your point about the potential for confusion, I > am genuinely open to revisiting it. IMHO you’re good. There’s one in every crowd. > > For now, though, I would like to keep the focus on the tool itself and > whether it is actually useful. I’m confident someone will find it useful. > If you (or anyone on the list) get a chance > to try it I would really value feedback on the functionality: which > day-to-day operations you want it to handle and anything that feels missing > or wrong. > > Thanks again for taking the time to look. > > Sinan > > Op za 11 jul 2026 om 17:58 schreef Eugen Block via ceph-users < > ceph-users(a)ceph.io>: > >> The only comment I have (at the moment) is the naming. Argonaut was >> the first stable Ceph release: >> >> https://docs.ceph.com/en/latest/releases/argonaut/ >> >> So that might be a bit misleading here, at least in combination with >> Ceph. There have been several discussions about naming on this list as >> well (for example the Ceph Nautilus release vs. file browser on gnome >> desktop). >> >> Just my 2 cents. >> >> Zitat von Sinan Polat via ceph-users <ceph-users(a)ceph.io>: >> >>> Hi everyone, >>> >>> I have been working on a new open-source project called Argonaut. I would >>> love to get feedback from the Ceph community. >>> >>> Argonaut is a terminal UI for administering Ceph clusters. The >> inspiration >>> comes from tools like k9s and lazydocker: a keyboard-driven interface for >>> common operational tasks while remaining transparent about what is >>> happening under the hood. >>> >>> It is still an early release and I am particularly interested in feedback >>> from operators who manage production clusters. >>> >>> Some questions I would like to ask you: >>> - Which operational workflows consume the most time today? >>> - Which information do you find difficult to discover quickly? >>> - Which features would make a TUI genuinely useful in your daily >> workflow? >>> >>> The project is available here: https://github.com/cinpol/argonaut >>> >>> Any feedback, feature requests, bug reports, or criticism would be >> greatly >>> appreciated. >>> >>> Kind regards >>> Sinan Polat >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: autoscale-status -no output
by listy 11 Jul '26

11 Jul '26

MONERO I created, I also removed other two-default classes, manually. I do use, prefer containerized env/lab, with: -> $ cephadm bootstrap ... all the hardware in VMs (Qemu KVM) which ceph use, is virtualized, as opposed to passed-through. In this lab of mine, used are host's LVM volumes utilized by Qemu XML's as: ... <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/cs_whale/ceph'/> ... In guests (hosts for ceph), centos 9, these are as 'virtio_blk': ... [ 1.580490] virtio_blk virtio4: [vdb] 39321600 4096-byte logical blocks (161 GB/150 GiB) ... I get that this is a not a use-case which ceph-devel think of much, if at all. thanks, L.

1 0

Re: Argonaut: an open-source terminal UI for Ceph administration
by Eugen Block 11 Jul '26

11 Jul '26

The only comment I have (at the moment) is the naming. Argonaut was the first stable Ceph release: https://docs.ceph.com/en/latest/releases/argonaut/ So that might be a bit misleading here, at least in combination with Ceph. There have been several discussions about naming on this list as well (for example the Ceph Nautilus release vs. file browser on gnome desktop). Just my 2 cents. Zitat von Sinan Polat via ceph-users <ceph-users(a)ceph.io>: > Hi everyone, > > I have been working on a new open-source project called Argonaut. I would > love to get feedback from the Ceph community. > > Argonaut is a terminal UI for administering Ceph clusters. The inspiration > comes from tools like k9s and lazydocker: a keyboard-driven interface for > common operational tasks while remaining transparent about what is > happening under the hood. > > It is still an early release and I am particularly interested in feedback > from operators who manage production clusters. > > Some questions I would like to ask you: > - Which operational workflows consume the most time today? > - Which information do you find difficult to discover quickly? > - Which features would make a TUI genuinely useful in your daily workflow? > > The project is available here: https://github.com/cinpol/argonaut > > Any feedback, feature requests, bug reports, or criticism would be greatly > appreciated. > > Kind regards > Sinan Polat > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: autoscale-status -no output
by Anthony D'Atri 11 Jul '26

11 Jul '26

> > Deployment process did create both _hdd_ & _ssd_ classes. > I've read somewhere in docs that device classes are more of a concept rather than a real thing Hmm. I've written that the *names* are arbitrary, so one can assign different ones, but it's best to only do that if you have a specific reason. Say you have NVMe SSDs, SATA SSDs, and SATA HDDs, and enough of each that you want to constrain which pool goes on which. By default, without anything exotic, the HDDs will be reported by the kernel as rotational, so OSDs on them will get the device class "hdd". The SATA and NVMe SSDs will be reported by the kernel as nonrotational, so they will get the device class "ssd". With both, one might, for example wish to place pools: * CephFS .meta : NVMe SSDs * CephFS first .data: SATA SSDs * CephFS additional .data pools: HDDs In that case one might assign the NVMe SSDs a CRUSH device class of "nvme" and define a CRUSH rule that only uses those. In a cluster with mixed media, one almost never wants a given pool to use more than one, like you wouldn't want a CephFS .meta pool on both HDDs and SSDs, or the data pools eating the SSD capacity. > - which would confirm since my cluster is a lab, all in virtual env, all storage devices/drives all the same "type" and yet these two classes were auto-created (and used by default if I remember, at osd-creation time) How is your cluster deployed? Something would have to go out of its way to create a device class called MONERO. I suspect that you might have misinterpreted a configuration item. Virtual storage devices as seen by Ceph may be classed differently than one would want, usually as non-rotational when there are SSDs behind the abstraction. This does matter in some ways, as some configuration options have _hdd and _ssd variants that are used depending on the rotational attribute. In such cases a udev rule can coerce what you want. Are you using Proxmox or Rook or something to deploy? > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Argonaut: an open-source terminal UI for Ceph administration
by Sinan Polat 11 Jul '26

11 Jul '26

Hi everyone, I have been working on a new open-source project called Argonaut. I would love to get feedback from the Ceph community. Argonaut is a terminal UI for administering Ceph clusters. The inspiration comes from tools like k9s and lazydocker: a keyboard-driven interface for common operational tasks while remaining transparent about what is happening under the hood. It is still an early release and I am particularly interested in feedback from operators who manage production clusters. Some questions I would like to ask you: - Which operational workflows consume the most time today? - Which information do you find difficult to discover quickly? - Which features would make a TUI genuinely useful in your daily workflow? The project is available here: https://github.com/cinpol/argonaut Any feedback, feature requests, bug reports, or criticism would be greatly appreciated. Kind regards Sinan Polat

1 0

Re: autoscale-status -no output
by Anthony D'Atri 11 Jul '26

11 Jul '26

.mgr will always have a single PG. If for some reason that PG wasn't placed on any OSDs, you'd have a bigger problem. > On Jul 11, 2026, at 11:14 AM, listy via ceph-users <ceph-users(a)ceph.io> wrote: > > Yes, changing: > > -> $ ceph osd pool set .mgr crush_rule replicated_rule_MONERO > > fixes: > > -> $ ceph osd pool autoscale-status > > What was (not)happening there? _.mgr_ had no OSDs assigned, if so how did it work, showing no errors (which I could spot)? You had a situation that was more complex than the PG autoscaler is equipped to deal with. Like I said, I have an RFE in to prevent this kind of thing. > > thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: autoscale-status -no output
by Anthony D'Atri 11 Jul '26

11 Jul '26

There you go. default~MONERO is unusual, though. Please send: ceph df | head -10 How / why did you name a device class MONERO? By default OSDs will populate as "hdd" or "ssd" depending on the block device's `rotational` attribute. On Jul 11, 2026, at 11:00 AM, listy via ceph-users <ceph-users(a)ceph.io> wrote: -> $ ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "replicated_rule_MONERO", "type": 1, "steps": [ { "op": "take", "item": -16, "item_name": "default~MONERO" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] -> $ ceph osd crush class ls [ "MONERO" ] _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: autoscale-status -no output
by Anthony D'Atri 11 Jul '26

11 Jul '26

I documented this a few years ago, see the blue note here: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#viewing-… > Hi guys. > > I get no output from: > -> $ ceph osd pool autoscale-status > > I read docs and I thought I should have checked: > > -> $ ceph osd pool get .mgr crush_rule > crush_rule: replicated_rule > -> $ ceph osd pool get cephfs.MONERO.data crush_rule > crush_rule: replicated_rule_MONERO > -> $ ceph osd pool get cephfs.MONERO.meta crush_rule > crush_rule: replicated_rule_MONERO Please send `ceph osd crush rule dump` replicated_rule most likely looks like this: { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, which does not specify a CRUSH device class. Do you have both HDDs and SSDs? replicated_rule_MONERO probably looks like this: { "rule_id": 3, "rule_name": "replicated_hdd", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, Note the different in item_name: it specifies a device class via a shadow CRUSH tree. Which means that the rules in use are seen as spanning multiple CRUSH roots and the autoscaler "can't even". I put in an RFE for this last year. I've asked that the .mgr pool be created with a device class and that a health warning be raised if this situation detected. Depending on your constellation of rules, the best bet may be to use the CLI to change the CRUSH rule for the .mgr pool to replicated_rule_MONERO. Send the requested dump and I can say for sure. That said, I'd keep rule names lowercase, I don't *know* that mixed causes a problem, but I like to not tempt fate. > > All I have are above three pools - _.mgr_ created at/by deployment(in containers) > > I must be missing something else - what that might be? > I see no clear & obvious errors in the logs. > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2