- ceph-users - lists.ceph.io

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Andras Pataki 24 Jul '26

24 Jul '26

I'm very glad that we have a working theory of what is happening in the MDS - it seems like the client process experience, the logs and the theory on what the code is doing all match up. You mentioned that it takes some load/turnover on the MDS to trim the cache a specific way - this might be the reason I could not reproduce the issue on a test system. Our production MDS is quite busy with a couple thousand clients hammering on its cache constantly - so it probably does test most of the caching related code paths. When you have a patch that you feel is safe to try on a large production system, I can patch our MDS and test it on our workload. Thanks, Andras On 7/24/26 4:37 AM, Xiubo Li via ceph-users wrote: > Hi Andras, > > Thanks very much for your logs. This is very helpful. Following is > Dhairya's and my analysis: > > Your issue was most probably triggered by cases like: > > 1. COW deletes GSK: > items: snap dentry [2, 0x1c9] primary <-- preserved for snapshot > head dentry [0x1ca, CEPH_NOSNAP] NULL <-- COW correctly set first > > 2. Cache trims the COW head NULL dentry: > null + clean --> clear_complete = FALSE > --> STATE_COMPLETE preserved! <-- KEY! > --> items now only has snap dentry > > 3. A HEAD lookup comes in for GSK: > lookup("GSK", CEPH_NOSNAP) --> lower_bound(CEPH_NOSNAP, "GSK", h) > --> no head-level dentry found --> returns NULL > > MISS path: is_complete() = TRUE <-- still complete! > --> add_null_dentry("GSK") <-- default first=2! > --> creates [2, CEPH_NOSNAP] NULL > <------------------------------------ HERE ------------------ > > 4. Now items has: > snap dentry [2, 0x1c9] primary > NULL dentry [2, CEPH_NOSNAP] <-- first=2 covers ALL snapids > > 5. Cache trim --> trims the snap dentry: > primary --> clear_complete = TRUE --> STATE_COMPLETE cleared. This > could happen under workload pressure > > 6. Snap lookup comes in: > dir incomplete, only NULL dentry [2,head] left > 2 < 0x1c9 < CEPH_NOSNAP hit ENOENT ----------> in CDir: lookup() > it will just return the NULL dentry, which is [2, CPEH_NOSNAP] and > then the kclient cached it. > > So the root cause is in the MDS, and in kclient we still need to fix > kclient to prevent caching NULL dentry for this bug. > > Dhairya or I will fix the MDS side bug later. I have sent out the > kclient patches and will create a PR to fix the libcephfs later. > > Thanks & Regards > Xiubo > > Xiubo > > On Fri, 24 Jul 2026 at 13:07, Dhairya Parmar via dev <dev(a)ceph.io> wrote: >> Hey Andas, sorry to stress your MDS a bit and thanks a ton for the debug logs. This is really helpful. This is the most important bit in [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] and my suspicion is on [2,head] range. Let's see what I find. I'll keep you posted. BTW @Xiubo Li mentioned he isn't able to reproduce this locally. Would you be able to share the reproducer if possible? >> >> On Thu, Jul 23, 2026 at 10:19 PM Andras Pataki <apataki(a)flatironinstitute.org> wrote: >>> Hi Xiubo and Dhairya, >>> >>> I had a fruitful debugging session with my clients and MDS. The problem seems to be with the MDS, not with the client. >>> The incorrectly nonexistent files stay not existing after a client remount: >>> >>> [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >>> [root@cephmds611 ~]# systemctl restart mnt-ceph.mount >>> [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >>> ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log': No such file or directory >>> >>> In the MDS, the problem is when the 'GSK' directory is looked up as part of the path lookup. >>> client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK >>> >>> It seems like the MDS is looking in its cache, and instead of the snapshot it finds an entry for the live tree where there is no GSK directory. >>> Then the MDS returns ENOENT for the GSK directory in the snapshot (!) - so anything under GSK will be invisible to the client in the snapshot: >>> >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 >>> 400 caller_uid=0, caller_gid=0{0,}) v6 >>> >>> Here are the full logs: >>> >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 4 mds.0.server handle_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0,} >>> ) v6 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.5902540 get_session have 0x556382924000 client.265674856 v1:10.250.129.151:0/2806743593 state open >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.server oldest_client_tid=7 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_start request(client.265674856:7 nref=2 cr=0x555b0018b500) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server dispatch_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0 >>> ,}) v6 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server rdlock_path_pin_ref request(client.265674856:7 nref=2 cr=0x555b0018b500) #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache traverse: opening base ino 0x100131197c4 snap head path depth 5 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 0 '' snapid head >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snapdir >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 1 'sync-20260701-0904' snapid snapdir >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) resolve_snapname 'sync-20260701-0904' in [0,head] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) ? snap(1c9 0x100131197c4 'sync-20260701-0904' 2026-07-01T09:04:25.746531-0400) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snap sync-20260701-0904 -> 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 2 'bturzo' snapid 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) lookup (bturzo, '1c9') >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) hit -> (bturzo,head) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 3 'Projects' snapid 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) lookup (Projects, '1c9') >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) hit -> (Projects,head) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 4 'GSK' snapid 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 >>> 400 caller_uid=0, caller_gid=0{0,}) v6 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server apply_allocated_inos 0x0 / [] / 0x0 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server lat 0.575392 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist snapid 1c9 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.ino(0x101a2629a23) encode_inodestat snapid 1c9 to old_inode [1a7,1d3] n(v1256297 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003833=931566+72267) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snaps 1c9,1ec,1ee,1f0,1f2,1f4 (seq 1c9 cached_seq 1f5) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snap_info snaps 1c9,1ec,1ee,1f0,1f2,1f4 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snaps 1ec,1ee,1f0,1f2,1f4 (seq 1f5 cached_seq 1f5) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snap_info snaps 1ec,1ee,1f0,1f2,1f4 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.ino(0x101a2629a23) pfile 0 pauth 0 plink 0 pxattr 0 plocal 0 mtime 2026-06-08T14:02:36.442123-0400 ctime 2026-06-08T14:02:36.442123-0400 change_attr 196 valid=1 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.ino(0x101a2629a23) encode_inodestat caps pAsLsXsFs seq 0 mseq 0 xattrv 1 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added diri [inode 0x101a2629a23 [...1ea,head] /users/bturzo/Projects/ auth v22346457 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+41) n(v1256393 rc2026-07-17T12:32:38. >>> 513951-0400 b70376541394148 1003833=931566+72267) old_inodes=2 (inest lock) (iversion lock) caps={254564708=pAsLsXsFs/-@9,258742051=pAsLsXsFs/pAsLsXsFsxcral@3} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=0 wait >>> er=0 authpin=0 0x556f6adfe580] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added dir [dir 0x101a2629a23 /users/bturzo/Projects/ [2,head] auth v=26589887 cv=26589887/26589887 REP state=1074003968 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+4 >>> 1) n(v1256393 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003832=931566+72266) hs=45+1,ss=0+0 | child=1 dirty=0 waiter=0 authpin=0 0x555a65593a80] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.locker issue_client_lease no/null lease on [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.5902540 send_message_client client.265674856 v1:10.250.129.151:0/2806743593 client_reply(???:7 = -2 (2) No such file or directory) v1 >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_finish request(client.265674856:7 nref=3 cr=0x555b0018b500) >>> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache request_cleanup request(client.265674856:7 nref=3 cr=0x555b0018b500) >>> >>> The file system I'm reading for the copying is running ceph Reef 18.2.8, so this is a Reef MDS, in case the version/patches matter. >>> Thanks very much for looking into this - the help is appreciated! >>> >>> Andras >>> >>> >>> >>> On 7/22/26 8:28 PM, Xiubo Li wrote: >>> >>> Hi Andras, >>> >>> Thanks for your quick feedback. >>> >>> Sure. Yesterday Dhairya and I found something important in both the >>> MDS and kclient sides. Without the logs we will review the code to >>> ensure our investigation covers this case. We will provide feedback >>> later. >>> >>> Regards >>> Xiubo >>> >>> On Thu, 23 Jul 2026 at 00:14, Andras Pataki via ceph-users >>> <ceph-users(a)ceph.io> wrote: >>> >>> Hi Xiubo and Dhariya, >>> >>> The way I was able to reproduce the problem was to look at the error log >>> of my copy processes and take those files and check on them on the >>> various clients. Today, unfortunately, all those files seem to be >>> visible - and at the moment I don't have a way of reproducing the issue >>> from scratch. I'm trying to see what could be done to set up the >>> reproduction from scratch. If I manage to do that, I'll try to gather >>> some information on whether this is a client issue (i.e. flushing the >>> client caches changes anything) or whether it comes from the MDS. >>> Unfortunately the MDS in question is a very busy one, our main file >>> system MDS, so there are limits to what I can do to it. >>> >>> Andras >>> >>> >>> On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote: >>> >>> Hi Dhairya, >>> >>> Yeah, makes sense. >>> >>> Since I couldn't reproduce this locally, @Andras Pataki Could you help >>> reproduce it by enabling the debug logs as Dhairya mentioned? And if >>> possible please also try with the fuse client to see could you see >>> too. >>> >>> Thanks >>> Xiubo >>> >>> On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev(a)ceph.io> wrote: >>> >>> Hi Xiubo, >>> >>> I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds… . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT? >>> >>> >>> On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li(a)clyso.com> wrote: >>> >>> Hi Andras, Dhairya, >>> >>> I recall hitting a similar issue with this and the workaround involves >>> running the 'ls' command, but I can't recall exactly which command or >>> the tracker reference. This happened years ago. >>> >>> This morning I tried to reproduce this locally but failed. And I just >>> went through the mds and kclient code just found two suspect places >>> could cause this: >>> >>> MDS side: in https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/… >>> >>> MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: >>> >>> if (curdir->is_complete() || // (1) >>> (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) >>> return -ENOENT; >>> >>> is_complete() means all live entries are cached in memory. Shortcut >>> (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), >>> but shortcut (1) applies to all snapids — including snapshots. >>> >>> When a snapshot exists but the file was later deleted from the live >>> tree, the MDS eventually evicts the deleted file's dentry from the >>> in-memory cache (a null dentry, which trim_dentry() removes without >>> clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits >>> is_complete()=true with the dentry absent (ENOENT), without ever >>> checking the OSD where the snapshot data actually lives. >>> >>> READDIR takes a different code path that fetches from the OSD, which >>> is why ls -l fixes it. >>> >>> Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: >>> if (snapid == CEPH_NOSNAP && >>> (curdir->is_complete() || ...bloom...)) { >>> >>> @Andras >>> >>> BTW, to verify the above could you try to reproduce this by using the >>> ceph-fuse mount ? Locally I couldn't reproduce it by trying many times >>> and I am planing to add some debug code in MDS to ammulated it to >>> reproduce it later. >>> >>> ==== >>> >>> Kclient side: https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-li… >>> Plus a kclient defense: ceph_d_revalidate() unconditionally trusts >>> negative dentries in snapshot directories, caching a transient ENOENT >>> forever. Change valid = 1 to valid = inode ? 1 : 0. >>> >>> I just created on trakcer to follow this: https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-a… >>> >>> Thanks >>> Xiubo >>> >>> >>> >>> >>> On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users >>> <ceph-users(a)ceph.io> wrote: >>> >>> Hi Andras, >>> >>> This sounds like the client served the lookups from its cached negative? >>> Can you check if you see the lookup calls being made to MDS and maybe set >>> debug_mds=20 to check if the client did send the request but it was MDS >>> which returned ENOENT but since the `ls -l` succeeded I'm wondering if this >>> is actually something from MDS. Maybe try a sync and drop_caches to >>> validate that? >>> >>> *Dhairya Parmar* >>> >>> Software Engineer, CephFS >>> >>> >>> On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < >>> ceph-users(a)ceph.io> wrote: >>> >>> Hi ceph users/devs, >>> >>> We've been encountering a strange issue with snapshots as part of a >>> project to copy our large ceph file system to a backup in parallel >>> across nodes. We are taking a snapshot, and then doing a copy of the >>> snapshot, which we expect to be static (unchanging) - using several >>> clients in parallel - all kernel mounts. Some clients are unable to >>> open files and directories that are known to exist (were discovered >>> traversing the file system on perhaps another node). >>> After some playing around, I could reproduce the issue on a single node >>> with just 'ls' as follows: >>> >>> # This file doesn't seems to exist - and some number of parent directories >>> are also "missing": [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >>> >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': >>> >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >>> ls: cannot access >>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': >>> No such file or directory [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >>> No such file or directory # Until we get to this 'Projects' parent >>> directory, which seems to be there [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 >>> 1961 1961 46 Jun 8 14:02 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it >>> is still not there ... [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >>> No such file or directory # But ... let's see if the "tickle" the Projects >>> directory a bit ... >>> [root@cephmds603 ~]# ls -l >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null >>> >>> # Now all subdirectories and the file in question immediately becomes >>> visible: >>> [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >>> drwxrwsr-x 5 1961 1961 3 May 7 17:18 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >>> [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >>> drwxrwsr-x 12 1961 1961 10 May 11 16:08 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >>> [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >>> drwxrwsr-x 5 1961 1961 3 May 11 16:18 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >>> [root@cephmds603 ~]# ls -ld >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >>> -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 >>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >>> >>> # This is the kernel client 6.18.32 (relatively recent LTS kernel) with >>> ceph squid 19.2.3: >>> [root@cephmds603 ~]# uname -a >>> Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 >>> x86_64 x86_64 GNU/Linux >>> [root@cephmds603 ~]# ceph --version >>> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid >>> (stable) >>> [root@cephmds603 ~]# >>> >>> What is most interesting is that in the live tree (non-snapshotted tree) - >>> the Projects directory exists, but everything under it has been removed! >>> >>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ >>> drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ >>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK >>> ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or >>> directory >>> >>> Am I running into some known issue? This looks to be specific to some >>> clients - i.e. some clients see the directory and some don't at the same >>> point in time, i.e. this is perhaps a kclient problem? >>> Any ideas/suggestions would be appreciated, >>> >>> Andras >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

subvolume in _nogroup
by Stéphane Delmotte 24 Jul '26

24 Jul '26

Hello, I find myself with a list of subvolumes in the|no_group|subvolume group, which, if I understand correctly, is a special subvolume group. Actually, Manila created all these subvolumes without specifying a subvolume group. Following an unexpected power outage and a restart, it no longer works. Manila can no longer create subvolumes, and I can no longer manipulate my subvolumes. For example: content_copyCopy code |cephfssubvolumermcephfsbidulestef ErrorEINVAL:invalidvaluespecifiedforceph.dir.subvolume| However, the state of my cluster is not critical. I only have: * 248 PGs not deep-scrubbed in time * 517 PGs not scrubbed in time My Ceph version is: content_copyCopy code |cephversion19.3.0-4637-g42c1c3d4(42c1c3d42aaa0b6c9bcbccb51c8b397fb980aa9f) squid (dev)| My CephFS mounts remain functional. If anyone has information about this, I’d appreciate it. Alternatively, how can I continue creating subvolumes without specifying a subvolume group? Thanks, Stéphane -- Delmotte Stéphane Laboratoire de Biométrie et Biologie Évolutive. UMR CNRS 5558 Campus de La Doua - Université Claude Bernard - Lyon 1 Bâtiment Grégoire Mendel - 1er étage 16 rue Raphael Dubois Villeurbanne 69100 Tel : +33 06 82 87 93 59

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Redouane Kachach 24 Jul '26

24 Jul '26

Just to add: although assigning the _admin label to many hosts may be convenient, it also increases the security exposure. Cephadm distributes the highly privileged client.admin keyring to every _admin host, so an attacker who compromises one of those hosts and gains access to the keyring could obtain full administrative control of the cluster and cause significant damage. I would therefore recommend limiting the _admin label to a small, carefully controlled and hardened set of hosts that are explicitly used for cluster administration. On Fri, Jul 24, 2026 at 1:58 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > > > > On Jul 24, 2026, at 7:23 AM, Michel Le Cocq via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > The point I wanted to explore is the following: > > > > We have three nodes for the MON MGR and MDS roles. > > > > To bootstrap the cluster, we installed cephadm via apt only on the first > > node (M1). > > > > Then, we copied the ceph.pub key to the second node (M2) and the third > > node (M3). > > > > After that, we ran the following commands on M1: > > > > ceph orch host add m2 <IP> > > ceph orch host add m3 <IP> > > > > Note: cephadm is not installed on M2 and M3. > > Eugen covered this, but I'll try to help: > > The copy of cephadm that you installed on M1 was used *only* to bootstrap > the cluster. > You don't use that copy ever again. > The bootstrap process and "ceph orch host add" install *another*, > different copy of cephadm in a different location. > You could "apt delete cephadm" on M1 and nothing would change. > > This admittedly is a bit confusing. > > > Question: > > > > What happens if M1 fails completely? > > Nothing. > > > > > > In this scenario, we would need the file: > > /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} > > to be able to launch the cephadm shell and recover the cluster. > > That is installed on every cluster node. But you don't strictly need the > cephadm shell. > > What you do want to do is > > ceph orch host label add m1 _admin > ceph orch host label add m2 _admin > ceph orch host label add m3_admin > > and ensure that your orch services are set up to let mon, mgr, prometheus, > grafana etc. run on all three nodes. > > Once you have these additional hosts added to the cluster, M2 and M3 are > no different from M1. > > And assuming that your pools in this scenario are replicated size=3, which > is default, if M1 goes up in smoke you don't need to recover the cluster > because the cluster and all services will continue serving data. Your > redundancy will be degraded until you get a third host back in, but the > cluster itself will be fine. > > >>> > >>> Is it a good idee to do on host : > >>> > >>> # apt purge cephadm > > > That doesn't hurt, but it isn't necessary unless you want to run the > cephadm shell. > > >>> # chmod +x /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} > >>> # ln -s /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} /root/cephadm > > Nononono. The orchestrator goes to great lengths to free you from > micromanagement. > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Anthony D'Atri 24 Jul '26

24 Jul '26

> On Jul 24, 2026, at 7:23 AM, Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> wrote: > > The point I wanted to explore is the following: > > We have three nodes for the MON MGR and MDS roles. > > To bootstrap the cluster, we installed cephadm via apt only on the first > node (M1). > > Then, we copied the ceph.pub key to the second node (M2) and the third > node (M3). > > After that, we ran the following commands on M1: > > ceph orch host add m2 <IP> > ceph orch host add m3 <IP> > > Note: cephadm is not installed on M2 and M3. Eugen covered this, but I'll try to help: The copy of cephadm that you installed on M1 was used *only* to bootstrap the cluster. You don't use that copy ever again. The bootstrap process and "ceph orch host add" install *another*, different copy of cephadm in a different location. You could "apt delete cephadm" on M1 and nothing would change. This admittedly is a bit confusing. > Question: > > What happens if M1 fails completely? Nothing. > > In this scenario, we would need the file: > /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} > to be able to launch the cephadm shell and recover the cluster. That is installed on every cluster node. But you don't strictly need the cephadm shell. What you do want to do is ceph orch host label add m1 _admin ceph orch host label add m2 _admin ceph orch host label add m3_admin and ensure that your orch services are set up to let mon, mgr, prometheus, grafana etc. run on all three nodes. Once you have these additional hosts added to the cluster, M2 and M3 are no different from M1. And assuming that your pools in this scenario are replicated size=3, which is default, if M1 goes up in smoke you don't need to recover the cluster because the cluster and all services will continue serving data. Your redundancy will be degraded until you get a third host back in, but the cluster itself will be fine. >>> >>> Is it a good idee to do on host : >>> >>> # apt purge cephadm That doesn't hurt, but it isn't necessary unless you want to run the cephadm shell. >>> # chmod +x /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} >>> # ln -s /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} /root/cephadm Nononono. The orchestrator goes to great lengths to free you from micromanagement.

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 24 Jul '26

24 Jul '26

The point I wanted to explore is the following: We have three nodes for the MON MGR and MDS roles. To bootstrap the cluster, we installed cephadm via apt only on the first node (M1). Then, we copied the ceph.pub key to the second node (M2) and the third node (M3). After that, we ran the following commands on M1: ceph orch host add m2 <IP> ceph orch host add m3 <IP> Note: cephadm is not installed on M2 and M3. Question: What happens if M1 fails completely? In this scenario, we would need the file: /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} to be able to launch the cephadm shell and recover the cluster. -- M Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > There’s no need to do that. Making it executable would be only required if you > wanted to use that binary manually. The difference between the package and > container version are negligible in this case. We still have cephadm version > 16.2.13 installed on the hosts while the cluster is running 19.2.3. > You’re fine with this minor difference between versions. > > Zitat von Michel Le Cocq <lecocq(a)ipgp.fr>: > >> Hi, thanks for your answer. >> >> I effectively have on my host : >> /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} >> >> But it's not executable ! >> I did a chmod +x ... >> >> Then I can see that debian pkg version and the one in /var/lib are not >> the same ... >> >> For Debian : >> cephadm version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef >> (stable) >> >> For Ceph in /var/lib ... >> cephadm version 18.2.8 (efac5a54607c13fa50d4822e50242b86e6e446df) reef >> (stable) >> >> Is it a good idee to do on host : >> >> # apt purge cephadm >> # chmod +x /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} >> # ln -s /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} /root/cephadm >> >> Thanks. >> >> -- >> Michel >> >> Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: >> >>> Hi, responses inline... >>> >>> >>> Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: >>> >>>> Actually, what confused me was this: >>>> >>>> cephadm bootstrap --mon-ip <my-ip> >>>> >>>> Among other things, this command: >>>> - Generates a new SSH key for the Ceph cluster and adds it >>>> to the /root/.ssh/authorized_keys file. >>>> - Saves a copy of the public key to the /etc/ceph/ceph.pub file. >>>> >>>> You then need to add a node. >>>> >>>> ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> >>>> >>>> ceph orch host add <newhost> [<ip>] [<label1> ...] >>>> >>>> I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! >>> >>> That's stored in the config-key db: >>> >>> ceph config-key get mgr/cephadm/ssh_identity_key >>> >>>> >>>> -- >>>> M >>>> >>>> Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: >>>> >>>>> Hi again. >>>>> >>>>> I'm answering my own question. >>>>> >>>>> It seems I read the wrong version of the doc >>>>> ... sic! >>>>> >>>>> It should be enough to set the “_admin” flag on all nodes: >>>>> >>>>> and that we also need to install cephadm on the nodes we >>>>> want to use for administration, right!? >>> >>> Not necessarily, but it can be more convenient for debugging. The >>> orchestrator >>> stores a cephadm binary on each node to be able to manage them, it is stored >>> in >>> /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}. That's used to remotely >>> orchestrate >>> the cluster/nodes etc. The original bootstrap node requires a cephadm binary >>> either installed as a package or downloaded via curl as described in the >>> docs to >>> be able to bootstrap. After that you'll only need to execute cephadm >>> commands >>> locally on the nodes if you're debugging issues etc., otherwise the >>> orchestrator >>> will only use the file in /var/lib/ceph. >>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 24 Jul '26

24 Jul '26

Hi, thanks for your answer. I effectively have on my host : /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} But it's not executable ! I did a chmod +x ... Then I can see that debian pkg version and the one in /var/lib are not the same ... For Debian : cephadm version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) For Ceph in /var/lib ... cephadm version 18.2.8 (efac5a54607c13fa50d4822e50242b86e6e446df) reef (stable) Is it a good idee to do on host : # apt purge cephadm # chmod +x /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} # ln -s /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} /root/cephadm Thanks. -- Michel Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > Hi, responses inline... > > > Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > >> Actually, what confused me was this: >> >> cephadm bootstrap --mon-ip <my-ip> >> >> Among other things, this command: >> - Generates a new SSH key for the Ceph cluster and adds it >> to the /root/.ssh/authorized_keys file. >> - Saves a copy of the public key to the /etc/ceph/ceph.pub file. >> >> You then need to add a node. >> >> ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> >> >> ceph orch host add <newhost> [<ip>] [<label1> ...] >> >> I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! > > That's stored in the config-key db: > > ceph config-key get mgr/cephadm/ssh_identity_key > >> >> -- >> M >> >> Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: >> >>> Hi again. >>> >>> I'm answering my own question. >>> >>> It seems I read the wrong version of the doc >>> ... sic! >>> >>> It should be enough to set the “_admin” flag on all nodes: >>> >>> and that we also need to install cephadm on the nodes we >>> want to use for administration, right!? > > Not necessarily, but it can be more convenient for debugging. The orchestrator > stores a cephadm binary on each node to be able to manage them, it is stored in > /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}. That's used to remotely orchestrate > the cluster/nodes etc. The original bootstrap node requires a cephadm binary > either installed as a package or downloaded via curl as described in the docs to > be able to bootstrap. After that you'll only need to execute cephadm commands > locally on the nodes if you're debugging issues etc., otherwise the orchestrator > will only use the file in /var/lib/ceph. > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

July Ceph Science User Group Virtual Meeting
by Thomas Byrne - STFC UKRI 24 Jul '26

24 Jul '26

Hi all, We will be having a Ceph science/research/big cluster call next Tuesday (July 28th). If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc. We usually start each meeting with a brief 10-15 minutes talk given by a community member, but currently don't have one scheduled for this meeting. If you are interested in signing up to give a short talk for a meeting please let me know! It is a very informal talk. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20260728 Virtual event details: July 28th, 2026 14:00 UTC See the Ceph Community Calendar<https://ceph.io/en/community/meetups/> for the series information. (please double check the meeting time for your timezone!) Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph YouTube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Thanks, Tom

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Xiubo Li 24 Jul '26

24 Jul '26

Hi Andras, Thanks very much for your logs. This is very helpful. Following is Dhairya's and my analysis: Your issue was most probably triggered by cases like: 1. COW deletes GSK: items: snap dentry [2, 0x1c9] primary <-- preserved for snapshot head dentry [0x1ca, CEPH_NOSNAP] NULL <-- COW correctly set first 2. Cache trims the COW head NULL dentry: null + clean --> clear_complete = FALSE --> STATE_COMPLETE preserved! <-- KEY! --> items now only has snap dentry 3. A HEAD lookup comes in for GSK: lookup("GSK", CEPH_NOSNAP) --> lower_bound(CEPH_NOSNAP, "GSK", h) --> no head-level dentry found --> returns NULL MISS path: is_complete() = TRUE <-- still complete! --> add_null_dentry("GSK") <-- default first=2! --> creates [2, CEPH_NOSNAP] NULL <------------------------------------ HERE ------------------ 4. Now items has: snap dentry [2, 0x1c9] primary NULL dentry [2, CEPH_NOSNAP] <-- first=2 covers ALL snapids 5. Cache trim --> trims the snap dentry: primary --> clear_complete = TRUE --> STATE_COMPLETE cleared. This could happen under workload pressure 6. Snap lookup comes in: dir incomplete, only NULL dentry [2,head] left 2 < 0x1c9 < CEPH_NOSNAP hit ENOENT ----------> in CDir: lookup() it will just return the NULL dentry, which is [2, CPEH_NOSNAP] and then the kclient cached it. So the root cause is in the MDS, and in kclient we still need to fix kclient to prevent caching NULL dentry for this bug. Dhairya or I will fix the MDS side bug later. I have sent out the kclient patches and will create a PR to fix the libcephfs later. Thanks & Regards Xiubo Xiubo On Fri, 24 Jul 2026 at 13:07, Dhairya Parmar via dev <dev(a)ceph.io> wrote: > > Hey Andas, sorry to stress your MDS a bit and thanks a ton for the debug logs. This is really helpful. This is the most important bit in [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] and my suspicion is on [2,head] range. Let's see what I find. I'll keep you posted. BTW @Xiubo Li mentioned he isn't able to reproduce this locally. Would you be able to share the reproducer if possible? > > On Thu, Jul 23, 2026 at 10:19 PM Andras Pataki <apataki(a)flatironinstitute.org> wrote: >> >> Hi Xiubo and Dhairya, >> >> I had a fruitful debugging session with my clients and MDS. The problem seems to be with the MDS, not with the client. >> The incorrectly nonexistent files stay not existing after a client remount: >> >> [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >> [root@cephmds611 ~]# systemctl restart mnt-ceph.mount >> [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log >> ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log': No such file or directory >> >> In the MDS, the problem is when the 'GSK' directory is looked up as part of the path lookup. >> client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK >> >> It seems like the MDS is looking in its cache, and instead of the snapshot it finds an entry for the live tree where there is no GSK directory. >> Then the MDS returns ENOENT for the GSK directory in the snapshot (!) - so anything under GSK will be invisible to the client in the snapshot: >> >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 >> 400 caller_uid=0, caller_gid=0{0,}) v6 >> >> Here are the full logs: >> >> 2026-07-23T10:34:53.281-0400 7ffff205a640 4 mds.0.server handle_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0,} >> ) v6 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.5902540 get_session have 0x556382924000 client.265674856 v1:10.250.129.151:0/2806743593 state open >> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.server oldest_client_tid=7 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_start request(client.265674856:7 nref=2 cr=0x555b0018b500) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server dispatch_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0 >> ,}) v6 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server rdlock_path_pin_ref request(client.265674856:7 nref=2 cr=0x555b0018b500) #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache traverse: opening base ino 0x100131197c4 snap head path depth 5 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 0 '' snapid head >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snapdir >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 1 'sync-20260701-0904' snapid snapdir >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) resolve_snapname 'sync-20260701-0904' in [0,head] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) ? snap(1c9 0x100131197c4 'sync-20260701-0904' 2026-07-01T09:04:25.746531-0400) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snap sync-20260701-0904 -> 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 2 'bturzo' snapid 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) lookup (bturzo, '1c9') >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) hit -> (bturzo,head) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 3 'Projects' snapid 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) lookup (Projects, '1c9') >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) hit -> (Projects,head) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 4 'GSK' snapid 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 >> 400 caller_uid=0, caller_gid=0{0,}) v6 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server apply_allocated_inos 0x0 / [] / 0x0 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server lat 0.575392 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist snapid 1c9 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.ino(0x101a2629a23) encode_inodestat snapid 1c9 to old_inode [1a7,1d3] n(v1256297 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003833=931566+72267) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snaps 1c9,1ec,1ee,1f0,1f2,1f4 (seq 1c9 cached_seq 1f5) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snap_info snaps 1c9,1ec,1ee,1f0,1f2,1f4 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snaps 1ec,1ee,1f0,1f2,1f4 (seq 1f5 cached_seq 1f5) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snap_info snaps 1ec,1ee,1f0,1f2,1f4 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.ino(0x101a2629a23) pfile 0 pauth 0 plink 0 pxattr 0 plocal 0 mtime 2026-06-08T14:02:36.442123-0400 ctime 2026-06-08T14:02:36.442123-0400 change_attr 196 valid=1 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.ino(0x101a2629a23) encode_inodestat caps pAsLsXsFs seq 0 mseq 0 xattrv 1 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added diri [inode 0x101a2629a23 [...1ea,head] /users/bturzo/Projects/ auth v22346457 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+41) n(v1256393 rc2026-07-17T12:32:38. >> 513951-0400 b70376541394148 1003833=931566+72267) old_inodes=2 (inest lock) (iversion lock) caps={254564708=pAsLsXsFs/-@9,258742051=pAsLsXsFs/pAsLsXsFsxcral@3} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=0 wait >> er=0 authpin=0 0x556f6adfe580] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added dir [dir 0x101a2629a23 /users/bturzo/Projects/ [2,head] auth v=26589887 cv=26589887/26589887 REP state=1074003968 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+4 >> 1) n(v1256393 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003832=931566+72266) hs=45+1,ss=0+0 | child=1 dirty=0 waiter=0 authpin=0 0x555a65593a80] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.locker issue_client_lease no/null lease on [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] >> 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.5902540 send_message_client client.265674856 v1:10.250.129.151:0/2806743593 client_reply(???:7 = -2 (2) No such file or directory) v1 >> 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_finish request(client.265674856:7 nref=3 cr=0x555b0018b500) >> 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache request_cleanup request(client.265674856:7 nref=3 cr=0x555b0018b500) >> >> The file system I'm reading for the copying is running ceph Reef 18.2.8, so this is a Reef MDS, in case the version/patches matter. >> Thanks very much for looking into this - the help is appreciated! >> >> Andras >> >> >> >> On 7/22/26 8:28 PM, Xiubo Li wrote: >> >> Hi Andras, >> >> Thanks for your quick feedback. >> >> Sure. Yesterday Dhairya and I found something important in both the >> MDS and kclient sides. Without the logs we will review the code to >> ensure our investigation covers this case. We will provide feedback >> later. >> >> Regards >> Xiubo >> >> On Thu, 23 Jul 2026 at 00:14, Andras Pataki via ceph-users >> <ceph-users(a)ceph.io> wrote: >> >> Hi Xiubo and Dhariya, >> >> The way I was able to reproduce the problem was to look at the error log >> of my copy processes and take those files and check on them on the >> various clients. Today, unfortunately, all those files seem to be >> visible - and at the moment I don't have a way of reproducing the issue >> from scratch. I'm trying to see what could be done to set up the >> reproduction from scratch. If I manage to do that, I'll try to gather >> some information on whether this is a client issue (i.e. flushing the >> client caches changes anything) or whether it comes from the MDS. >> Unfortunately the MDS in question is a very busy one, our main file >> system MDS, so there are limits to what I can do to it. >> >> Andras >> >> >> On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote: >> >> Hi Dhairya, >> >> Yeah, makes sense. >> >> Since I couldn't reproduce this locally, @Andras Pataki Could you help >> reproduce it by enabling the debug logs as Dhairya mentioned? And if >> possible please also try with the fuse client to see could you see >> too. >> >> Thanks >> Xiubo >> >> On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev(a)ceph.io> wrote: >> >> Hi Xiubo, >> >> I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds… . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT? >> >> >> On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li(a)clyso.com> wrote: >> >> Hi Andras, Dhairya, >> >> I recall hitting a similar issue with this and the workaround involves >> running the 'ls' command, but I can't recall exactly which command or >> the tracker reference. This happened years ago. >> >> This morning I tried to reproduce this locally but failed. And I just >> went through the mds and kclient code just found two suspect places >> could cause this: >> >> MDS side: in https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/… >> >> MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: >> >> if (curdir->is_complete() || // (1) >> (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) >> return -ENOENT; >> >> is_complete() means all live entries are cached in memory. Shortcut >> (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), >> but shortcut (1) applies to all snapids — including snapshots. >> >> When a snapshot exists but the file was later deleted from the live >> tree, the MDS eventually evicts the deleted file's dentry from the >> in-memory cache (a null dentry, which trim_dentry() removes without >> clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits >> is_complete()=true with the dentry absent (ENOENT), without ever >> checking the OSD where the snapshot data actually lives. >> >> READDIR takes a different code path that fetches from the OSD, which >> is why ls -l fixes it. >> >> Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: >> if (snapid == CEPH_NOSNAP && >> (curdir->is_complete() || ...bloom...)) { >> >> @Andras >> >> BTW, to verify the above could you try to reproduce this by using the >> ceph-fuse mount ? Locally I couldn't reproduce it by trying many times >> and I am planing to add some debug code in MDS to ammulated it to >> reproduce it later. >> >> ==== >> >> Kclient side: https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-li… >> Plus a kclient defense: ceph_d_revalidate() unconditionally trusts >> negative dentries in snapshot directories, caching a transient ENOENT >> forever. Change valid = 1 to valid = inode ? 1 : 0. >> >> I just created on trakcer to follow this: https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-a… >> >> Thanks >> Xiubo >> >> >> >> >> On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users >> <ceph-users(a)ceph.io> wrote: >> >> Hi Andras, >> >> This sounds like the client served the lookups from its cached negative? >> Can you check if you see the lookup calls being made to MDS and maybe set >> debug_mds=20 to check if the client did send the request but it was MDS >> which returned ENOENT but since the `ls -l` succeeded I'm wondering if this >> is actually something from MDS. Maybe try a sync and drop_caches to >> validate that? >> >> *Dhairya Parmar* >> >> Software Engineer, CephFS >> >> >> On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < >> ceph-users(a)ceph.io> wrote: >> >> Hi ceph users/devs, >> >> We've been encountering a strange issue with snapshots as part of a >> project to copy our large ceph file system to a backup in parallel >> across nodes. We are taking a snapshot, and then doing a copy of the >> snapshot, which we expect to be static (unchanging) - using several >> clients in parallel - all kernel mounts. Some clients are unable to >> open files and directories that are known to exist (were discovered >> traversing the file system on perhaps another node). >> After some playing around, I could reproduce the issue on a single node >> with just 'ls' as follows: >> >> # This file doesn't seems to exist - and some number of parent directories >> are also "missing": [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': >> >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> ls: cannot access >> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': >> No such file or directory [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >> No such file or directory # Until we get to this 'Projects' parent >> directory, which seems to be there [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 >> 1961 1961 46 Jun 8 14:02 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it >> is still not there ... [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >> No such file or directory # But ... let's see if the "tickle" the Projects >> directory a bit ... >> [root@cephmds603 ~]# ls -l >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null >> >> # Now all subdirectories and the file in question immediately becomes >> visible: >> [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >> drwxrwsr-x 5 1961 1961 3 May 7 17:18 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >> [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> drwxrwsr-x 12 1961 1961 10 May 11 16:08 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> drwxrwsr-x 5 1961 1961 3 May 11 16:18 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> [root@cephmds603 ~]# ls -ld >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 >> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> >> # This is the kernel client 6.18.32 (relatively recent LTS kernel) with >> ceph squid 19.2.3: >> [root@cephmds603 ~]# uname -a >> Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 >> x86_64 x86_64 GNU/Linux >> [root@cephmds603 ~]# ceph --version >> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid >> (stable) >> [root@cephmds603 ~]# >> >> What is most interesting is that in the live tree (non-snapshotted tree) - >> the Projects directory exists, but everything under it has been removed! >> >> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ >> drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ >> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK >> ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or >> directory >> >> Am I running into some known issue? This looks to be specific to some >> clients - i.e. some clients see the directory and some don't at the same >> point in time, i.e. this is perhaps a kclient problem? >> Any ideas/suggestions would be appreciated, >> >> Andras >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >>

1 0

Re: Ubuntu 24.04 with Tentacle
by Redouane Kachach 24 Jul '26

24 Jul '26

Regarding using packages vs cephadm I'd just like to add that cephadm + Podman has supported critical loads in production for several years now. Operations such as upgrading to a new version are generally much easier when using cephadm. Best, Redo. On Fri, Jul 24, 2026 at 3:16 AM Szabo, Istvan (Agoda) via ceph-users < ceph-users(a)ceph.io> wrote: > Thank you very much, this makes me thinking again should I use cephadm or > should we still stay with the manual setup and package based update for a > heavy loaded critical block storage cluster :/ > > Get Outlook for Android<https://aka.ms/AAb9ysg> > ________________________________ > From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> > Sent: Thursday, 23 July 2026 18:33:59 > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Ubuntu 24.04 with Tentacle > > Email received from the internet. If in doubt, don't click any link nor > open any attachment ! > ________________________________ > > On 2026-07-23 07:06, Szabo, Istvan (Agoda) via ceph-users wrote: > > > Reading throught the Ceph documentation regarding different versions and > I see here > https://docs.ceph.com/en/latest/start/os-recommendations/#platforms and > "Upcoming" text. > > What does this mean exactly, which release is going to be "A" on Ubuntu > 24.04? > > Hi Istvan, > > Not really the answer you have asked for because I am not a Ceph > developer, we have however got some experience with running Tentacle on > Noble so perhaps it will come in handy to you. > > We have in recent months upgraded all of our Ceph servers, in both > production and development clusters, from Jammy to Noble. This has been > originally for two reasons: > > - for several reasons I shan't bore you with we currently use Podman > rather than Docker with cephadm, and it transpired that the Jammy > version of Podman does not support namespace settings required by > multi-container services like the cephadm-managed SMB gateways; > > - since the upgrade from Squid to Tentacle one of our cluster kept > seeing CPU lock-ups (and subsequent "slow BlueStore operations" > warnings) progressively appear for more and more OSDs, and having tried > to no avail a lot of different configuration tweaks we got to the point > where the remaining choices were "upgrade to a newer kernel" and "start > replacing HBAs". > > So far - touch wood! - all the upgraded systems have been operating > perfectly as far as Ceph is concerned. > > -- > MS > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ________________________________ > This message is confidential and is for the sole use of the intended > recipient(s). It may also be privileged or otherwise protected by copyright > or other legal rules. If you have received it by mistake please let us know > by reply email and delete it from your system. It is prohibited to copy > this message or disclose its content to anyone. Any confidentiality or > privilege is not waived or lost by any mistaken delivery or unauthorized > disclosure of the message. All messages sent to and from Agoda may be > monitored to ensure compliance with company policies, to protect the > company's interests and to remove potential malware. Electronic messages > may be intercepted, amended, lost or deleted, or contain viruses. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Xiubo Li 24 Jul '26

24 Jul '26

Hi Andras, Thanks for your quick feedback. Sure. Yesterday Dhairya and I found something important in both the MDS and kclient sides. Without the logs we will review the code to ensure our investigation covers this case. We will provide feedback later. Regards Xiubo On Thu, 23 Jul 2026 at 00:14, Andras Pataki via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Xiubo and Dhariya, > > The way I was able to reproduce the problem was to look at the error log > of my copy processes and take those files and check on them on the > various clients. Today, unfortunately, all those files seem to be > visible - and at the moment I don't have a way of reproducing the issue > from scratch. I'm trying to see what could be done to set up the > reproduction from scratch. If I manage to do that, I'll try to gather > some information on whether this is a client issue (i.e. flushing the > client caches changes anything) or whether it comes from the MDS. > Unfortunately the MDS in question is a very busy one, our main file > system MDS, so there are limits to what I can do to it. > > Andras > > > On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote: > > Hi Dhairya, > > > > Yeah, makes sense. > > > > Since I couldn't reproduce this locally, @Andras Pataki Could you help > > reproduce it by enabling the debug logs as Dhairya mentioned? And if > > possible please also try with the fuse client to see could you see > > too. > > > > Thanks > > Xiubo > > > > On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev(a)ceph.io> wrote: > >> Hi Xiubo, > >> > >> I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds… . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT? > >> > >> > >> On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li(a)clyso.com> wrote: > >>> Hi Andras, Dhairya, > >>> > >>> I recall hitting a similar issue with this and the workaround involves > >>> running the 'ls' command, but I can't recall exactly which command or > >>> the tracker reference. This happened years ago. > >>> > >>> This morning I tried to reproduce this locally but failed. And I just > >>> went through the mds and kclient code just found two suspect places > >>> could cause this: > >>> > >>> MDS side: in https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/… > >>> > >>> MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: > >>> > >>> if (curdir->is_complete() || // (1) > >>> (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) > >>> return -ENOENT; > >>> > >>> is_complete() means all live entries are cached in memory. Shortcut > >>> (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), > >>> but shortcut (1) applies to all snapids — including snapshots. > >>> > >>> When a snapshot exists but the file was later deleted from the live > >>> tree, the MDS eventually evicts the deleted file's dentry from the > >>> in-memory cache (a null dentry, which trim_dentry() removes without > >>> clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits > >>> is_complete()=true with the dentry absent (ENOENT), without ever > >>> checking the OSD where the snapshot data actually lives. > >>> > >>> READDIR takes a different code path that fetches from the OSD, which > >>> is why ls -l fixes it. > >>> > >>> Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: > >>> if (snapid == CEPH_NOSNAP && > >>> (curdir->is_complete() || ...bloom...)) { > >>> > >>> @Andras > >>> > >>> BTW, to verify the above could you try to reproduce this by using the > >>> ceph-fuse mount ? Locally I couldn't reproduce it by trying many times > >>> and I am planing to add some debug code in MDS to ammulated it to > >>> reproduce it later. > >>> > >>> ==== > >>> > >>> Kclient side: https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-li… > >>> Plus a kclient defense: ceph_d_revalidate() unconditionally trusts > >>> negative dentries in snapshot directories, caching a transient ENOENT > >>> forever. Change valid = 1 to valid = inode ? 1 : 0. > >>> > >>> I just created on trakcer to follow this: https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-a… > >>> > >>> Thanks > >>> Xiubo > >>> > >>> > >>> > >>> > >>> On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users > >>> <ceph-users(a)ceph.io> wrote: > >>>> Hi Andras, > >>>> > >>>> This sounds like the client served the lookups from its cached negative? > >>>> Can you check if you see the lookup calls being made to MDS and maybe set > >>>> debug_mds=20 to check if the client did send the request but it was MDS > >>>> which returned ENOENT but since the `ls -l` succeeded I'm wondering if this > >>>> is actually something from MDS. Maybe try a sync and drop_caches to > >>>> validate that? > >>>> > >>>> *Dhairya Parmar* > >>>> > >>>> Software Engineer, CephFS > >>>> > >>>> > >>>> On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < > >>>> ceph-users(a)ceph.io> wrote: > >>>> > >>>>> Hi ceph users/devs, > >>>>> > >>>>> We've been encountering a strange issue with snapshots as part of a > >>>>> project to copy our large ceph file system to a backup in parallel > >>>>> across nodes. We are taking a snapshot, and then doing a copy of the > >>>>> snapshot, which we expect to be static (unchanging) - using several > >>>>> clients in parallel - all kernel mounts. Some clients are unable to > >>>>> open files and directories that are known to exist (were discovered > >>>>> traversing the file system on perhaps another node). > >>>>> After some playing around, I could reproduce the issue on a single node > >>>>> with just 'ls' as follows: > >>>>> > >>>>> # This file doesn't seems to exist - and some number of parent directories > >>>>> are also "missing": [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > >>>>> > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': > >>>>> > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > >>>>> ls: cannot access > >>>>> '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': > >>>>> No such file or directory [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > >>>>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > >>>>> No such file or directory # Until we get to this 'Projects' parent > >>>>> directory, which seems to be there [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 > >>>>> 1961 1961 46 Jun 8 14:02 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it > >>>>> is still not there ... [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > >>>>> access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > >>>>> No such file or directory # But ... let's see if the "tickle" the Projects > >>>>> directory a bit ... > >>>>> [root@cephmds603 ~]# ls -l > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null > >>>>> > >>>>> # Now all subdirectories and the file in question immediately becomes > >>>>> visible: > >>>>> [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > >>>>> drwxrwsr-x 5 1961 1961 3 May 7 17:18 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > >>>>> [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > >>>>> drwxrwsr-x 12 1961 1961 10 May 11 16:08 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > >>>>> [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > >>>>> drwxrwsr-x 5 1961 1961 3 May 11 16:18 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > >>>>> [root@cephmds603 ~]# ls -ld > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > >>>>> -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 > >>>>> /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > >>>>> > >>>>> # This is the kernel client 6.18.32 (relatively recent LTS kernel) with > >>>>> ceph squid 19.2.3: > >>>>> [root@cephmds603 ~]# uname -a > >>>>> Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 > >>>>> x86_64 x86_64 GNU/Linux > >>>>> [root@cephmds603 ~]# ceph --version > >>>>> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > >>>>> (stable) > >>>>> [root@cephmds603 ~]# > >>>>> > >>>>> What is most interesting is that in the live tree (non-snapshotted tree) - > >>>>> the Projects directory exists, but everything under it has been removed! > >>>>> > >>>>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ > >>>>> drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ > >>>>> [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK > >>>>> ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or > >>>>> directory > >>>>> > >>>>> Am I running into some known issue? This looks to be specific to some > >>>>> clients - i.e. some clients see the directory and some don't at the same > >>>>> point in time, i.e. this is perhaps a kclient problem? > >>>>> Any ideas/suggestions would be appreciated, > >>>>> > >>>>> Andras > >>>>> _______________________________________________ > >>>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>>>> > >>>>> > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2