Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
Hi Andras, Thanks for your quick feedback. Sure. Yesterday Dhairya and I found something important in both the MDS and kclient sides. Without the logs we will review the code to ensure our investigation covers this case. We will provide feedback later. Regards Xiubo On Thu, 23 Jul 2026 at 00:14, Andras Pataki via ceph-users <ceph-users@ceph.io> wrote:
Hi Xiubo and Dhariya,
The way I was able to reproduce the problem was to look at the error log of my copy processes and take those files and check on them on the various clients. Today, unfortunately, all those files seem to be visible - and at the moment I don't have a way of reproducing the issue from scratch. I'm trying to see what could be done to set up the reproduction from scratch. If I manage to do that, I'll try to gather some information on whether this is a client issue (i.e. flushing the client caches changes anything) or whether it comes from the MDS. Unfortunately the MDS in question is a very busy one, our main file system MDS, so there are limits to what I can do to it.
Andras
On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote:
Hi Dhairya,
Yeah, makes sense.
Since I couldn't reproduce this locally, @Andras Pataki Could you help reproduce it by enabling the debug logs as Dhairya mentioned? And if possible please also try with the fuse client to see could you see too.
Thanks Xiubo
On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev@ceph.io> wrote:
Hi Xiubo,
I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds/... . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT?
On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li@clyso.com> wrote:
Hi Andras, Dhairya,
I recall hitting a similar issue with this and the workaround involves running the 'ls' command, but I can't recall exactly which command or the tracker reference. This happened years ago.
This morning I tried to reproduce this locally but failed. And I just went through the mds and kclient code just found two suspect places could cause this:
MDS side: in https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/M...
MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist:
if (curdir->is_complete() || // (1) (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) return -ENOENT;
is_complete() means all live entries are cached in memory. Shortcut (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), but shortcut (1) applies to all snapids — including snapshots.
When a snapshot exists but the file was later deleted from the live tree, the MDS eventually evicts the deleted file's dentry from the in-memory cache (a null dentry, which trim_dentry() removes without clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits is_complete()=true with the dentry absent (ENOENT), without ever checking the OSD where the snapshot data actually lives.
READDIR takes a different code path that fetches from the OSD, which is why ls -l fixes it.
Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: if (snapid == CEPH_NOSNAP && (curdir->is_complete() || ...bloom...)) {
@Andras
BTW, to verify the above could you try to reproduce this by using the ceph-fuse mount ? Locally I couldn't reproduce it by trying many times and I am planing to add some debug code in MDS to ammulated it to reproduce it later.
====
Kclient side: https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-lin... Plus a kclient defense: ceph_d_revalidate() unconditionally trusts negative dentries in snapshot directories, caching a transient ENOENT forever. Change valid = 1 to valid = inode ? 1 : 0.
I just created on trakcer to follow this: https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-az...
Thanks Xiubo
On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users <ceph-users@ceph.io> wrote:
Hi Andras,
This sounds like the client served the lookups from its cached negative? Can you check if you see the lookup calls being made to MDS and maybe set debug_mds=20 to check if the client did send the request but it was MDS which returned ENOENT but since the `ls -l` succeeded I'm wondering if this is actually something from MDS. Maybe try a sync and drop_caches to validate that?
*Dhairya Parmar*
Software Engineer, CephFS
On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < ceph-users@ceph.io> wrote:
Hi ceph users/devs,
We've been encountering a strange issue with snapshots as part of a project to copy our large ceph file system to a backup in parallel across nodes. We are taking a snapshot, and then doing a copy of the snapshot, which we expect to be static (unchanging) - using several clients in parallel - all kernel mounts. Some clients are unable to open files and directories that are known to exist (were discovered traversing the file system on perhaps another node). After some playing around, I could reproduce the issue on a single node with just 'ls' as follows:
# This file doesn't seems to exist - and some number of parent directories are also "missing": [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': No such file or directory # Until we get to this 'Projects' parent directory, which seems to be there [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 1961 1961 46 Jun 8 14:02 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it is still not there ... [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': No such file or directory # But ... let's see if the "tickle" the Projects directory a bit ... [root@cephmds603 ~]# ls -l /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null
# Now all subdirectories and the file in question immediately becomes visible: [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK drwxrwsr-x 5 1961 1961 3 May 7 17:18 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM drwxrwsr-x 12 1961 1961 10 May 11 16:08 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen drwxrwsr-x 5 1961 1961 3 May 11 16:18 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
# This is the kernel client 6.18.32 (relatively recent LTS kernel) with ceph squid 19.2.3: [root@cephmds603 ~]# uname -a Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 x86_64 x86_64 GNU/Linux [root@cephmds603 ~]# ceph --version ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@cephmds603 ~]#
What is most interesting is that in the live tree (non-snapshotted tree) - the Projects directory exists, but everything under it has been removed!
[root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or directory
Am I running into some known issue? This looks to be specific to some clients - i.e. some clients see the directory and some don't at the same point in time, i.e. this is perhaps a kclient problem? Any ideas/suggestions would be appreciated,
Andras _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Xiubo and Dhairya, I had a fruitful debugging session with my clients and MDS. The problem seems to be with the MDS, not with the client. The incorrectly nonexistent files stay not existing after a client remount: [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log [root@cephmds611 ~]# systemctl restart mnt-ceph.mount [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log': No such file or directory In the MDS, the problem is when the 'GSK' directory is looked up as part of the path lookup. client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK It seems like the MDS is looking in its cache, and instead of the snapshot it finds an entry for the live tree where there is no GSK directory. Then the MDS returns ENOENT for the GSK directory in the snapshot (!) - so anything under GSK will be invisible to the client in the snapshot: 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 400 caller_uid=0, caller_gid=0{0,}) v6 Here are the full logs: 2026-07-23T10:34:53.281-0400 7ffff205a640 4 mds.0.server handle_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0,} ) v6 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.5902540 get_session have 0x556382924000 client.265674856 v1:10.250.129.151:0/2806743593 state open 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.server oldest_client_tid=7 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_start request(client.265674856:7 nref=2 cr=0x555b0018b500) 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server dispatch_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0 ,}) v6 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server rdlock_path_pin_ref request(client.265674856:7 nref=2 cr=0x555b0018b500) #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache traverse: opening base ino 0x100131197c4 snap head path depth 5 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 0 '' snapid head 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snapdir 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 1 'sync-20260701-0904' snapid snapdir 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) resolve_snapname 'sync-20260701-0904' in [0,head] 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) ? snap(1c9 0x100131197c4 'sync-20260701-0904' 2026-07-01T09:04:25.746531-0400) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snap sync-20260701-0904 -> 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 2 'bturzo' snapid 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) lookup (bturzo, '1c9') 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) hit -> (bturzo,head) 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 3 'Projects' snapid 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) lookup (Projects, '1c9') 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) hit -> (Projects,head) 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 4 'GSK' snapid 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 400 caller_uid=0, caller_gid=0{0,}) v6 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server apply_allocated_inos 0x0 / [] / 0x0 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server lat 0.575392 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist snapid 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.ino(0x101a2629a23) encode_inodestat snapid 1c9 to old_inode [1a7,1d3] n(v1256297 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003833=931566+72267) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snaps 1c9,1ec,1ee,1f0,1f2,1f4 (seq 1c9 cached_seq 1f5) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snap_info snaps 1c9,1ec,1ee,1f0,1f2,1f4 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snaps 1ec,1ee,1f0,1f2,1f4 (seq 1f5 cached_seq 1f5) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snap_info snaps 1ec,1ee,1f0,1f2,1f4 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.ino(0x101a2629a23) pfile 0 pauth 0 plink 0 pxattr 0 plocal 0 mtime 2026-06-08T14:02:36.442123-0400 ctime 2026-06-08T14:02:36.442123-0400 change_attr 196 valid=1 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.ino(0x101a2629a23) encode_inodestat caps pAsLsXsFs seq 0 mseq 0 xattrv 1 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added diri [inode 0x101a2629a23 [...1ea,head] /users/bturzo/Projects/ auth v22346457 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+41) n(v1256393 rc2026-07-17T12:32:38. 513951-0400 b70376541394148 1003833=931566+72267) old_inodes=2 (inest lock) (iversion lock) caps={254564708=pAsLsXsFs/-@9,258742051=pAsLsXsFs/pAsLsXsFsxcral@3} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=0 wait er=0 authpin=0 0x556f6adfe580] 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added dir [dir 0x101a2629a23 /users/bturzo/Projects/ [2,head] auth v=26589887 cv=26589887/26589887 REP state=1074003968 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+4 1) n(v1256393 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003832=931566+72266) hs=45+1,ss=0+0 | child=1 dirty=0 waiter=0 authpin=0 0x555a65593a80] 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.locker issue_client_lease no/null lease on [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.5902540 send_message_client client.265674856 v1:10.250.129.151:0/2806743593 client_reply(???:7 = -2 (2) No such file or directory) v1 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_finish request(client.265674856:7 nref=3 cr=0x555b0018b500) 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache request_cleanup request(client.265674856:7 nref=3 cr=0x555b0018b500) The file system I'm reading for the copying is running ceph Reef 18.2.8, so this is a Reef MDS, in case the version/patches matter. Thanks very much for looking into this - the help is appreciated! Andras On 7/22/26 8:28 PM, Xiubo Li wrote:
Hi Andras,
Thanks for your quick feedback.
Sure. Yesterday Dhairya and I found something important in both the MDS and kclient sides. Without the logs we will review the code to ensure our investigation covers this case. We will provide feedback later.
Regards Xiubo
On Thu, 23 Jul 2026 at 00:14, Andras Pataki via ceph-users <ceph-users@ceph.io> wrote:
Hi Xiubo and Dhariya,
The way I was able to reproduce the problem was to look at the error log of my copy processes and take those files and check on them on the various clients. Today, unfortunately, all those files seem to be visible - and at the moment I don't have a way of reproducing the issue from scratch. I'm trying to see what could be done to set up the reproduction from scratch. If I manage to do that, I'll try to gather some information on whether this is a client issue (i.e. flushing the client caches changes anything) or whether it comes from the MDS. Unfortunately the MDS in question is a very busy one, our main file system MDS, so there are limits to what I can do to it.
Andras
On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote:
Hi Dhairya,
Yeah, makes sense.
Since I couldn't reproduce this locally, @Andras Pataki Could you help reproduce it by enabling the debug logs as Dhairya mentioned? And if possible please also try with the fuse client to see could you see too.
Thanks Xiubo
On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev<dev@ceph.io> wrote:
Hi Xiubo,
I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this blockhttps://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds/... . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT?
On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li<xiubo.li@clyso.com> wrote:
Hi Andras, Dhairya,
I recall hitting a similar issue with this and the workaround involves running the 'ls' command, but I can't recall exactly which command or the tracker reference. This happened years ago.
This morning I tried to reproduce this locally but failed. And I just went through the mds and kclient code just found two suspect places could cause this:
MDS side: inhttps://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/M...
MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist:
if (curdir->is_complete() || // (1) (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) return -ENOENT;
is_complete() means all live entries are cached in memory. Shortcut (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), but shortcut (1) applies to all snapids — including snapshots.
When a snapshot exists but the file was later deleted from the live tree, the MDS eventually evicts the deleted file's dentry from the in-memory cache (a null dentry, which trim_dentry() removes without clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits is_complete()=true with the dentry absent (ENOENT), without ever checking the OSD where the snapshot data actually lives.
READDIR takes a different code path that fetches from the OSD, which is why ls -l fixes it.
Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: if (snapid == CEPH_NOSNAP && (curdir->is_complete() || ...bloom...)) {
@Andras
BTW, to verify the above could you try to reproduce this by using the ceph-fuse mount ? Locally I couldn't reproduce it by trying many times and I am planing to add some debug code in MDS to ammulated it to reproduce it later.
====
Kclient side:https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-lin... Plus a kclient defense: ceph_d_revalidate() unconditionally trusts negative dentries in snapshot directories, caching a transient ENOENT forever. Change valid = 1 to valid = inode ? 1 : 0.
I just created on trakcer to follow this:https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-az...
Thanks Xiubo
On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users <ceph-users@ceph.io> wrote:
Hi Andras,
This sounds like the client served the lookups from its cached negative? Can you check if you see the lookup calls being made to MDS and maybe set debug_mds=20 to check if the client did send the request but it was MDS which returned ENOENT but since the `ls -l` succeeded I'm wondering if this is actually something from MDS. Maybe try a sync and drop_caches to validate that?
*Dhairya Parmar*
Software Engineer, CephFS
On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users< ceph-users@ceph.io> wrote:
> Hi ceph users/devs, > > We've been encountering a strange issue with snapshots as part of a > project to copy our large ceph file system to a backup in parallel > across nodes. We are taking a snapshot, and then doing a copy of the > snapshot, which we expect to be static (unchanging) - using several > clients in parallel - all kernel mounts. Some clients are unable to > open files and directories that are known to exist (were discovered > traversing the file system on perhaps another node). > After some playing around, I could reproduce the issue on a single node > with just 'ls' as follows: > > # This file doesn't seems to exist - and some number of parent directories > are also "missing": [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: > cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > No such file or directory # Until we get to this 'Projects' > parent directory, which seems to be there [root@cephmds603 ~]# > ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > drwxr-sr-x 45 1961 1961 46 Jun 8 14:02 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK > under it is still not there ... [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: > cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > No such file or directory # But ... let's see if the "tickle" the Projects > directory a bit ... > [root@cephmds603 ~]# ls -l > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null > > # Now all subdirectories and the file in question immediately becomes > visible: > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > drwxrwsr-x 5 1961 1961 3 May 7 17:18 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > drwxrwsr-x 12 1961 1961 10 May 11 16:08 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > drwxrwsr-x 5 1961 1961 3 May 11 16:18 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > # This is the kernel client 6.18.32 (relatively recent LTS kernel) with > ceph squid 19.2.3: > [root@cephmds603 ~]# uname -a > Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 > x86_64 x86_64 GNU/Linux > [root@cephmds603 ~]# ceph --version > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > (stable) > [root@cephmds603 ~]# > > What is most interesting is that in the live tree (non-snapshotted tree) - > the Projects directory exists, but everything under it has been removed! > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ > drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK > ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or > directory > > Am I running into some known issue? This looks to be specific to some > clients - i.e. some clients see the directory and some don't at the same > point in time, i.e. this is perhaps a kclient problem? > Any ideas/suggestions would be appreciated, > > Andras > _______________________________________________ > ceph-users mailing list --ceph-users@ceph.io > To unsubscribe send an email toceph-users-leave@ceph.io > > _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Hey Andas, sorry to stress your MDS a bit and thanks a ton for the debug logs. This is really helpful. This is the most important bit in [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] and my suspicion is on [2,head] range. Let's see what I find. I'll keep you posted. BTW @Xiubo Li <xiubo.li@clyso.com> mentioned he isn't able to reproduce this locally. Would you be able to share the reproducer if possible? On Thu, Jul 23, 2026 at 10:19 PM Andras Pataki < apataki@flatironinstitute.org> wrote:
Hi Xiubo and Dhairya,
I had a fruitful debugging session with my clients and MDS. The problem seems to be with the MDS, not with the client. The incorrectly nonexistent files stay not existing after a client remount:
[root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log [root@cephmds611 ~]# systemctl restart mnt-ceph.mount [root@cephmds611 ~]# ls /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_unbound_total_scores/C_34_V_to_R/gamess_io/gamess_qm_energy_thread0_2026_05_18_18_11_39.log': No such file or directory
In the MDS, the problem is when the 'GSK' directory is looked up as part of the path lookup. client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK
It seems like the MDS is looking in its cache, and instead of the snapshot it finds an entry for the live tree where there is no GSK directory. Then the MDS returns ENOENT for the GSK directory in the snapshot (!) - so anything under GSK will be invisible to the client in the snapshot:
2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 400 caller_uid=0, caller_gid=0{0,}) v6
Here are the full logs:
2026-07-23T10:34:53.281-0400 7ffff205a640 4 mds.0.server handle_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0,} ) v6 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.5902540 get_session have 0x556382924000 client.265674856 v1:10.250.129.151:0/2806743593 state open 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.server oldest_client_tid=7 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_start request(client.265674856:7 nref=2 cr=0x555b0018b500) 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server dispatch_client_request client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0400 caller_uid=0, caller_gid=0{0 ,}) v6 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server rdlock_path_pin_ref request(client.265674856:7 nref=2 cr=0x555b0018b500) #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache traverse: opening base ino 0x100131197c4 snap head path depth 5 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 0 '' snapid head 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snapdir 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 1 'sync-20260701-0904' snapid snapdir 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) resolve_snapname 'sync-20260701-0904' in [0,head] 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) ? snap(1c9 0x100131197c4 'sync-20260701-0904' 2026-07-01T09:04:25.746531-0400) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: snap sync-20260701-0904 -> 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 2 'bturzo' snapid 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) lookup (bturzo, '1c9') 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100131197c4.101*) hit -> (bturzo,head) 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 3 'Projects' snapid 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) lookup (Projects, '1c9') 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x100fb6d2ec8) hit -> (Projects,head) 2026-07-23T10:34:53.281-0400 7ffff205a640 12 mds.0.cache traverse: path seg depth 4 'GSK' snapid 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) lookup (GSK, '1c9') 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.dir(0x101a2629a23) hit -> (GSK,head) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache traverse: null+readable dentry at [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.265674856:7 lookup #0x100131197c4//sync-20260701-0904/bturzo/Projects/GSK 2026-07-23T10:34:52.705921-0 400 caller_uid=0, caller_gid=0{0,}) v6 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.server apply_allocated_inos 0x0 / [] / 0x0 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server lat 0.575392 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist snapid 1c9 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache.ino(0x101a2629a23) encode_inodestat snapid 1c9 to old_inode [1a7,1d3] n(v1256297 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003833=931566+72267) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snaps 1c9,1ec,1ee,1f0,1f2,1f4 (seq 1c9 cached_seq 1f5) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x100131197c4 seq 457 0x556314ca4240) get_snap_info snaps 1c9,1ec,1ee,1f0,1f2,1f4 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snaps 1ec,1ee,1f0,1f2,1f4 (seq 1f5 cached_seq 1f5) 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.snaprealm(0x1 seq 501 0x555558ed2900) get_snap_info snaps 1ec,1ee,1f0,1f2,1f4 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.cache.ino(0x101a2629a23) pfile 0 pauth 0 plink 0 pxattr 0 plocal 0 mtime 2026-06-08T14:02:36.442123-0400 ctime 2026-06-08T14:02:36.442123-0400 change_attr 196 valid=1 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.cache.ino(0x101a2629a23) encode_inodestat caps pAsLsXsFs seq 0 mseq 0 xattrv 1 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added diri [inode 0x101a2629a23 [...1ea,head] /users/bturzo/Projects/ auth v22346457 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+41) n(v1256393 rc2026-07-17T12:32:38. 513951-0400 b70376541394148 1003833=931566+72267) old_inodes=2 (inest lock) (iversion lock) caps={254564708=pAsLsXsFs/-@9,258742051=pAsLsXsFs/pAsLsXsFsxcral@3} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=0 wait er=0 authpin=0 0x556f6adfe580] 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.server set_trace_dist added dir [dir 0x101a2629a23 /users/bturzo/Projects/ [2,head] auth v=26589887 cv=26589887/26589887 REP state=1074003968 f(v0 m2026-07-17T11:56:20.641134-0400 45=4+4 1) n(v1256393 rc2026-07-17T12:32:38.513951-0400 b70376541394148 1003832=931566+72266) hs=45+1,ss=0+0 | child=1 dirty=0 waiter=0 authpin=0 0x555a65593a80] 2026-07-23T10:34:53.281-0400 7ffff205a640 20 mds.0.locker issue_client_lease no/null lease on [dentry #0x1/users/bturzo/Projects/GSK [2,head] auth NULL (dversion lock) pv=0 v=26589887 ino=(nil) state=1073741824 0x55714a72d180] 2026-07-23T10:34:53.281-0400 7ffff205a640 10 mds.0.5902540 send_message_client client.265674856 v1:10.250.129.151:0/2806743593 client_reply(???:7 = -2 (2) No such file or directory) v1 2026-07-23T10:34:53.281-0400 7ffff205a640 7 mds.0.cache request_finish request(client.265674856:7 nref=3 cr=0x555b0018b500) 2026-07-23T10:34:53.281-0400 7ffff205a640 15 mds.0.cache request_cleanup request(client.265674856:7 nref=3 cr=0x555b0018b500)
The file system I'm reading for the copying is running ceph Reef 18.2.8, so this is a Reef MDS, in case the version/patches matter. Thanks very much for looking into this - the help is appreciated!
Andras
On 7/22/26 8:28 PM, Xiubo Li wrote:
Hi Andras,
Thanks for your quick feedback.
Sure. Yesterday Dhairya and I found something important in both the MDS and kclient sides. Without the logs we will review the code to ensure our investigation covers this case. We will provide feedback later.
Regards Xiubo
On Thu, 23 Jul 2026 at 00:14, Andras Pataki via ceph-users<ceph-users@ceph.io> <ceph-users@ceph.io> wrote:
Hi Xiubo and Dhariya,
The way I was able to reproduce the problem was to look at the error log of my copy processes and take those files and check on them on the various clients. Today, unfortunately, all those files seem to be visible - and at the moment I don't have a way of reproducing the issue from scratch. I'm trying to see what could be done to set up the reproduction from scratch. If I manage to do that, I'll try to gather some information on whether this is a client issue (i.e. flushing the client caches changes anything) or whether it comes from the MDS. Unfortunately the MDS in question is a very busy one, our main file system MDS, so there are limits to what I can do to it.
Andras
On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote:
Hi Dhairya,
Yeah, makes sense.
Since I couldn't reproduce this locally, @Andras Pataki Could you help reproduce it by enabling the debug logs as Dhairya mentioned? And if possible please also try with the fuse client to see could you see too.
Thanks Xiubo
On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev@ceph.io> <dev@ceph.io> wrote:
Hi Xiubo,
I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds/... . Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT?
On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li@clyso.com> <xiubo.li@clyso.com> wrote:
Hi Andras, Dhairya,
I recall hitting a similar issue with this and the workaround involves running the 'ls' command, but I can't recall exactly which command or the tracker reference. This happened years ago.
This morning I tried to reproduce this locally but failed. And I just went through the mds and kclient code just found two suspect places could cause this:
MDS side: in https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/M...
MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist:
if (curdir->is_complete() || // (1) (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) return -ENOENT;
is_complete() means all live entries are cached in memory. Shortcut (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), but shortcut (1) applies to all snapids — including snapshots.
When a snapshot exists but the file was later deleted from the live tree, the MDS eventually evicts the deleted file's dentry from the in-memory cache (a null dentry, which trim_dentry() removes without clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits is_complete()=true with the dentry absent (ENOENT), without ever checking the OSD where the snapshot data actually lives.
READDIR takes a different code path that fetches from the OSD, which is why ls -l fixes it.
Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: if (snapid == CEPH_NOSNAP && (curdir->is_complete() || ...bloom...)) {
@Andras
BTW, to verify the above could you try to reproduce this by using the ceph-fuse mount ? Locally I couldn't reproduce it by trying many times and I am planing to add some debug code in MDS to ammulated it to reproduce it later.
====
Kclient side: https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-lin... Plus a kclient defense: ceph_d_revalidate() unconditionally trusts negative dentries in snapshot directories, caching a transient ENOENT forever. Change valid = 1 to valid = inode ? 1 : 0.
I just created on trakcer to follow this: https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-az...
Thanks Xiubo
On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users<ceph-users@ceph.io> <ceph-users@ceph.io> wrote:
Hi Andras,
This sounds like the client served the lookups from its cached negative? Can you check if you see the lookup calls being made to MDS and maybe set debug_mds=20 to check if the client did send the request but it was MDS which returned ENOENT but since the `ls -l` succeeded I'm wondering if this is actually something from MDS. Maybe try a sync and drop_caches to validate that?
*Dhairya Parmar*
Software Engineer, CephFS
On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < ceph-users@ceph.io> <ceph-users@ceph.io> wrote:
Hi ceph users/devs,
We've been encountering a strange issue with snapshots as part of a project to copy our large ceph file system to a backup in parallel across nodes. We are taking a snapshot, and then doing a copy of the snapshot, which we expect to be static (unchanging) - using several clients in parallel - all kernel mounts. Some clients are unable to open files and directories that are known to exist (were discovered traversing the file system on perhaps another node). After some playing around, I could reproduce the issue on a single node with just 'ls' as follows:
# This file doesn't seems to exist - and some number of parent directories are also "missing": [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen
ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen':
No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': No such file or directory # Until we get to this 'Projects' parent directory, which seems to be there [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 1961 1961 46 Jun 8 14:02 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it is still not there ... [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': No such file or directory # But ... let's see if the " <:[root@cephmds603~]#ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.logls:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log':Nosuchfileordirectory[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_iols:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io':Nosuchfileordirectory[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_Wls:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W':Nosuchfileordirectory[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scoresls:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores':Nosuchfileordirectory[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysisls:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis':Nosuchfileordirectory[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/outputls:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output':Nosuchfileordirectory[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_genls:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen':Nosuchfileordirectory[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQMls:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM':Nosuchfileordirectory[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSKls:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK':Nosuchfileordirectory%23Untilwegettothis'Projects'parentdirectory,whichseemstobethere[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projectsdrwxr-sr-x451961196146Jun814:02/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects%23GSKunderitisstillnotthere...[root@cephmds603~]%23ls-ld/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSKls:cannotaccess'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK':Nosuchfileordirectory%23But...let'sseeifthe>tickle" the Projects directory a bit ... [root@cephmds603 ~]# ls -l /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null
# Now all subdirectories and the file in question immediately becomes visible: [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK drwxrwsr-x 5 1961 1961 3 May 7 17:18 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM drwxrwsr-x 12 1961 1961 10 May 11 16:08 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen drwxrwsr-x 5 1961 1961 3 May 11 16:18 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
# This is the kernel client 6.18.32 (relatively recent LTS kernel) with ceph squid 19.2.3: [root@cephmds603 ~]# uname -a Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 x86_64 x86_64 GNU/Linux [root@cephmds603 ~]# ceph --version ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@cephmds603 ~]#
What is most interesting is that in the live tree (non-snapshotted tree) - the Projects directory exists, but everything under it has been removed!
[root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or directory
Am I running into some known issue? This looks to be specific to some clients - i.e. some clients see the directory and some don't at the same point in time, i.e. this is perhaps a kclient problem? Any ideas/suggestions would be appreciated,
Andras _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Andras Pataki
-
Dhairya Parmar
-
Xiubo Li