CephFS subvolumes not inheriting ephemeral distributed pin configured on the subvolumegroups. References: https://github.com/ceph/ceph/pull/35759 https://github.com/ceph/ceph/pull/43896 # Environmet setup Ceph version: 18.2.4 and 19.2.0 CephFS with multiple MDS (2 active + 1 standby MDS) RHEL 9.4 Linux myserver 5.14.0-427.37.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 13 12:41:50 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux # Ceph FS configuration ``` ceph config set mds mds_export_ephemeral_distributed true ceph fs set midline-a balance_automate false ``` # Subvolume group (csi) creation ``` ceph fs subvolumegroup create midline-a csi ``` # Subvolume group (csi) pinning (tried both methods) ``` ceph fs subvolumegroup pin midline-a csi distributed 1 setfattr -n ceph.dir.pin.distributed -v 1 setfattr -n ceph.dir.pin.distributed -v 1 /mnt/cephfs/volumes/csi/ ``` # Subvolume creation ``` ceph fs subvolume create midline-a subvol1 csi ceph fs subvolume create midline-a subvol2 csi ``` # Get attribute (subvolume group) ``` getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/volumes/csi ceph.dir.pin.distributed="1" ``` # Get attribute (subvolume) - Subvolumes not inheriting the distributed pin attribute ``` getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi/subvol1 getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/volumes/csi/subvol1 ceph.dir.pin.distributed="0" ``` ``` getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi/subvol2 getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/volumes/csi/subvol2 ceph.dir.pin.distributed="0" ``` # MDS logs (subtree information) ``` ceph tell mds.abc123 get subtrees | jq '.[] | [.dir.path, .auth_first, .export_pin, .export_pin_target]' [ "~mds1", 1, -1, -1 ] [ "", 0, -1, -1 ] [ "/volumes/csi", 1, -1, 1 ] ``` ``` ceph tell mds.xyz123 get subtrees | jq '.[] | [.dir.path, .auth_first, .export_pin, .export_pin_target]' [ "/volumes/csi", 1, -1, 1 ] [ "", 0, -1, -1 ] [ "~mds0", 0, -1, -1 ] [ "/volumes/csi", 0, -1, 0 ] ``` Regards, Rajmohan R