``` Subvolumes do not "inherit" the distributed ephemeral pin. What you should expect below is that the "csi" subvolumegroup will be fragmented and distributed across the ranks. Consequently, the subvolumes will also be distributed across ranks as part of the subtrees rooted at each fragment of the "csi" subvolumegroup (directory). ``` How is subvolumegroup fragmentation handled? Are the subvolumes equally distributed across all available active MDS? In the following scenario, will 3 of the subvolumes be mapped to each of the MDS? Will setting the ephemeral distributed pin on Subvolumegroup ensure that the subvolumes in it will be equally distributed across MDS ? We are looking at ceph-csi use case for Kubernetes. PVs (subvolumes) are dynamically created by Kubernetes. # Ceph FS configuration ceph fs subvolumegroup create midline-a csi ceph fs subvolumegroup pin midline-a csi distributed 1 ceph fs subvolume create midline-a subvol1 csi ceph fs subvolume create midline-a subvol2 csi ceph fs subvolume create midline-a subvol3 csi ceph fs subvolume create midline-a subvol4 csi ceph fs subvolume create midline-a subvol5 csi ceph fs subvolume create midline-a subvol6 csi # ceph fs ls name: midline-a, metadata pool: fs-midline-metadata-a, data pools: [ fs-midline-data-a ] # ceph fs subvolumegroup ls midline-a [ { "name": "csi" } ] # ceph fs subvolume ls midline-a csi [ { "name": "subvol4" }, { "name": "subvol2" }, { "name": "subvol3" }, { "name": "subvol5" }, { "name": "subvol6" }, { "name": "subvol1" } ] # ceph fs status midline-a - 2 clients ========= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active midline.server1.njyfcn Reqs: 0 /s 514 110 228 36 1 active midline.server2.lpnjmx Reqs: 0 /s 47 22 17 6 POOL TYPE USED AVAIL fs-midline-metadata-a metadata 25.4M 25.9T fs-midline-data-a data 216k 25.9T STANDBY MDS midline.server3.wsbxsh MDS version: ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable) Following are the subtrees output from the MDSs. The directory fragments does not seem to be equally mapped to MDS. # ceph tell mds.midline.server1.njyfcn get subtrees | jq [ { "is_auth": true, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": false, "random_ephemeral_pin": false, "export_pin_target": -1, "dir": { "path": "", "dirfrag": "0x1", "snapid_first": 2, "projected_version": "1240", "version": "1240", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "0", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": { "1": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "subtree": 1, "subtreetemp": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 0 }, "nref": 4 } }, { "is_auth": true, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": false, "random_ephemeral_pin": false, "export_pin_target": -1, "dir": { "path": "~mds0", "dirfrag": "0x100", "snapid_first": 2, "projected_version": "1232", "version": "1232", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "0", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "subtree": 1, "subtreetemp": 0, "dirty": 1, "waiter": 0, "authpin": 0 }, "nref": 3 } }, { "is_auth": false, "auth_first": 1, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": true, "random_ephemeral_pin": false, "export_pin_target": 1, "dir": { "path": "/volumes/csi", "dirfrag": "0x100000006ae.11*", "snapid_first": 2, "projected_version": "50", "version": "50", "committing_version": "50", "committed_version": "50", "is_rep": false, "dir_auth": "1", "states": [], "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 0, "child": 0, "frozen": 0, "subtree": 1, "replicated": 0, "dirty": 0, "waiter": 0, "authpin": 0, "tempexporting": 0 }, "nref": 1 } }, { "is_auth": true, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": true, "random_ephemeral_pin": false, "export_pin_target": 0, "dir": { "path": "/volumes/csi", "dirfrag": "0x100000006ae.10*", "snapid_first": 2, "projected_version": "52", "version": "52", "committing_version": "50", "committed_version": "50", "is_rep": false, "dir_auth": "0", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "subtree": 1, "dirty": 1, "waiter": 0, "authpin": 0 }, "nref": 2 } }, { "is_auth": true, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": true, "random_ephemeral_pin": false, "export_pin_target": 0, "dir": { "path": "/volumes/csi", "dirfrag": "0x100000006ae.01*", "snapid_first": 2, "projected_version": "136", "version": "136", "committing_version": "82", "committed_version": "82", "is_rep": false, "dir_auth": "0", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": { "1": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "frozen": 0, "subtree": 1, "replicated": 1, "dirty": 1, "authpin": 0 }, "nref": 4 } } ] # ceph tell mds.midline.server2.lpnjmx get subtrees | jq [ { "is_auth": true, "auth_first": 1, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": false, "random_ephemeral_pin": false, "export_pin_target": -1, "dir": { "path": "~mds1", "dirfrag": "0x101", "snapid_first": 2, "projected_version": "332", "version": "332", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "1", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "subtree": 1, "subtreetemp": 0, "dirty": 1, "waiter": 0, "authpin": 0 }, "nref": 3 } }, { "is_auth": true, "auth_first": 1, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": true, "random_ephemeral_pin": false, "export_pin_target": 1, "dir": { "path": "/volumes/csi", "dirfrag": "0x100000006ae.11*", "snapid_first": 2, "projected_version": "66", "version": "66", "committing_version": "50", "committed_version": "50", "is_rep": false, "dir_auth": "1", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "child": 1, "frozen": 0, "subtree": 1, "importing": 0, "replicated": 1, "dirty": 1, "authpin": 0 }, "nref": 4 } }, { "is_auth": false, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": false, "random_ephemeral_pin": false, "export_pin_target": -1, "dir": { "path": "", "dirfrag": "0x1", "snapid_first": 2, "projected_version": "0", "version": "1216", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "0", "states": [], "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "subtree": 1 }, "nref": 2 } } ] Regards, Rajmohan R On Wed, Nov 20, 2024 at 12:48 PM Patrick Donnelly <pdonnell@redhat.com> wrote:

On Tue, Nov 19, 2024 at 8:38 PM Rajmohan Ramamoorthy <ram.rajmohanr@gmail.com> wrote:

...

CephFS subvolumes not inheriting ephemeral distributed pin configured on the subvolumegroups.

Subvolumes do not "inherit" the distributed ephemeral pin. What you should expect below is that the "csi" subvolumegroup will be fragmented and distributed across the ranks. Consequently, the subvolumes will also be distributed across ranks as part of the subtrees rooted at each fragment of the "csi" subvolumegroup (directory).

...

References:

https://github.com/ceph/ceph/pull/35759 https://github.com/ceph/ceph/pull/43896

# Environmet setup

Ceph version: 18.2.4 and 19.2.0

Meaning you tested two versions?

...

CephFS with multiple MDS (2 active + 1 standby MDS) RHEL 9.4 Linux myserver 5.14.0-427.37.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 13 12:41:50 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

# Ceph FS configuration ``` ceph config set mds mds_export_ephemeral_distributed true ceph fs set midline-a balance_automate false ``` # Subvolume group (csi) creation ``` ceph fs subvolumegroup create midline-a csi ``` # Subvolume group (csi) pinning (tried both methods) ``` ceph fs subvolumegroup pin midline-a csi distributed 1

setfattr -n ceph.dir.pin.distributed -v 1 setfattr -n ceph.dir.pin.distributed -v 1 /mnt/cephfs/volumes/csi/ ```

You ran setfattr and ` ceph fs subvolumegroup pin midline-a csi distributed 1` ?

...

# Subvolume creation ``` ceph fs subvolume create midline-a subvol1 csi ceph fs subvolume create midline-a subvol2 csi ```

# Get attribute (subvolume group) ``` getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi

getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/volumes/csi ceph.dir.pin.distributed="1" ```

# Get attribute (subvolume) - Subvolumes not inheriting the distributed pin attribute

``` getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi/subvol1

getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/volumes/csi/subvol1 ceph.dir.pin.distributed="0" ```

``` getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi/subvol2

getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/volumes/csi/subvol2 ceph.dir.pin.distributed="0" ```

these last two outputs are expected.

...

# MDS logs (subtree information) ``` ceph tell mds.abc123 get subtrees | jq '.[] | [.dir.path, .auth_first, .export_pin, .export_pin_target]'

[ "~mds1", 1, -1, -1 ] [ "", 0, -1, -1 ] [ "/volumes/csi", 1, -1, 1 ] ```

``` ceph tell mds.xyz123 get subtrees | jq '.[] | [.dir.path, .auth_first, .export_pin, .export_pin_target]'

[ "/volumes/csi", 1, -1, 1 ] [ "", 0, -1, -1 ] [ "~mds0", 0, -1, -1 ] [ "/volumes/csi", 0, -1, 0 ] ```

It looks like the "csi" directory was split but the output is incomplete. Please paste the full (anonymized) `get subtrees` output.

-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

Hi Patrick, Few other follow up questions. Is directory fragmentation applicable only when multiple active MDS is enabled for a Ceph FS? Will directory fragmenation and distribution of fragments amongs active MDS happen if we turn off balancer for a Ceph FS volume `ceph fs set midline-a balance_automate false` ? In Squide, the CephFS automatic metadata load (sometimes called “default”) balancer is now disabled by default. ( https://docs.ceph.com/en/latest/releases/squid/) Is there a way for us to ensure that the directory tree of a Subvolume (Kubernetes PV) is part of a same fragment and handled by a single MDS so that a client operations are handled by one MDS? What is the trigger to start fragmenting directories within a Subvolumegroup? With the `balance_automate` set to false and `ephemeral distributed pin` enabled for a Subvolumegroup, can we expect (almost) equal distribution of Subvolumes (Kubernetes PVs) amongst the active MDS daemons and stable operation without hotspot migrations? Regards, Rajmohan R On Wed, Nov 20, 2024 at 1:27 PM Patrick Donnelly <pdonnell@redhat.com> wrote:

On Tue, Nov 19, 2024 at 9:20 PM Rajmohan Ramamoorthy <ram.rajmohanr@gmail.com> wrote:

...

``` Subvolumes do not "inherit" the distributed ephemeral pin. What you should expect below is that the "csi" subvolumegroup will be fragmented and distributed across the ranks. Consequently, the subvolumes will also be distributed across ranks as part of the subtrees rooted at each fragment of the "csi" subvolumegroup (directory). ```

How is subvolumegroup fragmentation handled?

Fragmentation is automatically applied (to a minimum level) when a directory is marked with the distributed ephemeral pin.

...

Are the subvolumes equally distributed across all available active MDS?

As the documentation says, it's a consistent hash of the fragments (which include the subvolumes which fall into those fragments) across ranks.

...

In the following scenario, will 3 of the subvolumes be mapped to each of the MDS?

You cannot say. It depends how the fragments are hashed.

...

Will setting the ephemeral distributed pin on Subvolumegroup ensure that the subvolumes in it will be equally distributed across MDS ?

Approximately.

...

We are looking at ceph-csi use case for Kubernetes. PVs (subvolumes) are dynamically created by Kubernetes.

This is an ideal use-case for the distributed ephemeral pin.

...

# Ceph FS configuration

ceph fs subvolumegroup create midline-a csi ceph fs subvolumegroup pin midline-a csi distributed 1

ceph fs subvolume create midline-a subvol1 csi ceph fs subvolume create midline-a subvol2 csi ceph fs subvolume create midline-a subvol3 csi ceph fs subvolume create midline-a subvol4 csi ceph fs subvolume create midline-a subvol5 csi ceph fs subvolume create midline-a subvol6 csi

# ceph fs ls name: midline-a, metadata pool: fs-midline-metadata-a, data pools: [fs-midline-data-a ]

# ceph fs subvolumegroup ls midline-a [ { "name": "csi" } ]

# ceph fs subvolume ls midline-a csi [ { "name": "subvol4" }, { "name": "subvol2" }, { "name": "subvol3" }, { "name": "subvol5" }, { "name": "subvol6" }, { "name": "subvol1" } ]

# ceph fs status midline-a - 2 clients ========= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active midline.server1.njyfcn Reqs: 0 /s 514 110 228 36 1 active midline.server2.lpnjmx Reqs: 0 /s 47 22 17 6 POOL TYPE USED AVAIL fs-midline-metadata-a metadata 25.4M 25.9T fs-midline-data-a data 216k 25.9T STANDBY MDS midline.server3.wsbxsh MDS version: ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable)

Following are the subtrees output from the MDSs. The directory fragments does not seem to be equally mapped to MDS.

# ceph tell mds.midline.server1.njyfcn get subtrees | jq [ { "is_auth": true, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": false, "random_ephemeral_pin": false, "export_pin_target": -1, "dir": { "path": "", "dirfrag": "0x1", "snapid_first": 2, "projected_version": "1240", "version": "1240", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "0", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": { "1": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "subtree": 1, "subtreetemp": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 0 }, "nref": 4 } }, { "is_auth": true, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": false, "random_ephemeral_pin": false, "export_pin_target": -1, "dir": { "path": "~mds0", "dirfrag": "0x100", "snapid_first": 2, "projected_version": "1232", "version": "1232", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "0", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "subtree": 1, "subtreetemp": 0, "dirty": 1, "waiter": 0, "authpin": 0 }, "nref": 3 } }, { "is_auth": false, "auth_first": 1, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": true, "random_ephemeral_pin": false, "export_pin_target": 1, "dir": { "path": "/volumes/csi", "dirfrag": "0x100000006ae.11*",

This fragment is pinned to rank 1 (export_pin_target).

...

"snapid_first": 2, "projected_version": "50", "version": "50", "committing_version": "50", "committed_version": "50", "is_rep": false, "dir_auth": "1", "states": [], "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 0, "child": 0, "frozen": 0, "subtree": 1, "replicated": 0, "dirty": 0, "waiter": 0, "authpin": 0, "tempexporting": 0 }, "nref": 1 } }, { "is_auth": true, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": true, "random_ephemeral_pin": false, "export_pin_target": 0, "dir": { "path": "/volumes/csi", "dirfrag": "0x100000006ae.10*",

This fragment is pinned to rank 0 (export_pin_target).

...

"snapid_first": 2, "projected_version": "52", "version": "52", "committing_version": "50", "committed_version": "50", "is_rep": false, "dir_auth": "0", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "subtree": 1, "dirty": 1, "waiter": 0, "authpin": 0 }, "nref": 2 } }, { "is_auth": true, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": true, "random_ephemeral_pin": false, "export_pin_target": 0, "dir": { "path": "/volumes/csi", "dirfrag": "0x100000006ae.01*",

This fragment is pinned to rank 0 (export_pin_target).

...

"snapid_first": 2, "projected_version": "136", "version": "136", "committing_version": "82", "committed_version": "82", "is_rep": false, "dir_auth": "0", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": { "1": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "frozen": 0, "subtree": 1, "replicated": 1, "dirty": 1, "authpin": 0 }, "nref": 4 } } ]

# ceph tell mds.midline.server2.lpnjmx get subtrees | jq [ { "is_auth": true, "auth_first": 1, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": false, "random_ephemeral_pin": false, "export_pin_target": -1, "dir": { "path": "~mds1", "dirfrag": "0x101", "snapid_first": 2, "projected_version": "332", "version": "332", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "1", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "subtree": 1, "subtreetemp": 0, "dirty": 1, "waiter": 0, "authpin": 0 }, "nref": 3 } }, { "is_auth": true, "auth_first": 1, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": true, "random_ephemeral_pin": false, "export_pin_target": 1, "dir": { "path": "/volumes/csi", "dirfrag": "0x100000006ae.11*", "snapid_first": 2, "projected_version": "66", "version": "66", "committing_version": "50", "committed_version": "50", "is_rep": false, "dir_auth": "1", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "child": 1, "frozen": 0, "subtree": 1, "importing": 0, "replicated": 1, "dirty": 1, "authpin": 0 }, "nref": 4 } }, { "is_auth": false, "auth_first": 0, "auth_second": -2, "export_pin": -1, "distributed_ephemeral_pin": false, "random_ephemeral_pin": false, "export_pin_target": -1, "dir": { "path": "", "dirfrag": "0x1", "snapid_first": 2, "projected_version": "0", "version": "1216", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "0", "states": [], "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "subtree": 1 }, "nref": 2 } } ]

This all looks as expected. You can verify the location of the subvolumes by:

ceph tell mds.X dump tree /volumes/csi/X 0

and check auth_first. You should see that they are approximately uniformly distributed.

-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D