On Sun, Apr 9, 2023 at 11:21 PM Ulrich Pralle <Ulrich.Pralle@tu-berlin.de> wrote:
Hi,
we are using ceph version 17.2.5 on Ubuntu 22.04.1 LTS.
We deployed multi-mds (max_mds=4, plus standby-replay mds). Currently we statically directory-pinned our user home directories (~50k). The cephfs' root directory is pinned to '-1', ./homes is pinned to "0". All user home directories below ./homes/ are pinned to -1, 1, 2, or 3 depending on a simple hash algorithm. Cephfs is provided to our users as samba/cifs (clustered samba,ctdb).
We want to try ephemeral directory pinning.
We can successfully set the extended attribute "ceph.dir.pin.distributed" with setfattr(1), but cannot retrieve its setting afterwards.:
# setfattr -n ceph.dir.pin.distributed -v 1 ./units # getfattr -n ceph.dir.pin.distributed ./units ./units: ceph.dir.pin.distributed: No such attribute
strace setfattr reports success on setxattr
setxattr("./units", "ceph.dir.pin.distributed", "1", 1, 0) = 0
strace getfattr reports
lstat("./units", {st_mode=S_IFDIR|0751, st_size=1, ...}) = 0 getxattr("./units", "ceph.dir.pin.distributed", NULL, 0) = -1 ENODATA (No data available)
The file system is mounted rw,noatime,,name=<omitted>,mds_namespace=<omitted>.acl,recover_session=clean. The cephfs mds caps are "allow rwps". "./units" has a ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=fs_data_units" Ubuntu's setfattr is version 2.4.48.
Defining other cephfs extend attributes (like ceph.dir.pin, ceph.quota.max_bytes, etc.) works as expected.
What are we missing?
Your kernel doesn't appear to know how to check virtual extended attributes yet. It should be in 5.18.
Should we clear all static directory pinnings in advance?
Start by removing the pin on /home. Then remove a group of pins on some users directories. Confirm /home looks something like: ceph tell mds.<fsname>:0 dump tree /home 0 | jq '.[0].dirfrags[] | .dir_auth' "0" "0" "1" "1" "1" "1" "0" "0" Which tells you the dirfrags for /home are distributed across the ranks (in this case, 0 and 1). At that point, it should be fine to remove the rest of the manual pins.
Are there any experience with ephemeral directory pinning? Or should one refrain from multi-mds at all?
It should work fine. Please give it a try and report back! -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D