Thanks for the fast answer! Is there any (other) way to get a complete list of extended attributes? Is there something documented - meaning what can I rely on in the future? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Yan, Zheng <ukernel@gmail.com> Sent: 10 December 2019 13:25 To: Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] getfattr problem on ceph-fs On Tue, Dec 10, 2019 at 8:06 PM Frank Schilder <frans@dtu.dk> wrote:

I have a strange problem with ceph fs and extended attributes. I have two Centos machines where I mount cephfs in exactly the same way (I manually executed the exact same mount command on both machines). On one of the machines, getfattr returns this:

[root@ceph-01 ~]# getfattr -d -m 'ceph.*' getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/hpc/home ceph.dir.entries="49" ceph.dir.files="1" ceph.dir.rbytes="77816237666910" ceph.dir.rctime="1575978038.0976848840" ceph.dir.rentries="6673312" ceph.dir.rfiles="6271408" ceph.dir.rsubdirs="401904" ceph.dir.subdirs="48"

and on the other I get nothing:

[root@gnosis ~]# getfattr -d -m 'ceph.*' /mnt/cephfs/hpc/home

No error message, just nothing.

The only difference is, that ceph-01 was kickstarted with Centos7.6 while gnosis was kickstarted with Centos7.7. Otherwise, both machines are deployed identically. getfattr is the same version on both. Kernel versions are ceph-01:5.0.2-1.el7.elrepo.x86_64 and gnosis:5.4.2-1.el7.elrepo.x86_64.

Does anyone have a pointer what to look for?

recent version kclient/ceph-fuse hide ceph.* xattrs from listxattr(2) syscall. these xattrs still can be accessed by their names. such as getfattr -n ceph.dir.rctime /mnt/cephfs/hpc/home

Thanks!

================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi David, thanks for the link, I understand the problem now. I don't think this can be solved within ceph alone in a good way. There are other storage system that also use file system meta objects like .snap dirs and automatic xattribs for administrative purposes. All these are non-copy objects (dirs and attribs) and it almost sounds like this should become an extension of the respective APIs rather than workarounds on the storage system level. For example, it would also be nice if the .snap dirs (containing actual snaps) were discoverable directly on a file system with a tool like find but protected from copy. The MDS now has functions for showing such info, but holding a parallel data structure somewhere else than the mount point looks more like a workaround than an elegant integration and requires cluster admin access to answer a rather simple question a non-privileged user could answer herself. In the mean time, my attempt of a complete list: ceph.dir.entries ceph.dir.files ceph.dir.rbytes ceph.dir.rctime ceph.dir.rentries ceph.dir.rfiles ceph.dir.rsubdirs ceph.dir.subdirs ceph.dir.layout ceph.file.layout ceph.quota.max_bytes ceph.quota.max_files Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: David Disseldorp <ddiss@suse.de> Sent: 10 December 2019 15:55:07 To: Frank Schilder Cc: Yan, Zheng; ceph-users Subject: ORe: [ceph-users] Re: getfattr problem on ceph-fs Hi, On Tue, 10 Dec 2019 12:40:35 +0000, Frank Schilder wrote:

Thanks for the fast answer!

Is there any (other) way to get a complete list of extended attributes?

Not at the moment, as far as I can tell.

Is there something documented - meaning what can I rely on in the future?

When the listxattr() entries were removed, Greg proposed that we could add a "ceph" namespaced xattr to provide discoverability: https://www.spinics.net/lists/ceph-devel/msg46220.html Somebody just needs to do the work to implement this :-) Cheers, David