10 Dec
2019
10 Dec
'19
12:05 p.m.
I have a strange problem with ceph fs and extended attributes. I have two Centos machines
where I mount cephfs in exactly the same way (I manually executed the exact same mount
command on both machines). On one of the machines, getfattr returns this:
[root@ceph-01 ~]# getfattr -d -m 'ceph.*' /mnt/cephfs/hpc/home
getfattr: Removing leading '/' from absolute path names
# file: mnt/cephfs/hpc/home
ceph.dir.entries="49"
ceph.dir.files="1"
ceph.dir.rbytes="77816237666910"
ceph.dir.rctime="1575978038.0976848840"
ceph.dir.rentries="6673312"
ceph.dir.rfiles="6271408"
ceph.dir.rsubdirs="401904"
ceph.dir.subdirs="48"
and on the other I get nothing:
[root@gnosis ~]# getfattr -d -m 'ceph.*' /mnt/cephfs/hpc/home
No error message, just nothing.
The only difference is, that ceph-01 was kickstarted with Centos7.6 while gnosis was
kickstarted with Centos7.7. Otherwise, both machines are deployed identically. getfattr is
the same version on both. Kernel versions are ceph-01:5.0.2-1.el7.elrepo.x86_64 and
gnosis:5.4.2-1.el7.elrepo.x86_64.
Does anyone have a pointer what to look for?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
10 Dec
10 Dec
12:25 p.m.
On Tue, Dec 10, 2019 at 8:06 PM Frank Schilder <frans(a)dtu.dk> wrote:
I have a strange problem with ceph fs and extended attributes. I have two Centos machines
where I mount cephfs in exactly the same way (I manually executed the exact same mount
command on both machines). On one of the machines, getfattr returns this:
[root@ceph-01 ~]# getfattr -d -m 'ceph.*'
getfattr: Removing leading '/' from absolute path names
# file: mnt/cephfs/hpc/home
ceph.dir.entries="49"
ceph.dir.files="1"
ceph.dir.rbytes="77816237666910"
ceph.dir.rctime="1575978038.0976848840"
ceph.dir.rentries="6673312"
ceph.dir.rfiles="6271408"
ceph.dir.rsubdirs="401904"
ceph.dir.subdirs="48"
and on the other I get nothing:
[root@gnosis ~]# getfattr -d -m 'ceph.*' /mnt/cephfs/hpc/home
No error message, just nothing.
The only difference is, that ceph-01 was kickstarted with Centos7.6 while gnosis was
kickstarted with Centos7.7. Otherwise, both machines are deployed identically. getfattr is
the same version on both. Kernel versions are ceph-01:5.0.2-1.el7.elrepo.x86_64 and
gnosis:5.4.2-1.el7.elrepo.x86_64.
Does anyone have a pointer what to look for?
recent version kclient/ceph-fuse hide ceph.* xattrs from listxattr(2)
syscall. these xattrs still can be accessed by their names. such as
getfattr -n ceph.dir.rctime /mnt/cephfs/hpc/home
> Thanks!
>
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:40 p.m.
Thanks for the fast answer!
Is there any (other) way to get a complete list of extended attributes?
Is there something documented - meaning what can I rely on in the future?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Yan, Zheng <ukernel(a)gmail.com>
Sent: 10 December 2019 13:25
To: Frank Schilder
Cc: ceph-users
Subject: Re: [ceph-users] getfattr problem on ceph-fs
On Tue, Dec 10, 2019 at 8:06 PM Frank Schilder <frans(a)dtu.dk> wrote:
I have a strange problem with ceph fs and extended attributes. I have two Centos machines
where I mount cephfs in exactly the same way (I manually executed the exact same mount
command on both machines). On one of the machines, getfattr returns this:
[root@ceph-01 ~]# getfattr -d -m 'ceph.*'
getfattr: Removing leading '/' from absolute path names
# file: mnt/cephfs/hpc/home
ceph.dir.entries="49"
ceph.dir.files="1"
ceph.dir.rbytes="77816237666910"
ceph.dir.rctime="1575978038.0976848840"
ceph.dir.rentries="6673312"
ceph.dir.rfiles="6271408"
ceph.dir.rsubdirs="401904"
ceph.dir.subdirs="48"
and on the other I get nothing:
[root@gnosis ~]# getfattr -d -m 'ceph.*' /mnt/cephfs/hpc/home
No error message, just nothing.
The only difference is, that ceph-01 was kickstarted with Centos7.6 while gnosis was
kickstarted with Centos7.7. Otherwise, both machines are deployed identically. getfattr is
the same version on both. Kernel versions are ceph-01:5.0.2-1.el7.elrepo.x86_64 and
gnosis:5.4.2-1.el7.elrepo.x86_64.
Does anyone have a pointer what to look for?
recent version kclient/ceph-fuse hide ceph.* xattrs from listxattr(2)
syscall. these xattrs still can be accessed by their names. such as
getfattr -n ceph.dir.rctime /mnt/cephfs/hpc/home
> Thanks!
>
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:55 p.m.
New subject: ORe: Re: getfattr problem on ceph-fs
Hi,
On Tue, 10 Dec 2019 12:40:35 +0000, Frank Schilder wrote:
Thanks for the fast answer!
Is there any (other) way to get a complete list of extended attributes?
Not at the moment, as far as I can tell.
Is there something documented - meaning what can I
rely on in the future?
When the listxattr() entries were removed, Greg proposed that we could
add a "ceph" namespaced xattr to provide discoverability:
https://www.spinics.net/lists/ceph-devel/msg46220.html
Somebody just needs to do the work to implement this :-)
Cheers, David
11 Dec
11 Dec
8:04 a.m.
Hi David,
thanks for the link, I understand the problem now.
I don't think this can be solved within ceph alone in a good way. There are other
storage system that also use file system meta objects like .snap dirs and automatic
xattribs for administrative purposes. All these are non-copy objects (dirs and attribs)
and it almost sounds like this should become an extension of the respective APIs rather
than workarounds on the storage system level.
For example, it would also be nice if the .snap dirs (containing actual snaps) were
discoverable directly on a file system with a tool like find but protected from copy. The
MDS now has functions for showing such info, but holding a parallel data structure
somewhere else than the mount point looks more like a workaround than an elegant
integration and requires cluster admin access to answer a rather simple question a
non-privileged user could answer herself.
In the mean time, my attempt of a complete list:
ceph.dir.entries
ceph.dir.files
ceph.dir.rbytes
ceph.dir.rctime
ceph.dir.rentries
ceph.dir.rfiles
ceph.dir.rsubdirs
ceph.dir.subdirs
ceph.dir.layout
ceph.file.layout
ceph.quota.max_bytes
ceph.quota.max_files
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: David Disseldorp <ddiss(a)suse.de>
Sent: 10 December 2019 15:55:07
To: Frank Schilder
Cc: Yan, Zheng; ceph-users
Subject: ORe: [ceph-users] Re: getfattr problem on ceph-fs
Hi,
On Tue, 10 Dec 2019 12:40:35 +0000, Frank Schilder wrote:
Thanks for the fast answer!
Is there any (other) way to get a complete list of extended attributes?
Not at the moment, as far as I can tell.
Is there something documented - meaning what can I
rely on in the future?
When the listxattr() entries were removed, Greg proposed that we could
add a "ceph" namespaced xattr to provide discoverability:
https://www.spinics.net/lists/ceph-devel/msg46220.html
Somebody just needs to do the work to implement this :-)
Cheers, David
1620
days inactive
1621
days old
4 comments
3 participants
participants (3)
-
David Disseldorp -
Frank Schilder -
Yan, Zheng