OverlayFS with Cephfs to mount a snapshot read/write
Hello, I would like to use a cephfs snapshot as a read/write volume without having to clone it first as the cloning operation is - if I'm not mistaken - still inefficient as of now. This is for a data restore use case with Moodle application needing a writable data directory to start. The idea that came to mind was to use overlayFS with cephfs set up as a read-only lower layer and a writable local directory set up as an upper layer. With this set up, any modifications to the read-only .snap/testsnap directory would normally go to the upper directory making the snapshot directory somehow writable to the Moodle application. While this works fine when a local read-only filesystem is set up as the lower layer, it fails when cephfs is set up as the lower layer. Any modifications to the .snap/testsnap tree in the /cephfs-snap directory fails with an "Operation not supported". $ mkdir /cephfs /upperdir /workdir /cephfs-snap $ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o name=admin,secretfile=/etc/ceph/admin.secret $ mount -t overlay overlay -o redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir /cephfs-snap $ ls /cephfs-snap usr $ touch /cephfs-snap/foo.txt <---- writing outside the lowerdir succeeds $ ls /cephfs-snap foo.txt usr $ ls /usr/etc $ touch /cephfs-snap/usr/etc/foo <---- writing inside the lowerdir fails touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non supportée I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help either. There's been a recent discussion about this here [2] between Greg and Robert but with no real solution. Did someone manage to do this? Regards, Frédéric. [1] https://blog.fai-project.org/posts/overlayfs/ [2] https://tracker.ceph.com/issues/44821
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hello,
I would like to use a cephfs snapshot as a read/write volume without having to clone it first as the cloning operation is - if I'm not mistaken - still inefficient as of now. This is for a data restore use case with Moodle application needing a writable data directory to start.
The idea that came to mind was to use overlayFS with cephfs set up as a read-only lower layer and a writable local directory set up as an upper layer. With this set up, any modifications to the read-only .snap/testsnap directory would normally go to the upper directory making the snapshot directory somehow writable to the Moodle application. While this works fine when a local read-only filesystem is set up as the lower layer, it fails when cephfs is set up as the lower layer. Any modifications to the .snap/testsnap tree in the /cephfs-snap directory fails with an "Operation not supported".
$ mkdir /cephfs /upperdir /workdir /cephfs-snap
$ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o name=admin,secretfile=/etc/ceph/admin.secret
$ mount -t overlay overlay -o redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir /cephfs-snap
$ ls /cephfs-snap usr
$ touch /cephfs-snap/foo.txt <---- writing outside the lowerdir succeeds
$ ls /cephfs-snap foo.txt usr
$ ls /usr/etc
$ touch /cephfs-snap/usr/etc/foo <---- writing inside the lowerdir fails touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non supportée
I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help either. There's been a recent discussion about this here [2] between Greg and Robert but with no real solution.
I just commented on that bug tracker and, although I'm not really 100% sure, I suspect that the tmpfs on that system has been compiled without xattr support.
Did someone manage to do this?
I couldn't reproduce you're problem. Is it possible that your upper dir doesn't support xatttrs either? Also, kernel client details would help. Cheers, -- Luis
Regards,
Frédéric.
[1] https://blog.fai-project.org/posts/overlayfs/ [2] https://tracker.ceph.com/issues/44821 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Luis, Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9. ~/ uname -r 3.10.0-1160.2.2.el7.x86_64 ~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y upper directory /upperdir is using xattrs ~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0 ~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr ~/ setfattr -n user.name -v upperdir /upperdir ~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir" Are you able to modify the content of a snapshot directory using overlayfs on your side? Frédéric. Le 09/11/2020 à 12:39, Luis Henriques a écrit :
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hello,
I would like to use a cephfs snapshot as a read/write volume without having to clone it first as the cloning operation is - if I'm not mistaken - still inefficient as of now. This is for a data restore use case with Moodle application needing a writable data directory to start.
The idea that came to mind was to use overlayFS with cephfs set up as a read-only lower layer and a writable local directory set up as an upper layer. With this set up, any modifications to the read-only .snap/testsnap directory would normally go to the upper directory making the snapshot directory somehow writable to the Moodle application. While this works fine when a local read-only filesystem is set up as the lower layer, it fails when cephfs is set up as the lower layer. Any modifications to the .snap/testsnap tree in the /cephfs-snap directory fails with an "Operation not supported".
$ mkdir /cephfs /upperdir /workdir /cephfs-snap
$ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o name=admin,secretfile=/etc/ceph/admin.secret
$ mount -t overlay overlay -o redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir /cephfs-snap
$ ls /cephfs-snap usr
$ touch /cephfs-snap/foo.txt <---- writing outside the lowerdir succeeds
$ ls /cephfs-snap foo.txt usr
$ ls /usr/etc
$ touch /cephfs-snap/usr/etc/foo <---- writing inside the lowerdir fails touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non supportée
I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help either. There's been a recent discussion about this here [2] between Greg and Robert but with no real solution. I just commented on that bug tracker and, although I'm not really 100% sure, I suspect that the tmpfs on that system has been compiled without xattr support.
Did someone manage to do this? I couldn't reproduce you're problem. Is it possible that your upper dir doesn't support xatttrs either? Also, kernel client details would help.
Cheers, -- Luis
Regards,
Frédéric.
[1] https://blog.fai-project.org/posts/overlayfs/ [2] https://tracker.ceph.com/issues/44821 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
All, I'm not sure if this is relevant here, but I recently tried to use OverlayFS with an NFS share. It wouldn't work because NFS does not present to the kernel as a block device. OverlayFS requires a block device abstraction. If CephFS doesn't present as a block device you won't get it to work. Hope this helps. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On Mon, Nov 9, 2020 at 9:08 AM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Luis,
Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9.
~/ uname -r 3.10.0-1160.2.2.el7.x86_64
~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y
upper directory /upperdir is using xattrs
~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0
~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr
~/ setfattr -n user.name -v upperdir /upperdir
~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir"
Are you able to modify the content of a snapshot directory using overlayfs on your side?
Frédéric.
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hello,
I would like to use a cephfs snapshot as a read/write volume without having to clone it first as the cloning operation is - if I'm not mistaken - still inefficient as of now. This is for a data restore use case with Moodle application needing a writable data directory to start.
The idea that came to mind was to use overlayFS with cephfs set up as a read-only lower layer and a writable local directory set up as an upper layer. With this set up, any modifications to the read-only .snap/testsnap directory would normally go to the upper directory making the snapshot
somehow writable to the Moodle application. While this works fine when a local read-only filesystem is set up as the lower layer, it fails when cephfs is set up as the lower layer. Any modifications to the .snap/testsnap tree in
Le 09/11/2020 à 12:39, Luis Henriques a écrit : directory the
/cephfs-snap directory fails with an "Operation not supported".
$ mkdir /cephfs /upperdir /workdir /cephfs-snap
$ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o name=admin,secretfile=/etc/ceph/admin.secret
$ mount -t overlay overlay -o
redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir
/cephfs-snap
$ ls /cephfs-snap usr
$ touch /cephfs-snap/foo.txt <---- writing outside the lowerdir succeeds
$ ls /cephfs-snap foo.txt usr
$ ls /usr/etc
$ touch /cephfs-snap/usr/etc/foo <---- writing inside the lowerdir fails touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non supportée
I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help either. There's been a recent discussion about this here [2] between Greg and Robert but with no real solution. I just commented on that bug tracker and, although I'm not really 100% sure, I suspect that the tmpfs on that system has been compiled without xattr support.
Did someone manage to do this? I couldn't reproduce you're problem. Is it possible that your upper dir doesn't support xatttrs either? Also, kernel client details would help.
Cheers, -- Luis
Regards,
Frédéric.
[1] https://blog.fai-project.org/posts/overlayfs/ [2] https://tracker.ceph.com/issues/44821 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hi Luis,
Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9.
~/ uname -r 3.10.0-1160.2.2.el7.x86_64
~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y
upper directory /upperdir is using xattrs
~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0
~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr
~/ setfattr -n user.name -v upperdir /upperdir
~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir"
Are you able to modify the content of a snapshot directory using overlayfs on your side?
[ Cc'ing Jeff ] Yes, I'm able to do that using a *recent* kernel. I got curious and after some digging I managed to reproduce the issue with kernel 5.3. The culprit was commit e09580b343aa ("ceph: don't list vxattrs in listxattr()"), in 5.4. Getting a bit more into the whole rabbit hole, it looks like ovl_copy_xattr() will try to copy all the ceph-related vxattrs. And that won't work (for ex. for ceph.dir.entries). Can you try cherry-picking this commit into your kernel to see if that fixes it for you? Cheers, -- Luis
Frédéric.
Le 09/11/2020 à 12:39, Luis Henriques a écrit :
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hello,
I would like to use a cephfs snapshot as a read/write volume without having to clone it first as the cloning operation is - if I'm not mistaken - still inefficient as of now. This is for a data restore use case with Moodle application needing a writable data directory to start.
The idea that came to mind was to use overlayFS with cephfs set up as a read-only lower layer and a writable local directory set up as an upper layer. With this set up, any modifications to the read-only .snap/testsnap directory would normally go to the upper directory making the snapshot directory somehow writable to the Moodle application. While this works fine when a local read-only filesystem is set up as the lower layer, it fails when cephfs is set up as the lower layer. Any modifications to the .snap/testsnap tree in the /cephfs-snap directory fails with an "Operation not supported".
$ mkdir /cephfs /upperdir /workdir /cephfs-snap
$ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o name=admin,secretfile=/etc/ceph/admin.secret
$ mount -t overlay overlay -o redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir /cephfs-snap
$ ls /cephfs-snap usr
$ touch /cephfs-snap/foo.txt <---- writing outside the lowerdir succeeds
$ ls /cephfs-snap foo.txt usr
$ ls /usr/etc
$ touch /cephfs-snap/usr/etc/foo <---- writing inside the lowerdir fails touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non supportée
I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help either. There's been a recent discussion about this here [2] between Greg and Robert but with no real solution. I just commented on that bug tracker and, although I'm not really 100% sure, I suspect that the tmpfs on that system has been compiled without xattr support.
Did someone manage to do this? I couldn't reproduce you're problem. Is it possible that your upper dir doesn't support xatttrs either? Also, kernel client details would help.
Cheers, -- Luis
Regards,
Frédéric.
[1] https://blog.fai-project.org/posts/overlayfs/ [2] https://tracker.ceph.com/issues/44821 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I feel lucky to have you on this one. ;-) Do you mean applying a specific patch on 3.10 kernel? Or is this one too old to have it working anyways. Frédéric. Le 09/11/2020 à 19:07, Luis Henriques a écrit :
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hi Luis,
Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9.
~/ uname -r 3.10.0-1160.2.2.el7.x86_64
~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y
upper directory /upperdir is using xattrs
~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0
~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr
~/ setfattr -n user.name -v upperdir /upperdir
~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir"
Are you able to modify the content of a snapshot directory using overlayfs on your side? [ Cc'ing Jeff ]
Yes, I'm able to do that using a *recent* kernel. I got curious and after some digging I managed to reproduce the issue with kernel 5.3. The culprit was commit e09580b343aa ("ceph: don't list vxattrs in listxattr()"), in 5.4.
Getting a bit more into the whole rabbit hole, it looks like ovl_copy_xattr() will try to copy all the ceph-related vxattrs. And that won't work (for ex. for ceph.dir.entries).
Can you try cherry-picking this commit into your kernel to see if that fixes it for you?
Cheers,
Updating kernel versions has improved many Ceph-related things for me. I don't use CentOS, but at a glance I see that you can get newer kernels via "Elrepo". I would seriously consider doing so. Even sticking with an LTS kernel, you can still get much newer kernels (up to 5.4.75). (https://www.kernel.org/). This is likely to make a difference to Ceph performance. On Mon, Nov 9, 2020 at 1:21 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
I feel lucky to have you on this one. ;-) Do you mean applying a specific patch on 3.10 kernel? Or is this one too old to have it working anyways.
Frédéric.
Le 09/11/2020 à 19:07, Luis Henriques a écrit :
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hi Luis,
Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9.
~/ uname -r 3.10.0-1160.2.2.el7.x86_64
~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y
upper directory /upperdir is using xattrs
~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0
~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr
~/ setfattr -n user.name -v upperdir /upperdir
~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir"
Are you able to modify the content of a snapshot directory using overlayfs on your side? [ Cc'ing Jeff ]
Yes, I'm able to do that using a *recent* kernel. I got curious and after some digging I managed to reproduce the issue with kernel 5.3. The culprit was commit e09580b343aa ("ceph: don't list vxattrs in listxattr()"), in 5.4.
Getting a bit more into the whole rabbit hole, it looks like ovl_copy_xattr() will try to copy all the ceph-related vxattrs. And that won't work (for ex. for ceph.dir.entries).
Can you try cherry-picking this commit into your kernel to see if that fixes it for you?
Cheers,
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, you'd have to apply the patch to that kernel yourself. No RHEL7 kernels have that patch (so far). Newer RHEL8 kernels _do_ if that's an option for you. -- Jeff On Mon, 2020-11-09 at 19:21 +0100, Frédéric Nass wrote:
I feel lucky to have you on this one. ;-) Do you mean applying a specific patch on 3.10 kernel? Or is this one too old to have it working anyways.
Frédéric.
Le 09/11/2020 à 19:07, Luis Henriques a écrit :
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hi Luis,
Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9.
~/ uname -r 3.10.0-1160.2.2.el7.x86_64
~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y
upper directory /upperdir is using xattrs
~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0
~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr
~/ setfattr -n user.name -v upperdir /upperdir
~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir"
Are you able to modify the content of a snapshot directory using overlayfs on your side? [ Cc'ing Jeff ]
Yes, I'm able to do that using a *recent* kernel. I got curious and after some digging I managed to reproduce the issue with kernel 5.3. The culprit was commit e09580b343aa ("ceph: don't list vxattrs in listxattr()"), in 5.4.
Getting a bit more into the whole rabbit hole, it looks like ovl_copy_xattr() will try to copy all the ceph-related vxattrs. And that won't work (for ex. for ceph.dir.entries).
Can you try cherry-picking this commit into your kernel to see if that fixes it for you?
Cheers,
-- Jeff Layton <jlayton@kernel.org>
Hi Jeff, I understand the idea behind patch [1] but it breaks the operation of overlayfs with cephfs. Should the patch be abandoned and tests be modified or should overlayfs code be adapted to work with cephfs, if that's possible? Either way, it'd be nice if overlayfs could work again with cephfs out of the box without requiring users to patch and build their own kernels past 5.4+. :-) Regards, Frédéric. [1] https://www.spinics.net/lists/ceph-devel/msg46183.html Le 09/11/2020 à 19:52, Jeff Layton a écrit :
Yes, you'd have to apply the patch to that kernel yourself. No RHEL7 kernels have that patch (so far). Newer RHEL8 kernels _do_ if that's an option for you. -- Jeff
On Mon, 2020-11-09 at 19:21 +0100, Frédéric Nass wrote:
I feel lucky to have you on this one. ;-) Do you mean applying a specific patch on 3.10 kernel? Or is this one too old to have it working anyways.
Frédéric.
Le 09/11/2020 à 19:07, Luis Henriques a écrit :
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hi Luis,
Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9.
~/ uname -r 3.10.0-1160.2.2.el7.x86_64
~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y
upper directory /upperdir is using xattrs
~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0
~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr
~/ setfattr -n user.name -v upperdir /upperdir
~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir"
Are you able to modify the content of a snapshot directory using overlayfs on your side? [ Cc'ing Jeff ]
Yes, I'm able to do that using a *recent* kernel. I got curious and after some digging I managed to reproduce the issue with kernel 5.3. The culprit was commit e09580b343aa ("ceph: don't list vxattrs in listxattr()"), in 5.4.
Getting a bit more into the whole rabbit hole, it looks like ovl_copy_xattr() will try to copy all the ceph-related vxattrs. And that won't work (for ex. for ceph.dir.entries).
Can you try cherry-picking this commit into your kernel to see if that fixes it for you?
Cheers,
Hi Jeff, I understand the idea behind patch [1] but it breaks the operation of overlayfs with cephfs. Should the patch be abandoned and tests be modified or should overlayfs code be adapted to work with cephfs, if that's possible? Either way, it'd be nice if overlayfs could work again with cephfs out of the box without requiring users to patch and build their own kernels past 5.4+. Regards, Frédéric. [1] https://www.spinics.net/lists/ceph-devel/msg46183.html (CCing Greg) PS : Please forgive me if you received this message twice. My previous message was flagged spam due to the dynamic IP address of my router that was seen in a spam campain in the past, so I sent it again. ----- Le 9 Nov 20, à 19:52, Jeff Layton jlayton@kernel.org a écrit :
Yes, you'd have to apply the patch to that kernel yourself. No RHEL7 kernels have that patch (so far). Newer RHEL8 kernels _do_ if that's an option for you. -- Jeff
On Mon, 2020-11-09 at 19:21 +0100, Frédéric Nass wrote:
I feel lucky to have you on this one. ;-) Do you mean applying a specific patch on 3.10 kernel? Or is this one too old to have it working anyways.
Frédéric.
Le 09/11/2020 à 19:07, Luis Henriques a écrit :
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hi Luis,
Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9.
~/ uname -r 3.10.0-1160.2.2.el7.x86_64
~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y
upper directory /upperdir is using xattrs
~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0
~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr
~/ setfattr -n user.name -v upperdir /upperdir
~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir"
Are you able to modify the content of a snapshot directory using overlayfs on your side? [ Cc'ing Jeff ]
Yes, I'm able to do that using a *recent* kernel. I got curious and after some digging I managed to reproduce the issue with kernel 5.3. The culprit was commit e09580b343aa ("ceph: don't list vxattrs in listxattr()"), in 5.4.
Getting a bit more into the whole rabbit hole, it looks like ovl_copy_xattr() will try to copy all the ceph-related vxattrs. And that won't work (for ex. for ceph.dir.entries).
Can you try cherry-picking this commit into your kernel to see if that fixes it for you?
Cheers,
-- Jeff Layton <jlayton@kernel.org>
Luis, I gave RHEL 8 and kernel 4.18 a try and it's working perfectly! \o/ Same commands, same mount options. Does anyone know why and if there's any chances I can have this working with CentOS/RHEL 7 and 3.10 kernel? Best regards, Frédéric. Le 09/11/2020 à 15:04, Frédéric Nass a écrit :
Hi Luis,
Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9.
~/ uname -r 3.10.0-1160.2.2.el7.x86_64
~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y
upper directory /upperdir is using xattrs
~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0
~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr
~/ setfattr -n user.name -v upperdir /upperdir
~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir"
Are you able to modify the content of a snapshot directory using overlayfs on your side?
Frédéric.
Le 09/11/2020 à 12:39, Luis Henriques a écrit :
Frédéric Nass <frederic.nass@univ-lorraine.fr> writes:
Hello,
I would like to use a cephfs snapshot as a read/write volume without having to clone it first as the cloning operation is - if I'm not mistaken - still inefficient as of now. This is for a data restore use case with Moodle application needing a writable data directory to start.
The idea that came to mind was to use overlayFS with cephfs set up as a read-only lower layer and a writable local directory set up as an upper layer. With this set up, any modifications to the read-only .snap/testsnap directory would normally go to the upper directory making the snapshot directory somehow writable to the Moodle application. While this works fine when a local read-only filesystem is set up as the lower layer, it fails when cephfs is set up as the lower layer. Any modifications to the .snap/testsnap tree in the /cephfs-snap directory fails with an "Operation not supported".
$ mkdir /cephfs /upperdir /workdir /cephfs-snap
$ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o name=admin,secretfile=/etc/ceph/admin.secret
$ mount -t overlay overlay -o redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir
/cephfs-snap
$ ls /cephfs-snap usr
$ touch /cephfs-snap/foo.txt <---- writing outside the lowerdir succeeds
$ ls /cephfs-snap foo.txt usr
$ ls /usr/etc
$ touch /cephfs-snap/usr/etc/foo <---- writing inside the lowerdir fails touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non supportée
I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help either. There's been a recent discussion about this here [2] between Greg and Robert but with no real solution. I just commented on that bug tracker and, although I'm not really 100% sure, I suspect that the tmpfs on that system has been compiled without xattr support.
Did someone manage to do this? I couldn't reproduce you're problem. Is it possible that your upper dir doesn't support xatttrs either? Also, kernel client details would help.
Cheers, -- Luis
Regards,
Frédéric.
[1] https://blog.fai-project.org/posts/overlayfs/ [2] https://tracker.ceph.com/issues/44821 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Dave Hall
-
Frédéric Nass
-
Jeff Layton
-
Luis Henriques
-
Nathan Fish