Hi Luis, Thanks for your help. Sorry I forgot about the kernel details. This is latest RHEL 7.9. ~/ uname -r 3.10.0-1160.2.2.el7.x86_64 ~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 CONFIG_TMPFS_XATTR=y upper directory /upperdir is using xattrs ~/ ls -l /dev/mapper/vg0-racine lrwxrwxrwx 1 root root 7  6 mars   2020 /dev/mapper/vg0-racine -> ../dm-0 ~/ cat /proc/fs/ext4/dm-0/options | grep xattr user_xattr ~/ setfattr -n user.name -v upperdir /upperdir ~/ getfattr -n user.name /upperdir getfattr: Suppression des « / » en tête des chemins absolus # file: upperdir user.name="upperdir" Are you able to modify the content of a snapshot directory using overlayfs on your side? Frédéric. Le 09/11/2020 à 12:39, Luis Henriques a écrit : > Frédéric Nass &lt;frederic.nass(a)univ-lorraine.fr&gt; writes: > >> Hello, >> >> I would like to use a cephfs snapshot as a read/write volume without having to >> clone it first as the cloning operation is - if I'm not mistaken - still >> inefficient as of now. This is for a data restore use case with Moodle >> application needing a writable data directory to start. >> >> The idea that came to mind was to use overlayFS with cephfs set up as a >> read-only lower layer and a writable local directory set up as an upper >> layer. With this set up, any modifications to the read-only .snap/testsnap >> directory would normally go to the upper directory making the snapshot directory >> somehow writable to the Moodle application. While this works fine when a local >> read-only filesystem is set up as the lower layer, it fails when cephfs is set >> up as the lower layer. Any modifications to the .snap/testsnap tree in the >> /cephfs-snap directory fails with an "Operation not supported". >> >> $ mkdir /cephfs /upperdir /workdir /cephfs-snap >> >> $ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o >> name=admin,secretfile=/etc/ceph/admin.secret >> >> $ mount -t overlay overlay -o >> redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir >> /cephfs-snap >> >> $ ls /cephfs-snap >> usr >> >> $ touch /cephfs-snap/foo.txt            <---- writing outside the lowerdir >> succeeds >> >> $ ls /cephfs-snap >> foo.txt  usr >> >> $ ls /usr/etc >> >> $ touch /cephfs-snap/usr/etc/foo            <---- writing inside the lowerdir >> fails >> touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non >> supportée >> >> I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs >> (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help >> either. There's been a recent discussion about this here [2] between Greg and >> Robert but with no real solution. > I just commented on that bug tracker and, although I'm not really 100% > sure, I suspect that the tmpfs on that system has been compiled without > xattr support. > >> Did someone manage to do this? > I couldn't reproduce you're problem. Is it possible that your upper dir > doesn't support xatttrs either? Also, kernel client details would help. > > Cheers, > -- > Luis > >> Regards, >> >> Frédéric. >> >> [1] https://blog.fai-project.org/posts/overlayfs/ >> [2] https://tracker.ceph.com/issues/44821 >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Frédéric Nass &lt;frederic.nass(a)univ-lorraine.fr&gt; writes: [ Cc'ing Jeff ] Yes, I'm able to do that using a *recent* kernel. I got curious and after some digging I managed to reproduce the issue with kernel 5.3. The culprit was commit e09580b343aa ("ceph: don't list vxattrs in listxattr()"), in 5.4. Getting a bit more into the whole rabbit hole, it looks like ovl_copy_xattr() will try to copy all the ceph-related vxattrs. And that won't work (for ex. for ceph.dir.entries). Can you try cherry-picking this commit into your kernel to see if that fixes it for you? Cheers, -- Luis > Frédéric. > > Le 09/11/2020 à 12:39, Luis Henriques a écrit : >> Frédéric Nass &lt;frederic.nass(a)univ-lorraine.fr&gt; writes: >> >>> Hello, >>> >>> I would like to use a cephfs snapshot as a read/write volume without having to >>> clone it first as the cloning operation is - if I'm not mistaken - still >>> inefficient as of now. This is for a data restore use case with Moodle >>> application needing a writable data directory to start. >>> >>> The idea that came to mind was to use overlayFS with cephfs set up as a >>> read-only lower layer and a writable local directory set up as an upper >>> layer. With this set up, any modifications to the read-only .snap/testsnap >>> directory would normally go to the upper directory making the snapshot directory >>> somehow writable to the Moodle application. While this works fine when a local >>> read-only filesystem is set up as the lower layer, it fails when cephfs is set >>> up as the lower layer. Any modifications to the .snap/testsnap tree in the >>> /cephfs-snap directory fails with an "Operation not supported". >>> >>> $ mkdir /cephfs /upperdir /workdir /cephfs-snap >>> >>> $ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o >>> name=admin,secretfile=/etc/ceph/admin.secret >>> >>> $ mount -t overlay overlay -o >>> redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir >>> /cephfs-snap >>> >>> $ ls /cephfs-snap >>> usr >>> >>> $ touch /cephfs-snap/foo.txt            <---- writing outside the lowerdir >>> succeeds >>> >>> $ ls /cephfs-snap >>> foo.txt  usr >>> >>> $ ls /usr/etc >>> >>> $ touch /cephfs-snap/usr/etc/foo            <---- writing inside the lowerdir >>> fails >>> touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non >>> supportée >>> >>> I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs >>> (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help >>> either. There's been a recent discussion about this here [2] between Greg and >>> Robert but with no real solution. >> I just commented on that bug tracker and, although I'm not really 100% >> sure, I suspect that the tmpfs on that system has been compiled without >> xattr support. >> >>> Did someone manage to do this? >> I couldn't reproduce you're problem. Is it possible that your upper dir >> doesn't support xatttrs either? Also, kernel client details would help. >> >> Cheers, >> -- >> Luis >> >>> Regards, >>> >>> Frédéric. >>> >>> [1] https://blog.fai-project.org/posts/overlayfs/ >>> [2] https://tracker.ceph.com/issues/44821 >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

Updating kernel versions has improved many Ceph-related things for me. I don't use CentOS, but at a glance I see that you can get newer kernels via "Elrepo". I would seriously consider doing so. Even sticking with an LTS kernel, you can still get much newer kernels (up to 5.4.75). (https://www.kernel.org/). This is likely to make a difference to Ceph performance. On Mon, Nov 9, 2020 at 1:21 PM Frédéric Nass &lt;frederic.nass(a)univ-lorraine.fr&gt; wrote: > > I feel lucky to have you on this one. ;-) Do you mean applying a > specific patch on 3.10 kernel? Or is this one too old to have it working > anyways. > > Frédéric. > > Le 09/11/2020 à 19:07, Luis Henriques a écrit : > > Frédéric Nass &lt;frederic.nass(a)univ-lorraine.fr&gt; writes: > > > >> Hi Luis, > >> > >> Thanks for your help. Sorry I forgot about the kernel details. This is latest > >> RHEL 7.9. > >> > >> ~/ uname -r > >> 3.10.0-1160.2.2.el7.x86_64 > >> > >> ~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 > >> CONFIG_TMPFS_XATTR=y > >> > >> upper directory /upperdir is using xattrs > >> > >> ~/ ls -l /dev/mapper/vg0-racine > >> lrwxrwxrwx 1 root root 7 6 mars 2020 /dev/mapper/vg0-racine -> ../dm-0 > >> > >> ~/ cat /proc/fs/ext4/dm-0/options | grep xattr > >> user_xattr > >> > >> ~/ setfattr -n user.name -v upperdir /upperdir > >> > >> ~/ getfattr -n user.name /upperdir > >> getfattr: Suppression des « / » en tête des chemins absolus > >> # file: upperdir > >> user.name="upperdir" > >> > >> Are you able to modify the content of a snapshot directory using overlayfs on > >> your side? > > [ Cc'ing Jeff ] > > > > Yes, I'm able to do that using a *recent* kernel. I got curious and after > > some digging I managed to reproduce the issue with kernel 5.3. The > > culprit was commit e09580b343aa ("ceph: don't list vxattrs in > > listxattr()"), in 5.4. > > > > Getting a bit more into the whole rabbit hole, it looks like > > ovl_copy_xattr() will try to copy all the ceph-related vxattrs. And that > > won't work (for ex. for ceph.dir.entries). > > > > Can you try cherry-picking this commit into your kernel to see if that > > fixes it for you? > > > > Cheers, > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi Jeff, I understand the idea behind patch [1] but it breaks the operation of overlayfs with cephfs. Should the patch be abandoned and tests be modified or should overlayfs code be adapted to work with cephfs, if that's possible? Either way, it'd be nice if overlayfs could work again with cephfs out of the box without requiring users to patch and build their own kernels past 5.4+. :-) Regards, Frédéric. [1] https://www.spinics.net/lists/ceph-devel/msg46183.html Le 09/11/2020 à 19:52, Jeff Layton a écrit : > Yes, you'd have to apply the patch to that kernel yourself. No RHEL7 > kernels have that patch (so far). Newer RHEL8 kernels _do_ if that's an > option for you. > -- Jeff > > On Mon, 2020-11-09 at 19:21 +0100, Frédéric Nass wrote: >> I feel lucky to have you on this one. ;-) Do you mean applying a >> specific patch on 3.10 kernel? Or is this one too old to have it working >> anyways. >> >> Frédéric. >> >> Le 09/11/2020 à 19:07, Luis Henriques a écrit : >>> Frédéric Nass &lt;frederic.nass(a)univ-lorraine.fr&gt; writes: >>> >>>> Hi Luis, >>>> >>>> Thanks for your help. Sorry I forgot about the kernel details. This is latest >>>> RHEL 7.9. >>>> >>>> ~/ uname -r >>>> 3.10.0-1160.2.2.el7.x86_64 >>>> >>>> ~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 >>>> CONFIG_TMPFS_XATTR=y >>>> >>>> upper directory /upperdir is using xattrs >>>> >>>> ~/ ls -l /dev/mapper/vg0-racine >>>> lrwxrwxrwx 1 root root 7  6 mars   2020 /dev/mapper/vg0-racine -> ../dm-0 >>>> >>>> ~/ cat /proc/fs/ext4/dm-0/options | grep xattr >>>> user_xattr >>>> >>>> ~/ setfattr -n user.name -v upperdir /upperdir >>>> >>>> ~/ getfattr -n user.name /upperdir >>>> getfattr: Suppression des « / » en tête des chemins absolus >>>> # file: upperdir >>>> user.name="upperdir" >>>> >>>> Are you able to modify the content of a snapshot directory using overlayfs on >>>> your side? >>> [ Cc'ing Jeff ] >>> >>> Yes, I'm able to do that using a *recent* kernel. I got curious and after >>> some digging I managed to reproduce the issue with kernel 5.3. The >>> culprit was commit e09580b343aa ("ceph: don't list vxattrs in >>> listxattr()"), in 5.4. >>> >>> Getting a bit more into the whole rabbit hole, it looks like >>> ovl_copy_xattr() will try to copy all the ceph-related vxattrs. And that >>> won't work (for ex. for ceph.dir.entries). >>> >>> Can you try cherry-picking this commit into your kernel to see if that >>> fixes it for you? >>> >>> Cheers,

Luis, I gave RHEL 8 and kernel 4.18 a try and it's working perfectly! \o/ Same commands, same mount options. Does anyone know why and if there's any chances I can have this working with CentOS/RHEL 7 and 3.10 kernel? Best regards, Frédéric. Le 09/11/2020 à 15:04, Frédéric Nass a écrit : > Hi Luis, > > Thanks for your help. Sorry I forgot about the kernel details. This is > latest RHEL 7.9. > > ~/ uname -r > 3.10.0-1160.2.2.el7.x86_64 > > ~/ grep CONFIG_TMPFS_XATTR /boot/config-3.10.0-1160.2.2.el7.x86_64 > CONFIG_TMPFS_XATTR=y > > upper directory /upperdir is using xattrs > > ~/ ls -l /dev/mapper/vg0-racine > lrwxrwxrwx 1 root root 7  6 mars   2020 /dev/mapper/vg0-racine -> ../dm-0 > > ~/ cat /proc/fs/ext4/dm-0/options | grep xattr > user_xattr > > ~/ setfattr -n user.name -v upperdir /upperdir > > ~/ getfattr -n user.name /upperdir > getfattr: Suppression des « / » en tête des chemins absolus > # file: upperdir > user.name="upperdir" > > Are you able to modify the content of a snapshot directory using > overlayfs on your side? > > Frédéric. > > Le 09/11/2020 à 12:39, Luis Henriques a écrit : >> Frédéric Nass &lt;frederic.nass(a)univ-lorraine.fr&gt; writes: >> >>> Hello, >>> >>> I would like to use a cephfs snapshot as a read/write volume without >>> having to >>> clone it first as the cloning operation is - if I'm not mistaken - >>> still >>> inefficient as of now. This is for a data restore use case with Moodle >>> application needing a writable data directory to start. >>> >>> The idea that came to mind was to use overlayFS with cephfs set up as a >>> read-only lower layer and a writable local directory set up as an upper >>> layer. With this set up, any modifications to the read-only >>> .snap/testsnap >>> directory would normally go to the upper directory making the >>> snapshot directory >>> somehow writable to the Moodle application. While this works fine >>> when a local >>> read-only filesystem is set up as the lower layer, it fails when >>> cephfs is set >>> up as the lower layer. Any modifications to the .snap/testsnap tree >>> in the >>> /cephfs-snap directory fails with an "Operation not supported". >>> >>> $ mkdir /cephfs /upperdir /workdir /cephfs-snap >>> >>> $ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o >>> name=admin,secretfile=/etc/ceph/admin.secret >>> >>> $ mount -t overlay overlay -o >>> redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir >>> >>> /cephfs-snap >>> >>> $ ls /cephfs-snap >>> usr >>> >>> $ touch /cephfs-snap/foo.txt            <---- writing outside the >>> lowerdir >>> succeeds >>> >>> $ ls /cephfs-snap >>> foo.txt  usr >>> >>> $ ls /usr/etc >>> >>> $ touch /cephfs-snap/usr/etc/foo            <---- writing inside the >>> lowerdir >>> fails >>> touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: >>> Opération non >>> supportée >>> >>> I tried to mount the whole cephfs tree read-only (-o ro), tried to >>> disable ACLs >>> (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse >>> didn't help >>> either. There's been a recent discussion about this here [2] between >>> Greg and >>> Robert but with no real solution. >> I just commented on that bug tracker and, although I'm not really 100% >> sure, I suspect that the tmpfs on that system has been compiled without >> xattr support. >> >>> Did someone manage to do this? >> I couldn't reproduce you're problem.  Is it possible that your upper dir >> doesn't support xatttrs either?  Also, kernel client details would help. >> >> Cheers, >> -- >> Luis >> >>> Regards, >>> >>> Frédéric. >>> >>> [1] https://blog.fai-project.org/posts/overlayfs/ >>> [2] https://tracker.ceph.com/issues/44821 >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io