mount issues with rbd running xfs - Structure needs cleaning
Hello All, One of the use cases (e.g. machine learning workloads) for RBD volumes in our production environment is that, users could mount an RBD volume in RW mode in a container, write some data to it and later use the same volume in RO mode into a number of containers in parallel to consume the data. I am trying to test this scenario with different file systems (ext3/4 and xfs). I have an automated test code that creates a volume, maps it to a node, mounts in RW mode and write some data into it. Later the same volume is mounted in RO mode in a number of other nodes and a process reads from the file. I dont see any issues with ext3 or 4 filesystems, but with XFS, I notice that 1 or 2 (out of 6) parallel read-only mounts fail with "Structure needs cleaning" error. What is surprising is that, the rest of 4 or 5 mounts will be successful and I dont see any I/O issues on those - which suggests that there shouldn't be any corruptions on the volume itself. Also note that there is no other process writing to the volume at this time so no chance of corruption that way. I am doing xfs mounts with "ro,nouuid" mount options. Any inputs on why I may be seeing this issue randomly? Regards, Shridhar
I'm pretty sure to XFS, "read-only" is not quite "read-only." My understanding is that XFS replays the journal on mount, unless it is also mounted with norecovery. -- Adam On Sun, May 3, 2020, 22:14 Void Star Nill <void.star.nill@gmail.com> wrote:
Hello All,
One of the use cases (e.g. machine learning workloads) for RBD volumes in our production environment is that, users could mount an RBD volume in RW mode in a container, write some data to it and later use the same volume in RO mode into a number of containers in parallel to consume the data.
I am trying to test this scenario with different file systems (ext3/4 and xfs). I have an automated test code that creates a volume, maps it to a node, mounts in RW mode and write some data into it. Later the same volume is mounted in RO mode in a number of other nodes and a process reads from the file.
I dont see any issues with ext3 or 4 filesystems, but with XFS, I notice that 1 or 2 (out of 6) parallel read-only mounts fail with "Structure needs cleaning" error. What is surprising is that, the rest of 4 or 5 mounts will be successful and I dont see any I/O issues on those - which suggests that there shouldn't be any corruptions on the volume itself. Also note that there is no other process writing to the volume at this time so no chance of corruption that way.
I am doing xfs mounts with "ro,nouuid" mount options.
Any inputs on why I may be seeing this issue randomly?
Regards, Shridhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Brad, Adam, Thanks for the quick responses. I am not passing any arguments other than "ro,nouuid" on mount. One thing I forgot to mention is that, there could be more than one mount of the same volume on a host - I dont know how this plays out for xfs. Appreciate your inputs. Regards, Shridhar On Sun, 3 May 2020 at 21:43, <brad.swanson@adtran.com> wrote:
Are you mounting the RO with noatime? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den mån 4 maj 2020 kl 05:14 skrev Void Star Nill <void.star.nill@gmail.com>:
One of the use cases (e.g. machine learning workloads) for RBD volumes in our production environment is that, users could mount an RBD volume in RW mode in a container, write some data to it and later use the same volume in RO mode into a number of containers in parallel to consume the data.
I am trying to test this scenario with different file systems (ext3/4 and xfs). I have an automated test code that creates a volume, maps it to a node, mounts in RW mode and write some data into it. Later the same volume is mounted in RO mode in a number of other nodes and a process reads from the file.
Is the RW unmounted or not? You write "stopped writing" but that doesn't clearly indicate if you make it impossible or just "I ask it to not make much IO". Given that many filesystems are doing very lazy writes, caches a lot and so on, it would be very important to make sure 1) ALL writes are done, which is easiest with umount I think and 2) that mounting clients knows can't write to it at all, or otherwise as someone said, it might still be updating some metainfo like the journals or "last mounted on /X" or whatever magic fs's store even while not altering the files inside the fs. It's kind of hard to tell filesystems that are accustomed to being in charge of all mounted instances to sit in the back seat and not be allowed to control stuff. -- May the most significant bit of your life be positive.
Thanks Janne. I actually meant that the RW mount is unmounted already - sorry about the confusion. - Shridhar On Mon, 4 May 2020 at 00:35, Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 4 maj 2020 kl 05:14 skrev Void Star Nill <void.star.nill@gmail.com
:
One of the use cases (e.g. machine learning workloads) for RBD volumes in our production environment is that, users could mount an RBD volume in RW mode in a container, write some data to it and later use the same volume in RO mode into a number of containers in parallel to consume the data.
I am trying to test this scenario with different file systems (ext3/4 and xfs). I have an automated test code that creates a volume, maps it to a node, mounts in RW mode and write some data into it. Later the same volume is mounted in RO mode in a number of other nodes and a process reads from the file.
Is the RW unmounted or not? You write "stopped writing" but that doesn't clearly indicate if you make it impossible or just "I ask it to not make much IO". Given that many filesystems are doing very lazy writes, caches a lot and so on, it would be very important to make sure 1) ALL writes are done, which is easiest with umount I think and 2) that mounting clients knows can't write to it at all, or otherwise as someone said, it might still be updating some metainfo like the journals or "last mounted on /X" or whatever magic fs's store even while not altering the files inside the fs.
It's kind of hard to tell filesystems that are accustomed to being in charge of all mounted instances to sit in the back seat and not be allowed to control stuff.
-- May the most significant bit of your life be positive.
Yeah, file systems rarely really do a read-only mount without providing some very obscure options, no idea about xfs specifically. Suggestion: use a keyring with profile rbd-read-only to ensure that it definitely can't write when mapping the rbd. xfs might just do the right thing automatically when encountering a read-only block device Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, May 4, 2020 at 7:05 PM Void Star Nill <void.star.nill@gmail.com> wrote:
Thanks Janne. I actually meant that the RW mount is unmounted already - sorry about the confusion.
- Shridhar
On Mon, 4 May 2020 at 00:35, Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 4 maj 2020 kl 05:14 skrev Void Star Nill < void.star.nill@gmail.com
:
One of the use cases (e.g. machine learning workloads) for RBD volumes in our production environment is that, users could mount an RBD volume in RW mode in a container, write some data to it and later use the same volume in RO mode into a number of containers in parallel to consume the data.
I am trying to test this scenario with different file systems (ext3/4 and xfs). I have an automated test code that creates a volume, maps it to a node, mounts in RW mode and write some data into it. Later the same volume is mounted in RO mode in a number of other nodes and a process reads from the file.
Is the RW unmounted or not? You write "stopped writing" but that doesn't clearly indicate if you make it impossible or just "I ask it to not make much IO". Given that many filesystems are doing very lazy writes, caches a lot and so on, it would be very important to make sure 1) ALL writes are done, which is easiest with umount I think and 2) that mounting clients knows can't write to it at all, or otherwise as someone said, it might still be updating some metainfo like the journals or "last mounted on /X" or whatever magic fs's store even while not altering the files inside the fs.
It's kind of hard to tell filesystems that are accustomed to being in charge of all mounted instances to sit in the back seat and not be allowed to control stuff.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Using "profile rbd-read-only" with krbd wouldn't work unless you are on kernel 5.5 or later. Prior to 5.5, "rbd map" code in the kernel did some things that are incompatible with "profile rbd-read-only", such as establishing a watch on the image header and more. This was overlooked because it is sufficient to map with "-o ro" to get a read-only block device. "profile rbd-read-only" just provides an extra assurance on the OSD side and helps with user management. Thanks, Ilya On Mon, May 4, 2020 at 7:56 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Yeah, file systems rarely really do a read-only mount without providing some very obscure options, no idea about xfs specifically.
Suggestion: use a keyring with profile rbd-read-only to ensure that it definitely can't write when mapping the rbd. xfs might just do the right thing automatically when encountering a read-only block device
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
On Mon, May 4, 2020 at 7:05 PM Void Star Nill <void.star.nill@gmail.com> wrote:
Thanks Janne. I actually meant that the RW mount is unmounted already - sorry about the confusion.
- Shridhar
On Mon, 4 May 2020 at 00:35, Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 4 maj 2020 kl 05:14 skrev Void Star Nill < void.star.nill@gmail.com
:
One of the use cases (e.g. machine learning workloads) for RBD volumes in our production environment is that, users could mount an RBD volume in RW mode in a container, write some data to it and later use the same volume in RO mode into a number of containers in parallel to consume the data.
I am trying to test this scenario with different file systems (ext3/4 and xfs). I have an automated test code that creates a volume, maps it to a node, mounts in RW mode and write some data into it. Later the same volume is mounted in RO mode in a number of other nodes and a process reads from the file.
Is the RW unmounted or not? You write "stopped writing" but that doesn't clearly indicate if you make it impossible or just "I ask it to not make much IO". Given that many filesystems are doing very lazy writes, caches a lot and so on, it would be very important to make sure 1) ALL writes are done, which is easiest with umount I think and 2) that mounting clients knows can't write to it at all, or otherwise as someone said, it might still be updating some metainfo like the journals or "last mounted on /X" or whatever magic fs's store even while not altering the files inside the fs.
It's kind of hard to tell filesystems that are accustomed to being in charge of all mounted instances to sit in the back seat and not be allowed to control stuff.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Adam Tygart
-
brad.swanson＠adtran.com
-
Ilya Dryomov
-
Janne Johansson
-
Paul Emmerich
-
Void Star Nill