4 May
2020
4 May
'20
7:43 a.m.
Hello All,
One of the use cases (e.g. machine learning workloads) for RBD volumes in
our production environment is that, users could mount an RBD volume in RW
mode in a container, write some data to it and later use the same volume in
RO mode into a number of containers in parallel to consume the data.
I am trying to test this scenario with different file systems (ext3/4 and
xfs). I have an automated test code that creates a volume, maps it to a
node, mounts in RW mode and write some data into it. Later the same volume
is mounted in RO mode in a number of other nodes and a process reads from
the file.
I dont see any issues with ext3 or 4 filesystems, but with XFS, I notice
that 1 or 2 (out of 6) parallel read-only mounts fail with "Structure needs
cleaning" error. What is surprising is that, the rest of 4 or 5 mounts will
be successful and I dont see any I/O issues on those - which suggests that
there shouldn't be any corruptions on the volume itself. Also note that
there is no other process writing to the volume at this time so no chance
of corruption that way.
I am doing xfs mounts with "ro,nouuid" mount options.
Any inputs on why I may be seeing this issue randomly?
Regards,
Shridhar
4 May
4 May
8 a.m.
I'm pretty sure to XFS, "read-only" is not quite "read-only." My
understanding is that XFS replays the journal on mount, unless it is also
mounted with norecovery.
--
Adam
On Sun, May 3, 2020, 22:14 Void Star Nill <void.star.nill(a)gmail.com> wrote:
Hello All,
One of the use cases (e.g. machine learning workloads) for RBD volumes in
our production environment is that, users could mount an RBD volume in RW
mode in a container, write some data to it and later use the same volume in
RO mode into a number of containers in parallel to consume the data.
I am trying to test this scenario with different file systems (ext3/4 and
xfs). I have an automated test code that creates a volume, maps it to a
node, mounts in RW mode and write some data into it. Later the same volume
is mounted in RO mode in a number of other nodes and a process reads from
the file.
I dont see any issues with ext3 or 4 filesystems, but with XFS, I notice
that 1 or 2 (out of 6) parallel read-only mounts fail with "Structure needs
cleaning" error. What is surprising is that, the rest of 4 or 5 mounts will
be successful and I dont see any I/O issues on those - which suggests that
there shouldn't be any corruptions on the volume itself. Also note that
there is no other process writing to the volume at this time so no chance
of corruption that way.
I am doing xfs mounts with "ro,nouuid" mount options.
Any inputs on why I may be seeing this issue randomly?
Regards,
Shridhar
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:37 a.m.
Hello Brad, Adam,
Thanks for the quick responses.
I am not passing any arguments other than "ro,nouuid" on mount.
One thing I forgot to mention is that, there could be more than one mount
of the same volume on a host - I dont know how this plays out for xfs.
Appreciate your inputs.
Regards,
Shridhar
On Sun, 3 May 2020 at 21:43, <brad.swanson(a)adtran.com> wrote:
Are you mounting the RO with noatime?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:05 p.m.
Den mån 4 maj 2020 kl 05:14 skrev Void Star Nill <void.star.nill(a)gmail.com>om>:
One of the use cases (e.g. machine learning workloads)
for RBD volumes in
our production environment is that, users could mount an RBD volume in RW
mode in a container, write some data to it and later use the same volume in
RO mode into a number of containers in parallel to consume the data.
I am trying to test this scenario with different file systems (ext3/4 and
xfs). I have an automated test code that creates a volume, maps it to a
node, mounts in RW mode and write some data into it. Later the same volume
is mounted in RO mode in a number of other nodes and a process reads from
the file.
Is the RW unmounted or not? You write "stopped writing" but that doesn't
clearly
indicate if you make it impossible or just "I ask it to not make much IO".
Given that many filesystems are doing very lazy writes, caches a lot and so
on,
it would be very important to make sure 1) ALL writes are done, which is
easiest with
umount I think and 2) that mounting clients knows can't write to it at all,
or otherwise
as someone said, it might still be updating some metainfo like the journals
or
"last mounted on /X" or whatever magic fs's store even while not altering
the files
inside the fs.
It's kind of hard to tell filesystems that are accustomed to being in
charge of all
mounted instances to sit in the back seat and not be allowed to control
stuff.
--
May the most significant bit of your life be positive.
9:33 p.m.
Thanks Janne. I actually meant that the RW mount is unmounted already -
sorry about the confusion.
- Shridhar
On Mon, 4 May 2020 at 00:35, Janne Johansson <icepic.dz(a)gmail.com> wrote:
Den mån 4 maj 2020 kl 05:14 skrev Void Star Nill
<void.star.nill(a)gmail.com
:
One of the use cases (e.g. machine learning
workloads) for RBD volumes in
our production environment is that, users could mount an RBD volume in RW
mode in a container, write some data to it and later use the same volume
in
RO mode into a number of containers in parallel to consume the data.
I am trying to test this scenario with different file systems (ext3/4 and
xfs). I have an automated test code that creates a volume, maps it to a
node, mounts in RW mode and write some data into it. Later the same volume
is mounted in RO mode in a number of other nodes and a process reads from
the file.
Is the RW unmounted or not? You write "stopped writing" but that doesn't
clearly
indicate if you make it impossible or just "I ask it to not make much IO".
Given that many filesystems are doing very lazy writes, caches a lot and
so on,
it would be very important to make sure 1) ALL writes are done, which is
easiest with
umount I think and 2) that mounting clients knows can't write to it at
all, or otherwise
as someone said, it might still be updating some metainfo like the
journals or
"last mounted on /X" or whatever magic fs's store even while not altering
the files
inside the fs.
It's kind of hard to tell filesystems that are accustomed to being in
charge of all
mounted instances to sit in the back seat and not be allowed to control
stuff.
--
May the most significant bit of your life be positive.
10:25 p.m.
Yeah, file systems rarely really do a read-only mount without providing
some very obscure options, no idea about xfs specifically.
Suggestion: use a keyring with profile rbd-read-only to ensure that it
definitely can't write when mapping the rbd. xfs might just do the right
thing automatically when encountering a read-only block device
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Mon, May 4, 2020 at 7:05 PM Void Star Nill <void.star.nill(a)gmail.com>
wrote:
Thanks Janne. I actually meant that the RW mount is
unmounted already -
sorry about the confusion.
- Shridhar
On Mon, 4 May 2020 at 00:35, Janne Johansson <icepic.dz(a)gmail.com> wrote:
in
IO".
Den mån 4 maj 2020 kl 05:14 skrev Void Star Nill
<
void.star.nill(a)gmail.com
:
> One of the use cases (e.g. machine learning workloads) for RBD volumes > our production environment is that, users
could mount an RBD volume in
RW
> mode in a container, write some data to it
and later use the same volume
> in
> RO mode into a number of containers in parallel to consume the data.
>
> I am trying to test this scenario with different file systems (ext3/4
and
> xfs). I have an automated test code that
creates a volume, maps it to a
> node, mounts in RW mode and write some data into it. Later the same
volume
> is mounted in RO mode in a number of other
nodes and a process reads
from
the
file.
Is the RW unmounted or not? You write "stopped writing" but that doesn't
clearly
indicate if you make it impossible or just "I ask it to not make much Given that many filesystems are doing very lazy
writes, caches a lot and
so on,
it would be very important to make sure 1) ALL writes are done, which is
easiest with
umount I think and 2) that mounting clients knows can't write to it at
all, or otherwise
as someone said, it might still be updating some metainfo like the
journals or
"last mounted on /X" or whatever magic fs's store even while not altering
the files
inside the fs.
It's kind of hard to tell filesystems that are accustomed to being in
charge of all
mounted instances to sit in the back seat and not be allowed to control
stuff.
--
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:01 p.m.
Using "profile rbd-read-only" with krbd wouldn't work unless you are
on kernel 5.5 or later. Prior to 5.5, "rbd map" code in the kernel
did some things that are incompatible with "profile rbd-read-only",
such as establishing a watch on the image header and more.
This was overlooked because it is sufficient to map with "-o ro" to
get a read-only block device. "profile rbd-read-only" just provides
an extra assurance on the OSD side and helps with user management.
Thanks,
Ilya
On Mon, May 4, 2020 at 7:56 PM Paul Emmerich <paul.emmerich(a)croit.io> wrote:
>
> Yeah, file systems rarely really do a read-only mount without providing
> some very obscure options, no idea about xfs specifically.
>
> Suggestion: use a keyring with profile rbd-read-only to ensure that it
> definitely can't write when mapping the rbd. xfs might just do the right
> thing automatically when encountering a read-only block device
>
>
> Paul
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
>
> On Mon, May 4, 2020 at 7:05 PM Void Star Nill <void.star.nill(a)gmail.com>
> wrote:
>
> > Thanks Janne. I actually meant that the RW mount is unmounted already -
> > sorry about the confusion.
> >
> > - Shridhar
> >
> > On Mon, 4 May 2020 at 00:35, Janne Johansson <icepic.dz(a)gmail.com> wrote:
> >
> > > Den mån 4 maj 2020 kl 05:14 skrev Void Star Nill <
> > void.star.nill(a)gmail.com
> > > >:
> > >
> > >> One of the use cases (e.g. machine learning workloads) for RBD volumes
> > in
> > >> our production environment is that, users could mount an RBD volume in
> > RW
> > >> mode in a container, write some data to it and later use the same
volume
> > >> in
> > >> RO mode into a number of containers in parallel to consume the data.
> > >>
> > >> I am trying to test this scenario with different file systems (ext3/4
> > and
> > >> xfs). I have an automated test code that creates a volume, maps it to
a
> > >> node, mounts in RW mode and write some data into it. Later the same
> > volume
> > >> is mounted in RO mode in a number of other nodes and a process reads
> > from
> > >> the file.
> > >>
> > >
> > > Is the RW unmounted or not? You write "stopped writing" but that
doesn't
> > > clearly
> > > indicate if you make it impossible or just "I ask it to not make much
> > IO".
> > > Given that many filesystems are doing very lazy writes, caches a lot and
> > > so on,
> > > it would be very important to make sure 1) ALL writes are done, which is
> > > easiest with
> > > umount I think and 2) that mounting clients knows can't write to it at
> > > all, or otherwise
> > > as someone said, it might still be updating some metainfo like the
> > > journals or
> > > "last mounted on /X" or whatever magic fs's store even while
not altering
> > > the files
> > > inside the fs.
> > >
> > > It's kind of hard to tell filesystems that are accustomed to being in
> > > charge of all
> > > mounted instances to sit in the back seat and not be allowed to control
> > > stuff.
> > >
> > > --
> > > May the most significant bit of your life be positive.
> > >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1454
days inactive
1454
days old
7 comments
6 participants
participants (6)
-
Adam Tygart -
brad.swanson＠adtran.com -
Ilya Dryomov -
Janne Johansson -
Paul Emmerich -
Void Star Nill