16 Jan
2023
16 Jan
'23
3:26 a.m.
Good morning everyone.
On this Thursday night we went through an accident, where they accidentally renamed the
.data pool of a File System making it instantly inaccessible, when renaming it again to
the correct name it was possible to mount and list the files, but could not read or write.
When trying to write, the FS returned as Read Only, when trying to read it returned
Operation not allowed.
After a period of breaking my head I tried to mount with the ADMIN user and everything
worked correctly.
I tried to remove the authentication of the current user through `ceph auth rm`, I created
a new user through `ceph fs authorize <fs_name> client.<user> / rw` and it
continued the same way, I also tried to recreate it through `ceph auth get-or-create` and
nothing different happened, it stayed exactly the same.
After setting `allow *` in mon, mds and osd I was able to mount, read and write again with
the new user.
I can understand why the File System stopped after renaming the pool, what I don't
understand is why users are unable to perform operations on FS even with RW or any other
user created.
What could have happened behind the scenes to not be able to perform IO even with the
correct permissions? Or did I apply incorrect permissions that caused this problem?
Right now everything is working, I would really like to understand what happened, because
I didn't find anything documented about this type of incident.
19 Jan
19 Jan
4 a.m.
Does anyone know what could have happened?
Em seg., 16 de jan. de 2023 às 13:44, <murilo(a)evocorp.com.br> escreveu:
Good morning everyone.
On this Thursday night we went through an accident, where they
accidentally renamed the .data pool of a File System making it instantly
inaccessible, when renaming it again to the correct name it was possible to
mount and list the files, but could not read or write. When trying to
write, the FS returned as Read Only, when trying to read it returned
Operation not allowed.
After a period of breaking my head I tried to mount with the ADMIN user
and everything worked correctly.
I tried to remove the authentication of the current user through `ceph
auth rm`, I created a new user through `ceph fs authorize <fs_name>
client.<user> / rw` and it continued the same way, I also tried to recreate
it through `ceph auth get-or-create` and nothing different happened, it
stayed exactly the same.
After setting `allow *` in mon, mds and osd I was able to mount, read and
write again with the new user.
I can understand why the File System stopped after renaming the pool, what
I don't understand is why users are unable to perform operations on FS even
with RW or any other user created.
What could have happened behind the scenes to not be able to perform IO
even with the correct permissions? Or did I apply incorrect permissions
that caused this problem?
Right now everything is working, I would really like to understand what
happened, because I didn't find anything documented about this type of
incident.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
22 Feb
22 Feb
8:47 a.m.
On Mon, Jan 16, 2023 at 11:43 AM <murilo(a)evocorp.com.br> wrote:
Good morning everyone.
On this Thursday night we went through an accident, where they accidentally renamed the
.data pool of a File System making it instantly inaccessible, when renaming it again to
the correct name it was possible to mount and list the files, but could not read or write.
When trying to write, the FS returned as Read Only, when trying to read it returned
Operation not allowed.
This should only happen if the osd caps for the credential are like:
allow rw pool=<cephfs data pool>
In general, we recommend for cephfs clients to have osd caps like:
allow rw tag cephfs data=<fs name>
The pool name could therefore change without affecting clients.
After a period of breaking my head I tried to mount
with the ADMIN user and everything worked correctly.
I tried to remove the authentication of the current user through `ceph auth rm`, I
created a new user through `ceph fs authorize <fs_name> client.<user> / rw`
and it continued the same way, I also tried to recreate it through `ceph auth
get-or-create` and nothing different happened, it stayed exactly the same.
After setting `allow *` in mon, mds and osd I was able to mount, read and write again
with the new user.
I can understand why the File System stopped after renaming the pool, what I don't
understand is why users are unable to perform operations on FS even with RW or any other
user created.
What could have happened behind the scenes to not be able to perform IO even with the
correct permissions? Or did I apply incorrect permissions that caused this problem?
I suspect you didn't recreate the cap like you thought. Be sure to
verify with `ceph auth get` that the credential changed as expected.
FWIW, I tested that data pool renames do not break client I/O for a
cap generated with `ceph fs authorize...`. It works fine.
--
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
429
days inactive
466
days old
2 comments
3 participants
participants (3)
-
Murilo Morais -
murilo＠evocorp.com.br
-
Patrick Donnelly