Re: 17.2.6 fs 'ls' ok, but 'cat' 'operation not permitted' puzzle
This problem of inaccessible file systems post upgrade by other than client.admin date back from v14 carries on through v17. It also applies to any case of specifying other than the default pool names for new file systems. Solved because Curt remembered link on this list. (Thanks Curt!) Here's what the official ceph docs ought have provided, for others who hit this. YMMV: IF you have ceph file systems which have data and meta data pools that were specified in the 'ceph fs new' command (meaning not left to the defaults which create them for you), OR you have an existing ceph file system and are upgrading to a new major version of ceph THEN for the documented 'ceph fs authorize...' commands to do as documented (and avoid strange 'operation not permitted' errors when doing file I/O or similar security related problems for all but such as the client.admin user), you must first run: ceph osd pool application set <your metadata pool name> cephfs metadata <your ceph fs filesystem name> and ceph osd pool application set <your data pool name> cephfs data <your ceph fs filesystem name> Otherwise when the OSD's get a request to read or write data (not the directory info, but file data) they won't know which ceph file system name to look up, nevermind the names you may have chosen for the pools, as the 'defaults' themselves changed in the major releases, from data pool=fsname metadata pool=fsname_metadata to data pool=fsname.data and metadata pool=fsname.meta as the ceph revisions came and went. Any setup that just used 'client.admin' for all mounts didn't see the problem as the admin key gave blanket permission. A temporary 'fix' is to change mount requests to the 'client.admin' and associated key. A less drastic but still half-fix is to change the osd cap for your user to just 'caps osd = "allow rw" and delete "tag cephfs data=...." The only documentation I could find for this upgrade security-related ceph-ending catastrophe was in the NFS, not cephfs docs: https://docs.ceph.com/en/latest/cephfs/nfs/ and the Genius level much appreciated pointer from Curt here: On 5/2/23 14:21, Curt wrote:
This thread might be of use, it's an older version of ceph 14, but might still apply, https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDVM... ?
On Tue, May 2, 2023 at 11:06 PM Harry G Coin <hgcoin@gmail.com> wrote:
In 17.2.6 is there a security requirement that pool names supporting a ceph fs filesystem match the filesystem name.data for the data and name.meta for the associated metadata pool? (multiple file systems are enabled)
I have filesystems from older versions with the data pool name matching the filesystem and appending _metadata for that,
and even older filesystems with the pool name as in 'library' and 'library_metadata' supporting a filesystem called 'libraryfs'
The pools all have the cephfs tag.
But using the documented:
ceph fs authorize libraryfs client.basicuser / rw
command allows the root user to mount and browse the library directory tree, but fails with 'operation not permitted' when even reading any file.
However, changing the client.basicuser osd auth to 'allow rw' instead of 'allow rw tag...' allows normal operations.
So:
[client.basicuser] key = <key stuff>== caps mds = "allow rw fsname=libraryfs" caps mon = "allow r fsname=libraryfs" caps osd = "allow rw"
works, but the same with
caps osd = "allow rw tag cephfs data=libraryfs"
leads to the 'operation not permitted' on read, or write or any actual access.
It remains a puzzle. Help appreciated!
Were there upgrade instructions about that, any help pointing me to them?
Thanks
Harry Coin Rock Stable Systems
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, we had the NFS discussion a few weeks back [2] and at the Cephalocon I talked to Zac about it. @Zac: seems like not only NFS over CephFS is affected but CephFS in general. Could you add that note about the application metadata to the general CephFS docs as well? Thanks, Eugen [2] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2NL2Q57HTSGD... Zitat von Harry G Coin <hgcoin@gmail.com>:
This problem of inaccessible file systems post upgrade by other than client.admin date back from v14 carries on through v17. It also applies to any case of specifying other than the default pool names for new file systems. Solved because Curt remembered link on this list. (Thanks Curt!) Here's what the official ceph docs ought have provided, for others who hit this. YMMV:
IF
you have ceph file systems which have data and meta data pools that were specified in the 'ceph fs new' command (meaning not left to the defaults which create them for you),
OR
you have an existing ceph file system and are upgrading to a new major version of ceph
THEN
for the documented 'ceph fs authorize...' commands to do as documented (and avoid strange 'operation not permitted' errors when doing file I/O or similar security related problems for all but such as the client.admin user), you must first run:
ceph osd pool application set <your metadata pool name> cephfs metadata <your ceph fs filesystem name>
and
ceph osd pool application set <your data pool name> cephfs data <your ceph fs filesystem name>
Otherwise when the OSD's get a request to read or write data (not the directory info, but file data) they won't know which ceph file system name to look up, nevermind the names you may have chosen for the pools, as the 'defaults' themselves changed in the major releases, from
data pool=fsname metadata pool=fsname_metadata
to
data pool=fsname.data and metadata pool=fsname.meta
as the ceph revisions came and went. Any setup that just used 'client.admin' for all mounts didn't see the problem as the admin key gave blanket permission.
A temporary 'fix' is to change mount requests to the 'client.admin' and associated key. A less drastic but still half-fix is to change the osd cap for your user to just 'caps osd = "allow rw" and delete "tag cephfs data=...."
The only documentation I could find for this upgrade security-related ceph-ending catastrophe was in the NFS, not cephfs docs:
https://docs.ceph.com/en/latest/cephfs/nfs/
and the Genius level much appreciated pointer from Curt here:
On 5/2/23 14:21, Curt wrote:
This thread might be of use, it's an older version of ceph 14, but might still apply, https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDVM... ?
On Tue, May 2, 2023 at 11:06 PM Harry G Coin <hgcoin@gmail.com> wrote:
In 17.2.6 is there a security requirement that pool names supporting a ceph fs filesystem match the filesystem name.data for the data and name.meta for the associated metadata pool? (multiple file systems are enabled)
I have filesystems from older versions with the data pool name matching the filesystem and appending _metadata for that,
and even older filesystems with the pool name as in 'library' and 'library_metadata' supporting a filesystem called 'libraryfs'
The pools all have the cephfs tag.
But using the documented:
ceph fs authorize libraryfs client.basicuser / rw
command allows the root user to mount and browse the library directory tree, but fails with 'operation not permitted' when even reading any file.
However, changing the client.basicuser osd auth to 'allow rw' instead of 'allow rw tag...' allows normal operations.
So:
[client.basicuser] key = <key stuff>== caps mds = "allow rw fsname=libraryfs" caps mon = "allow r fsname=libraryfs" caps osd = "allow rw"
works, but the same with
caps osd = "allow rw tag cephfs data=libraryfs"
leads to the 'operation not permitted' on read, or write or any actual access.
It remains a puzzle. Help appreciated!
Were there upgrade instructions about that, any help pointing me to them?
Thanks
Harry Coin Rock Stable Systems
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eugen, TL;DR: https://github.com/ceph/ceph/pull/51359 Thanks, Eugen. Thanks, Harry G Coin. Longer version: Thanks for bringing this to my attention. I've added Harry G Coin's excellent procedure to the Troubleshooting page in the CephFS documentation. The PR that contains the commit that targets the main branch is here: https://github.com/ceph/ceph/pull/51359 The Troubleshooting page might not be the permanent location for this procedure (and, to be honest, it seems to me that maybe using documentation to solve this kind of problem would possibly not the first choice of the informed system operator), but I wanted to get this into the documentation as soon as possible. I've added Venky as a reviewer of PR#51359 and I've CCed him in this email. His review should guard against any accidental introduction of instructions that might lead readers to engage in unacceptable practices. Zac Dover Upstream Docs Ceph Foundation ------- Original Message ------- On Wednesday, May 3rd, 2023 at 5:32 PM, Eugen Block <eblock@nde.ag> wrote:
Hi,
we had the NFS discussion a few weeks back [2] and at the Cephalocon I talked to Zac about it.
@Zac: seems like not only NFS over CephFS is affected but CephFS in general. Could you add that note about the application metadata to the general CephFS docs as well?
Thanks, Eugen
[2] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2NL2Q57HTSGD...
Zitat von Harry G Coin hgcoin@gmail.com:
This problem of inaccessible file systems post upgrade by other than client.admin date back from v14 carries on through v17. It also applies to any case of specifying other than the default pool names for new file systems. Solved because Curt remembered link on this list. (Thanks Curt!) Here's what the official ceph docs ought have provided, for others who hit this. YMMV:
IF
you have ceph file systems which have data and meta data pools that were specified in the 'ceph fs new' command (meaning not left to the defaults which create them for you),
OR
you have an existing ceph file system and are upgrading to a new major version of ceph
THEN
for the documented 'ceph fs authorize...' commands to do as documented (and avoid strange 'operation not permitted' errors when doing file I/O or similar security related problems for all but such as the client.admin user), you must first run:
ceph osd pool application set <your metadata pool name> cephfs metadata <your ceph fs filesystem name>
and
ceph osd pool application set <your data pool name> cephfs data <your ceph fs filesystem name>
Otherwise when the OSD's get a request to read or write data (not the directory info, but file data) they won't know which ceph file system name to look up, nevermind the names you may have chosen for the pools, as the 'defaults' themselves changed in the major releases, from
data pool=fsname metadata pool=fsname_metadata
to
data pool=fsname.data and metadata pool=fsname.meta
as the ceph revisions came and went. Any setup that just used 'client.admin' for all mounts didn't see the problem as the admin key gave blanket permission.
A temporary 'fix' is to change mount requests to the 'client.admin' and associated key. A less drastic but still half-fix is to change the osd cap for your user to just 'caps osd = "allow rw" and delete "tag cephfs data=...."
The only documentation I could find for this upgrade security-related ceph-ending catastrophe was in the NFS, not cephfs docs:
https://docs.ceph.com/en/latest/cephfs/nfs/
and the Genius level much appreciated pointer from Curt here:
On 5/2/23 14:21, Curt wrote:
This thread might be of use, it's an older version of ceph 14, but might still apply, https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDVM... ?
On Tue, May 2, 2023 at 11:06 PM Harry G Coin hgcoin@gmail.com wrote:
In 17.2.6 is there a security requirement that pool names supporting a ceph fs filesystem match the filesystem name.data for the data and name.meta for the associated metadata pool? (multiple file systems are enabled)
I have filesystems from older versions with the data pool name matching the filesystem and appending _metadata for that,
and even older filesystems with the pool name as in 'library' and 'library_metadata' supporting a filesystem called 'libraryfs'
The pools all have the cephfs tag.
But using the documented:
ceph fs authorize libraryfs client.basicuser / rw
command allows the root user to mount and browse the library directory tree, but fails with 'operation not permitted' when even reading any file.
However, changing the client.basicuser osd auth to 'allow rw' instead of 'allow rw tag...' allows normal operations.
So:
[client.basicuser] key = <key stuff>== caps mds = "allow rw fsname=libraryfs" caps mon = "allow r fsname=libraryfs" caps osd = "allow rw"
works, but the same with
caps osd = "allow rw tag cephfs data=libraryfs"
leads to the 'operation not permitted' on read, or write or any actual access.
It remains a puzzle. Help appreciated!
Were there upgrade instructions about that, any help pointing me to them?
Thanks
Harry Coin Rock Stable Systems
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Harry G Coin
-
Zac Dover