cephfs : write error: Operation not permitted
Hello, On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data " [snip] ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin $ ceph auth get client.cephfsadmin exported keyring for client.cephfsadmin [client.cephfsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs pool=cephfs_data "
I don't where to look to get more information about that issue. Anyone can help me? Thanks Best regards, -- Yoann Moulin EPFL IC-IT
Hi Yoann, On Tue, Jan 21, 2020 at 11:58 PM Yoann Moulin <yoann.moulin@epfl.ch> wrote:
Hello,
On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data " [snip] ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin $ ceph auth get client.cephfsadmin exported keyring for client.cephfsadmin [client.cephfsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs pool=cephfs_data "
This should be: caps osd = "allow rw tag cephfs data=cephfs" See also: https://docs.ceph.com/docs/nautilus/cephfs/client-auth/ -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi Yoann, for some reason, the command "ceph fs authorize" does not add the required permissions for a FS with data pools any more, older versions did. Now you need to add these caps by hand. It needs to look something like this: caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data" an easy way is: - ceph auth export - add the caps with an editor - ceph auth import I consider this a bug and thought it was fixed in newer versions already. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Yoann Moulin <yoann.moulin@epfl.ch> Sent: 22 January 2020 08:58:29 To: ceph-users Subject: [ceph-users] cephfs : write error: Operation not permitted Hello, On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data " [snip] ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin $ ceph auth get client.cephfsadmin exported keyring for client.cephfsadmin [client.cephfsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs pool=cephfs_data "
I don't where to look to get more information about that issue. Anyone can help me? Thanks Best regards, -- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry, I had a typo. If you have separate meta- and data pools, the data pool is not added properly. The caps should look like: caps osd = "allow rw tag cephfs pool=cephfs-meta-pool, allow rw pool=cephfs-data-pool" If you don't have a separate data pool, it should work out of the box. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 23 January 2020 09:10:41 To: Yoann Moulin; ceph-users Subject: [ceph-users] Re: cephfs : write error: Operation not permitted Hi Yoann, for some reason, the command "ceph fs authorize" does not add the required permissions for a FS with data pools any more, older versions did. Now you need to add these caps by hand. It needs to look something like this: caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data" an easy way is: - ceph auth export - add the caps with an editor - ceph auth import I consider this a bug and thought it was fixed in newer versions already. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Yoann Moulin <yoann.moulin@epfl.ch> Sent: 22 January 2020 08:58:29 To: ceph-users Subject: [ceph-users] cephfs : write error: Operation not permitted Hello, On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data " [snip] ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin $ ceph auth get client.cephfsadmin exported keyring for client.cephfsadmin [client.cephfsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs pool=cephfs_data "
I don't where to look to get more information about that issue. Anyone can help me? Thanks Best regards, -- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank,
for some reason, the command "ceph fs authorize" does not add the required permissions for a FS with data pools any more, older versions did. Now you need to add these caps by hand. It needs to look something like this:
caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data"
an easy way is:
- ceph auth export - add the caps with an editor - ceph auth import
I consider this a bug and thought it was fixed in newer versions already.
Sorry, I had a typo. If you have separate meta- and data pools, the data pool is not added properly. The caps should look like:
caps osd = "allow rw tag cephfs pool=cephfs-meta-pool, allow rw pool=cephfs-data-pool"
If you don't have a separate data pool, it should work out of the box.
Thanks, for the feed back, I also got that information from dirtwash on the IRC channel that fix my issue. And yes for me it's also an issue. But I have another cluster also on ubuntu 18.04 (4.15.0-74-generic) and Nautilus 14.2.6 installed one week before and it works. I did the same command on both and I don't have the same behaviour. there are few diff between the 2 clusters, not the same hw config (No SSD on the dslab2020 cluster) and cephfs_data on an 8+3 EC pool on Artemis (see the end of artemis.txt). in attachement, I put the result of the commands I did on both cluster without the same behaviors at the end. Best, Yoann
________________________________________ From: Yoann Moulin <yoann.moulin@epfl.ch> Sent: 22 January 2020 08:58:29 To: ceph-users Subject: [ceph-users] cephfs : write error: Operation not permitted
Hello,
On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data " [snip] ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin $ ceph auth get client.cephfsadmin exported keyring for client.cephfsadmin [client.cephfsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs pool=cephfs_data "
I don't where to look to get more information about that issue. Anyone can help me? Thanks
Best regards,
-- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Yoann Moulin EPFL IC-IT
You should probably enable the application "cephfs" on the fs-pools. In both your cases, the osd caps should read caps osd = "allow rw tag cephfs data=cephfs_metadata, allow rw pool=cephfs_data" This is independent of the replication type of cephfs_data. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Yoann Moulin <yoann.moulin@epfl.ch> Sent: 23 January 2020 10:38:42 To: Frank Schilder; ceph-users Subject: Re: [ceph-users] Re: cephfs : write error: Operation not permitted Hi Frank,
for some reason, the command "ceph fs authorize" does not add the required permissions for a FS with data pools any more, older versions did. Now you need to add these caps by hand. It needs to look something like this:
caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data"
an easy way is:
- ceph auth export - add the caps with an editor - ceph auth import
I consider this a bug and thought it was fixed in newer versions already.
Sorry, I had a typo. If you have separate meta- and data pools, the data pool is not added properly. The caps should look like:
caps osd = "allow rw tag cephfs pool=cephfs-meta-pool, allow rw pool=cephfs-data-pool"
If you don't have a separate data pool, it should work out of the box.
Thanks, for the feed back, I also got that information from dirtwash on the IRC channel that fix my issue. And yes for me it's also an issue. But I have another cluster also on ubuntu 18.04 (4.15.0-74-generic) and Nautilus 14.2.6 installed one week before and it works. I did the same command on both and I don't have the same behaviour. there are few diff between the 2 clusters, not the same hw config (No SSD on the dslab2020 cluster) and cephfs_data on an 8+3 EC pool on Artemis (see the end of artemis.txt). in attachement, I put the result of the commands I did on both cluster without the same behaviors at the end. Best, Yoann
________________________________________ From: Yoann Moulin <yoann.moulin@epfl.ch> Sent: 22 January 2020 08:58:29 To: ceph-users Subject: [ceph-users] cephfs : write error: Operation not permitted
Hello,
On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data " [snip] ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin $ ceph auth get client.cephfsadmin exported keyring for client.cephfsadmin [client.cephfsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs pool=cephfs_data "
I don't where to look to get more information about that issue. Anyone can help me? Thanks
Best regards,
-- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Yoann Moulin EPFL IC-IT
Le 23.01.20 à 12:50, Frank Schilder a écrit :
You should probably enable the application "cephfs" on the fs-pools.
it's already activated : pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs
In both your cases, the osd caps should read
caps osd = "allow rw tag cephfs data=cephfs_metadata, allow rw pool=cephfs_data"
Currently, I have this and it works: caps: [osd] allow class-read object_prefix rbd_children, allow rw pool=cephfs_data I opened a bug on tracker : https://tracker.ceph.com/issues/43761
This is independent of the replication type of cephfs_data.
Yup, this is what I understood. Yoann
________________________________________ From: Yoann Moulin <yoann.moulin@epfl.ch> Sent: 23 January 2020 10:38:42 To: Frank Schilder; ceph-users Subject: Re: [ceph-users] Re: cephfs : write error: Operation not permitted
Hi Frank,
for some reason, the command "ceph fs authorize" does not add the required permissions for a FS with data pools any more, older versions did. Now you need to add these caps by hand. It needs to look something like this:
caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data"
an easy way is:
- ceph auth export - add the caps with an editor - ceph auth import
I consider this a bug and thought it was fixed in newer versions already.
Sorry, I had a typo. If you have separate meta- and data pools, the data pool is not added properly. The caps should look like:
caps osd = "allow rw tag cephfs pool=cephfs-meta-pool, allow rw pool=cephfs-data-pool"
If you don't have a separate data pool, it should work out of the box.
Thanks, for the feed back, I also got that information from dirtwash on the IRC channel that fix my issue. And yes for me it's also an issue.
But I have another cluster also on ubuntu 18.04 (4.15.0-74-generic) and Nautilus 14.2.6 installed one week before and it works. I did the same command on both and I don't have the same behaviour.
there are few diff between the 2 clusters, not the same hw config (No SSD on the dslab2020 cluster) and cephfs_data on an 8+3 EC pool on Artemis (see the end of artemis.txt). in attachement, I put the result of the commands I did on both cluster without the same behaviors at the end.
Best,
Yoann
________________________________________ From: Yoann Moulin <yoann.moulin@epfl.ch> Sent: 22 January 2020 08:58:29 To: ceph-users Subject: [ceph-users] cephfs : write error: Operation not permitted
Hello,
On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data " [snip] ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin $ ceph auth get client.cephfsadmin exported keyring for client.cephfsadmin [client.cephfsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs pool=cephfs_data "
I don't where to look to get more information about that issue. Anyone can help me? Thanks
Best regards,
-- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Yoann Moulin EPFL IC-IT
-- Yoann Moulin EPFL IC-IT
On Wed, Jan 22, 2020 at 8:58 AM Yoann Moulin <yoann.moulin@epfl.ch> wrote:
Hello,
On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
For "allow rw tag cephfs data=cephfs" to work, having your pools tagged with cephfs is not enough. There should be a key-value pair in the tag metadata with the name of the filesystem as well. What is the output of: ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name | startswith("cephfs")) | .pool_name, .application_metadata' Thanks, Ilya
Dear Ilya, I had exactly the same problem with authentication of cephfs clients on a mimic-13.2.2 cluster. The key created with "ceph fs authorize ..." did not grant access to the data pool. I ended up adding "rw" access to this pool by hand. Following up on your remark about pool tags, could you please point me to any documentation about how this tagging is used and what key-value pair you are referring to? It sounds like this is the new way to go, but I cannot find anything useful about it in here: https://docs.ceph.com/docs/mimic/cephfs/client-auth/ https://docs.ceph.com/docs/mimic/rados/operations/user-management/?highlight... Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
On Fri, Jan 24, 2020 at 1:43 PM Frank Schilder <frans@dtu.dk> wrote:
Dear Ilya,
I had exactly the same problem with authentication of cephfs clients on a mimic-13.2.2 cluster. The key created with "ceph fs authorize ..." did not grant access to the data pool. I ended up adding "rw" access to this pool by hand.
Following up on your remark about pool tags, could you please point me to any documentation about how this tagging is used and what key-value pair you are referring to? It sounds like this is the new way to go, but I cannot find anything useful about it in here:
Hi Frank, This is the correct page, but this key-value pair is more or less an internal implementation detail. "ceph fs authorize" is all the users should know about, but there seems to be a bug lurking there. In general, for a cap that looks like allow <r/w/x> tag <tag name> <key>=<value> the OSD will allow <r/w/x> access to the pool iff a) the pool is tagged with <tag name> and b) the tag metadata has that <key>: <value> pair in it. In the cephfs case, the key is "data" for data pool and "metadata" for metadata pools, the value is the name of the filesystem. Thanks, Ilya
Thanks a lot! I will fix the pool meta data and clean up my keys. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Ilya Dryomov <idryomov@gmail.com> Sent: 25 January 2020 09:01 To: Frank Schilder Cc: Yoann Moulin; ceph-users Subject: Re: [ceph-users] Re: cephfs : write error: Operation not permitted On Fri, Jan 24, 2020 at 1:43 PM Frank Schilder <frans@dtu.dk> wrote:
Dear Ilya,
I had exactly the same problem with authentication of cephfs clients on a mimic-13.2.2 cluster. The key created with "ceph fs authorize ..." did not grant access to the data pool. I ended up adding "rw" access to this pool by hand.
Following up on your remark about pool tags, could you please point me to any documentation about how this tagging is used and what key-value pair you are referring to? It sounds like this is the new way to go, but I cannot find anything useful about it in here:
Hi Frank, This is the correct page, but this key-value pair is more or less an internal implementation detail. "ceph fs authorize" is all the users should know about, but there seems to be a bug lurking there. In general, for a cap that looks like allow <r/w/x> tag <tag name> <key>=<value> the OSD will allow <r/w/x> access to the pool iff a) the pool is tagged with <tag name> and b) the tag metadata has that <key>: <value> pair in it. In the cephfs case, the key is "data" for data pool and "metadata" for metadata pools, the value is the name of the filesystem. Thanks, Ilya
Le 23.01.20 à 15:51, Ilya Dryomov a écrit :
On Wed, Jan 22, 2020 at 8:58 AM Yoann Moulin <yoann.moulin@epfl.ch> wrote:
Hello,
On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
For "allow rw tag cephfs data=cephfs" to work, having your pools tagged with cephfs is not enough. There should be a key-value pair in the tag metadata with the name of the filesystem as well. What is the output of:
ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name | startswith("cephfs")) | .pool_name, .application_metadata'
But it's a fresh install, pools have been created with the ceph-ansible playbook and it used to work before, maybe something is missing in the latest release of the ceph-ansible playbook. dslab2020@icitsrv5:~$ ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data" { "cephfs": {} } "cephfs_metadata" { "cephfs": {} } Best, -- Yoann Moulin EPFL IC-IT
There are two bugs that can cause these application tags to be missing, one of them is fixed (but old pools aren't fixed automatically), the other is https://tracker.ceph.com/issues/43061 which happens if you create the cephfs pools manually. You can fix the pools like this: ceph osd pool application set <pool> cephfs <data/metadata> <cephfsname> To work with "ceph fs authorize" We automatically runs this on croit on startup on all cephfs pools to make the permissions work properly for our users. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Jan 24, 2020 at 2:11 PM Yoann Moulin <yoann.moulin@epfl.ch> wrote:
Le 23.01.20 à 15:51, Ilya Dryomov a écrit :
On Wed, Jan 22, 2020 at 8:58 AM Yoann Moulin <yoann.moulin@epfl.ch> wrote:
Hello,
On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
For "allow rw tag cephfs data=cephfs" to work, having your pools tagged with cephfs is not enough. There should be a key-value pair in the tag metadata with the name of the filesystem as well. What is the output of:
ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name | startswith("cephfs")) | .pool_name, .application_metadata'
But it's a fresh install, pools have been created with the ceph-ansible playbook and it used to work before, maybe something is missing in the latest release of the ceph-ansible playbook.
dslab2020@icitsrv5:~$ ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data" { "cephfs": {} } "cephfs_metadata" { "cephfs": {} }
Best,
-- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Jan 24, 2020 at 2:10 PM Yoann Moulin <yoann.moulin@epfl.ch> wrote:
Le 23.01.20 à 15:51, Ilya Dryomov a écrit :
On Wed, Jan 22, 2020 at 8:58 AM Yoann Moulin <yoann.moulin@epfl.ch> wrote:
Hello,
On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
For "allow rw tag cephfs data=cephfs" to work, having your pools tagged with cephfs is not enough. There should be a key-value pair in the tag metadata with the name of the filesystem as well. What is the output of:
ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name | startswith("cephfs")) | .pool_name, .application_metadata'
But it's a fresh install, pools have been created with the ceph-ansible playbook and it used to work before, maybe something is missing in the latest release of the ceph-ansible playbook.
dslab2020@icitsrv5:~$ ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data" { "cephfs": {} } "cephfs_metadata" { "cephfs": {} }
Yeah, it looks like you hit one of the bugs that Paul mentions in his reply. Running ceph osd pool application set cephfs_data cephfs data cephfs ceph osd pool application set cephfs_data cephfs metadata cephfs should fix it. Thanks, Ilya
On Sat, Jan 25, 2020 at 8:42 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Fri, Jan 24, 2020 at 2:10 PM Yoann Moulin <yoann.moulin@epfl.ch> wrote:
Le 23.01.20 à 15:51, Ilya Dryomov a écrit :
On Wed, Jan 22, 2020 at 8:58 AM Yoann Moulin <yoann.moulin@epfl.ch> wrote:
Hello,
On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
$ ceph -s cluster: id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 health: HEALTH_OK
services: mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} osd: 24 osds: 24 up (since 22h), 24 in (since 22h) rgw: 1 daemon active (iccluster043.rgw0)
data: pools: 9 pools, 568 pgs objects: 800 objects, 225 KiB usage: 24 GiB used, 87 TiB / 87 TiB avail pgs: 568 active+clean
The 2 cephfs pools:
$ ceph osd pool ls detail | grep cephfs pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
The status of the cephfs filesystem:
$ ceph fs status cephfs - 1 clients ====== +------+--------+--------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------------+---------------+-------+-------+ | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | +------+--------+--------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 4608k | 27.6T | | cephfs_data | data | 0 | 27.6T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
# mkdir folder # echo "foo" > bar -bash: echo: write error: Operation not permitted # ls -al total 4 drwxrwxrwx 1 root root 2 Jan 22 07:30 . drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. -rw-r--r-- 1 root root 0 Jan 22 07:30 bar drwxrwxrwx 1 root root 1 Jan 21 16:49 folder
# df -hT . Filesystem Type Size Used Avail Use% Mounted on 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs
I try 2 client config :
$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw [snip] $ ceph auth get client.fsadmin exported keyring for client.fsadmin [client.fsadmin] key = [snip] caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs"
For "allow rw tag cephfs data=cephfs" to work, having your pools tagged with cephfs is not enough. There should be a key-value pair in the tag metadata with the name of the filesystem as well. What is the output of:
ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name | startswith("cephfs")) | .pool_name, .application_metadata'
But it's a fresh install, pools have been created with the ceph-ansible playbook and it used to work before, maybe something is missing in the latest release of the ceph-ansible playbook.
dslab2020@icitsrv5:~$ ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data" { "cephfs": {} } "cephfs_metadata" { "cephfs": {} }
Yeah, it looks like you hit one of the bugs that Paul mentions in his reply. Running
ceph osd pool application set cephfs_data cephfs data cephfs ceph osd pool application set cephfs_data cephfs metadata cephfs
Sorry, this should be ceph osd pool application set cephfs_data cephfs data cephfs ceph osd pool application set cephfs_metadata cephfs metadata cephfs so that the json output looks like "cephfs_data" { "cephfs": { "data": "cephfs" } } "cephfs_metadata" { "cephfs": { "metadata": "cephfs" } } Thanks, Ilya
Hello,
Sorry, this should be
ceph osd pool application set cephfs_data cephfs data cephfs ceph osd pool application set cephfs_metadata cephfs metadata cephfs
so that the json output looks like
"cephfs_data" { "cephfs": { "data": "cephfs" } } "cephfs_metadata" { "cephfs": { "metadata": "cephfs" } }
Thanks a lot, that has fixed my issue! Best, -- Yoann Moulin EPFL IC-IT
participants (5)
-
Frank Schilder
-
Ilya Dryomov
-
Patrick Donnelly
-
Paul Emmerich
-
Yoann Moulin