- ceph-users - lists.ceph.io

ceph fs authorize doesn't work correctly / Operation not permitted
by Rudenko Aleksandr 26 Jul '24

26 Jul '24

Hi, I have two ceph clusters on 16.2.13 and one CephFS in each. I can see only one difference between FSs: the second one FS has two data pools and one of the root directories is pinned to the second pool. In the first one FS we have only one default data pool. And, when I do: ceph fs authorize fs1 client.XXX /root-dir/some-dir rw I have next permissions in first and second FSs: [client.XXX] key = AQCmbp9myJ0pBhAAnoCBxvKbOfyGH5vs0g2QhQ== caps mds = "allow rw fsname=fs1 path=/root-dir/some-dir" caps mon = "allow r fsname=fs1" caps osd = "allow rw tag cephfs data=fs1" and zero problem in the first FS, the user can mount and write new files. But, on the second FS user can't write files. Operation not permitted. In client logs I can see: check_pool_perm on pool 15 ns need AsFw, but no write perm If I update caps in second FS like this: ceph auth caps client.XXX mds 'allow rw fsname=fs1, allow rw fsname=fs1 path=/root-dir/some-dir' mon 'allow r fsname=fs1' osd 'allow rw pool=fs1_data' and caps will looks like: [client.XXX] key = AQA2e6JmeNrWABAAoIiBstYFSr4/UZ3N1vIrHg== caps mds = "allow rw fsname=fs1, allow rw fsname=fs1 path=/root-dir/some-dir" caps mon = "allow r fsname=fs1" caps osd = "allow rw pool=fs1_data" Everything goes well. So, I have two questions: 1. Where is the problem with "allow rw tag cephfs data=fs1", how can I debug this? 2. What is the difference between "allow rw pool=fs1_data"? Can I use this caps format instead of ceph fs authorize

1 0

[RGW] radosgw does not respond after some time after upgrade from pacific to quincy
by Rok Jaklič 26 Jul '24

26 Jul '24

Hi, we've just updated from pacific(16.2.15) to quincy(17.2.7) and everything seems to work, however after some time radosgw stops responding and we have to restart it. At first look, it seems that radosgw stops responding sometimes during recovery. Does this maybe have to do something with mclock https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ ? ceph.conf looks something like this: -------- [global] fsid = ... mon initial members = x1,x2 mon host = # public network = 192.168.0.0/24 auth cluster required = none auth service required = none auth client required = none ms_mon_client_mode = crc osd journal size = 1024 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd crush chooseleaf type = 1 [osd] osd_scrub_begin_hour = 18 osd_scrub_end_hour = 6 osd_class_update_on_start = false osd_scrub_during_recovery = false #scrub during recovery osd_scrub_max_interval = 1209600 osd_deep_scrub_interval = 1209600 osd_max_scrubs = 3 osd_scrub_load_threshold = 1 [client.radosgw.mon2] host = mon2 # keyring = /etc/ceph/ceph.client.radosgw.keyring log_file = /var/log/ceph/client.radosgw.mon2.log rgw_dns_name = ... rgw_frontends = "beast port=4444" rgw_max_put_param_size = 15728640 rgw_crypt_require_ssl = false rgw_max_concurrent_requests = 2048 -------- We have nginx in front of rgw which upstream is set to client.radosgw.mon2 port 4444. Kind regards, Rok

2 3

Ceph Upgrade 18.2.2 -> 18.2.4 fails | Fatal glibc error: CPU does not support x86-64-v2 on virtualized hosts
by Björn Lässig 25 Jul '24

25 Jul '24

Hi there, Last week I upgraded from 17.2.7 to 18.2.2 and the cluster worked fine except for a few bugs about IPv6. But it is working and cluster.health is HEALTH_OK. Hoping to fix some of these minor IPv6 issues, I tried upgrading to 18.2.4. I see 2 errors: The first is that the upgrade fails to: { "target_image": "quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "0/89 daemons upgraded", "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", "is_paused": true } The error message is misleading as: # podman images --digests REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE quay.io/ceph/ceph <none> sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 2bc0b0f4375d 33 hours ago 1.25 GB […] the target image was pulled without problems. (podman pull runs without problems) The error occures while starting the image: Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.752832293 +0200 CEST m=+0.047066303 container create 4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e (image=quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906, name=reverent_hypatia, org.label-schema.build-date=20240716, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.33.8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True) Jul 25 09:16:18 cephmgr1 systemd[1]: Started libpod-conmon-4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e.scope. Jul 25 09:16:18 cephmgr1 systemd[1]: Started libpod-4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e.scope - libcrun container. Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.820823054 +0200 CEST m=+0.115057064 container init 4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e (image=quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906, name=reverent_hypatia, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.4, org.label-schema.build-date=20240716, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.33.8, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.826093069 +0200 CEST m=+0.120327079 container start 4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e (image=quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906, name=reverent_hypatia, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=-18.2.4, RELEASE=HEAD, org.label-schema.build-date=20240716, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.33.8) Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.726428757 +0200 CEST m=+0.020662767 image pull quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.82623823 +0200 CEST m=+0.120472250 container attach 4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e (image=quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906, name=reverent_hypatia, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.4, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240716, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.33.8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) Jul 25 09:16:18 cephmgr1 reverent_hypatia[1171489]: Fatal glibc error: CPU does not support x86-64-v2 Jul 25 09:16:18 cephmgr1 systemd[1]: libpod-4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e.scope: Deactivated successfully. When starting the container for 18.2.4. a glibc error occurs. 18.2.2. to 18.2.4 is a minor upgrade and should run on same hardware but this is not hardware: Both mgr and 3 of 5 monitor daemons run on virtual machines on a proxmox (qemu-based virtualizaton) which has the default CPU "kvm64". This means for the qemu options: kvm ... -smp 1,sockets=1,cores=4,maxcpus=4 \ -device kvm64-x86_64-cpu,id=cpu2,socket-id=0,core-id=1,thread-id=0\ -device kvm64-x86_64-cpu,id=cpu3,socket-id=0,core-id=2,thread-id=0\ -device kvm64-x86_64-cpu,id=cpu4,socket-id=0,core-id=3,thread-id=0 Which CPU type should I choose for my VMs for this libc? regards Björn Lässig

2 4

Subscribe
by Dan O'Brien 25 Jul '24

25 Jul '24

2 2

Release 18.2.4
by Alfredo Rezinovsky 24 Jul '24

24 Jul '24

Ceph dashboard offers me to upgrade to v18.2.4. I can't find any information on 18.2.4. The is no 18.2.4 in https://docs.ceph.com/en/latest/releases/ Not a tag in https://github.com/ceph/ceph I don´t understand why there's a 18.2.4 image and what's in it. -- Alfrenovsky

3 2

ingress for mgr service
by farhad kh 24 Jul '24

24 Jul '24

It is very good to be able to use Ingress for service manager This possibility of high availability for Ed Will this feature be added in the next version or should it still be implemented manually?

1 0

[MDS] Pacific memory leak
by Adrien Georget 23 Jul '24

23 Jul '24

Hi, For the last 2 months, our MDS is frequently switching to another because of a sudden memory leak. The host has 128G RAM and most of the time the MDS occupies ~20% of memory. And in less than 3 minutes it increases to 100% and crashs with tcmalloc: allocation failed. We tried to run heap stats / perf dump on the host but we couldn't find any reasons why the memory used by the MDS exploses so quickly. MDS log available here : https://filesender.renater.fr/?s=download&token=c1e60c3c-7f02-4f1e-b23e-f5b… Any idea what could lead to this memory leak? Anything we can try to understand what happens or prevent this? We use Pacific 16.2.14. Cheers, Adrien |cluster: id: 86cd8a68-7649-11ed-b2be-5cba2c7fdb30 health: HEALTH_OK services: mon: 5 daemons, quorum cccephadm40,cccephadm42,cccephadm41,cccephadm43,cccephadm44 (age 3d) mgr: cccephadm40.ucqhkr(active, since 105m), standbys: cccephadm41.osnxsd mds: 4/4 daemons up, 1 standby osd: 720 osds: 720 up (since 9d), 720 in (since 7w) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 12 pools, 8801 pgs objects: 1.25G objects, 2.7 PiB usage: 3.5 PiB used, 2.6 PiB / 6.1 PiB avail pgs: 8731 active+clean 59 active+clean+scrubbing+deep 11 active+clean+scrubbing io: client: 2.1 GiB/s rd, 1.2 GiB/s wr, 3.63k op/s rd, 17.47k op/s wr ceph fs status cephfs_astro - 1628 clients ============ RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs_astro.cccephadm41.nbgvxk Reqs: 7205 /s 11.2M 11.1M 708k 2782k 1 active cephfs_astro.cccephadm42.ieexgj Reqs: 812 /s 11.6M 11.5M 554k 1366k 2 active cephfs_astro.cccephadm40.aafhps Reqs: 1 /s 11.4M 11.3M 1346k 462k 3 active cephfs_astro.cccephadm44.jodfcx Reqs: 3 /s 11.4M 11.4M 1094k 29.3k POOL TYPE USED AVAIL cephfs_metadata metadata 484G 13.5T cephfs_default data 0 13.5T cephfs_EC_data data 3463T 1830T cephfs_ssd data 5314G 13.5T STANDBY MDS cephfs_astro.cccephadm43.wetlrp MDS version: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) |

2 2

A problem with the CEPH PG state getting stuck
by 苏察哈尔灿 23 Jul '24

23 Jul '24

My ceph cluster sometimes gets stuck in the active+clean+snaptrim state when doing regular snapshot deletion, and the corresponding pg does not change for a long time. As follows: 27.7c     14350         0          0        0  38073876582            0           0   2864      3000        active+clean+snaptrim     9h  43777'677286  43777:1018248    [52,55,20,36,29,91,9,63,14,2]p52    [52,55,20,36,29,91,9,63,14,2]p52  2024-07-21T13:57:12.602903+0000  2024-07-20T11:02:06.953985+0000                   52  queued for scrub                                                27.c1     14055         0          0        0  37408096205            0           0   2754      3000        active+clean+snaptrim     9h  43777'676887   43777:952875     [0,21,39,62,26,58,41,86,66,2]p0     [0,21,39,62,26,58,41,86,66,2]p0  2024-07-22T02:26:43.470883+0000  2024-07-20T05:09:23.763918+0000                   44  periodic scrub scheduled @ 2024-07-23T07:46:50.084316+0000      27.19b    14711         0          0        0  38926389125            0           0   2765      3000        active+clean+snaptrim     9h  43777'698478   43777:849574      [5,7,29,14,61,10,22,0,19,37]p5      [5,7,29,14,61,10,22,0,19,37]p5  2024-07-22T03:28:39.036189+0000  2024-07-17T14:00:43.082766+0000                   40  periodic scrub scheduled @ 2024-07-23T12:30:57.488823+0000      27.1a3    14323         0          0        0  37918899397            0           0   2930      3000        active+clean+snaptrim     9h  43777'675713   43777:943324   [52,4,10,46,20,69,49,39,44,30]p52   [52,4,10,46,20,69,49,39,44,30]p52  2024-07-21T11:15:16.442506+0000  2024-07-20T04:10:45.391550+0000                   50  queued for scrub                  No change for 9 hours. Then, after I typed the command "ceph pg repeer 27.7c", the corresponding pg state was restored to normal. I don't know what this "repeer" command is for, will it have any effect on pg? What's the difference between "repeer" and "ceph pg repair"? Then, why does the pg get stuck so often? Thanks for your help! My ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)

1 0

cephadm rgw ssl certificate config
by Eugen Block 22 Jul '24

22 Jul '24

Hi, I came across [1] and wanted to try to have all certificates/keys in one file. But it appears that the validation happens only against the first cert. So what I did was to concatenate all certs/keys into one file, then added that to ceph: ceph config-key set rgw/cert/rgw.realm.zone -i all-hosts.cert.pem ceph orch redeploy rgw The daemons pick up the config correctly and start, but running a curl only succeeds for the first in the list: # first host ok curl https://host3.dmain:443 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult... # second host fails curl -v https://host4.domain:443 ... * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: [...] CN=host3.domain; emailAddress=[...] * start date: Jul 18 09:59:07 2024 GMT * expire date: Aug 17 09:59:07 2025 GMT * subjectAltName does not match host4.domain * SSL: no alternative certificate subject name matches target host name 'host4.domain' This is still on Pacific 16.2.13 (testing something for a customer). What I tried in a Reef cluster was to set the certs per host by mapping the cert directory into the containers, then changing this config: ceph config set client.rgw.realm.hostname.suffix rgw_frontends "beast ssl_port=443 ssl_certificate=/etc/ceph/mycert.pem" And after restarting the daemon, it seems to work. So my question is, how do you deal with per-host certificates and rgw? Any comments are appreciated. Thanks! Eugen [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/F2GXMK6PE2…

2 3

Ceph 19 Squid released?
by Nicola Mori 21 Jul '24

21 Jul '24

Dear Ceph users, on quay.io I see available images for 19.1.0. Yet I can't find any public release announcement, and on this page: https://docs.ceph.com/en/latest/releases/ version 19 is still not mentioned at all. So what's going on? Nicola

2 1