August 2026 - ceph-users

Re: Affects of high pg count on OSD peering
by Eugen Block 18 Aug '26

18 Aug '26

Hi, the IO should not be blocked for the entire pool but only for those repeering PGs. Warnings during that phase are expected. Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement. One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds). That could (or probably would) increase network traffic significantly, so it's a trade-off. If you have a test cluster with similar behavior, test it there first. Regards, Eugen Zitat von Elias Carter via ceph-users <ceph-users(a)ceph.io>: > We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total PGs and > upwards). > > Something that I have noticed is that CRUSH changes (marking an OSD as out, > or a new OSD joining the cluster) can cause many hundreds of PGs to remap > and peer. During this time we get PG_AVAILABILITY health check warnings as > the OSDs need to peer with their new PGs before they can serve IOs. RADOS > tail latency suffers because IOs sent to a peering PG are blocked until > peering completes. > > I believe that the more PGs we have, the more PGs need to peer, and the > longer IOs are blocked for the pool. > > Does anyone have suggestions on how to deal with PG peering blocking IOs? > > I was thinking about gradually re-weighting OSDs instead of marking them > out or in, but that can result in multiple redundant data moves. > > Elias > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Bucket listing suddenly 404
by Thomas Schneider 17 Aug '26

17 Aug '26

Ok it's completely solved. I too late realized that suddenly when calling the s3 gateway instead of seeing the default xml I saw our domain as Bucket. I _somehow_ after nearly 10 years of running the rgw had to set rgw_dns_name (again?) I'm too tired right now to analyze it any further. Hope this helps _someone_ in the future. On 2026-08-17 19:18, Thomas Schneider wrote: > We "solved" the sync_owner() problem by deleting all Buckets owned by > the user by changing the placement >> # rubgw is radosgw-admin with realm/zone/zonegroup set >> $rubgw metadata get "bucket.instance:$bucket:$meta" > $bucket-old.json >> cp $bucket-old.json $bucket-new.json >> sed -i 's/"placement_rule": "[^"]*"/"placement_rule": >> "default-placement"/g' $bucket-new.json >> $rubgw metadata put "bucket.instance:$bucket:$meta" >> --infile=$bucket-new.json > > and then deleting the buckets, but the 404 problem when trying to > list/access buckets still exist :/ > > I'm all out of ideas right now. > > On 2026-08-17 16:57, Thomas Schneider wrote: >> Hey you all, >> >> so today our certificates got renewed and we suddenly have the problem >> that every ls returns a 404 >>> # rclone ls s3-nichtmuk: >>> 2026/08/17 16:49:58 NOTICE: Failed to ls: operation error S3: >>> ListBuckets, https response error StatusCode: 404, RequestID: >>> tx00000d5116646fe615090-006a831f96-885923856-rub-z, HostID: >>> 885923856-rub-z-rub-zg, api error NoSuchBucket: UnknownError >> >> We are not sure what caused it but one likely culprit is a test with a >> placement rule we created and removed a while ago. >> We created a test placement and later removed it, but still got a user >> that is in that placement and because ceph still things that the user >> exists we get an error on the rgw startup >> >>> Aug 17 16:43:03 my-hostname radosgw[49228]: deferred set uid:gid to >>> 167:167 (ceph:ceph) >>> Aug 17 16:43:03 my-hostname radosgw[49228]: ceph version 20.2.0 >>> (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - >>> RelWithDebInfo), process radosgw, pid 8 >>> Aug 17 16:43:03 my-hostname radosgw[49228]: framework: beast >>> Aug 17 16:43:03 my-hostname radosgw[49228]: framework conf key: port, >>> val: 8001 >>> Aug 17 16:43:03 my-hostname radosgw[49228]: init_numa not setting >>> numa affinity >>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Realm: >>> rub-realm (4d725f78-42cc-4ec1-8a0e-feff5800bdfb) >>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: ZoneGroup: >>> rub-zg (4170a834-cbe5-4cee-b4f9-83a47941c5a3) >>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Zone: >>> rub-z (99d273b2-e74c-432f-94de-1a73886419e8) >>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: using period >>> configuration: bcf5d281-9eb3-4a13-a011-7cc94c040cdf:19 >>> Aug 17 16:43:04 my-hostname radosgw[49228]: all 8 watchers are set, >>> enabling cache >>> Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: >>> garbage collection: start >>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw object expirer Worker >>> thread: object expiration: start >>> Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: >>> garbage collection: stop >>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: >>> watch_reload::DEDUP_WATCH_OBJ exists >>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: >>> dedup_bg::watch_reload(): Started watching >>> DEDUP_WATCH_OBJ::watch_handle=94372625010048 >>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework: beast >>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: >>> ssl_certificate, val: config://rgw/cert/$realm/$zone.crt >>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: >>> ssl_private_key, val: config://rgw/cert/$realm/$zone.key >>> Aug 17 16:43:04 my-hostname radosgw[49228]: starting handler: beast >>> Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on >>> 0.0.0.0:8001 >>> Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on >>> [::]:8001 >>> Aug 17 16:43:05 my-hostname radosgw[49228]: set uid:gid to 167:167 >>> (ceph:ceph) >>> Aug 17 16:43:05 my-hostname radosgw[49228]: mgrc >>> service_daemon_register rgw.885923856 metadata >>> {arch=x86_64,ceph_release=tentacle,ceph_version=ceph version 20.2.0 >>> (69f84cc2651aa259a15bc >>> 192ddaabd3baba07489) tentacle (stable - >>> RelWithDebInfo),ceph_version_short=20.2.0,container_hostname=my-hostname,container_image=quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b >>> 6dac688bcc77369b06009b5830fa8d86,cpu=Intel(R) Xeon(R) Silver 4114 CPU >>> @ 2.20GHz,distro=centos,distro_description=CentOS Stream >>> 9,distro_version=9,frontend_config#0=beast >>> port=8001,frontend_type#0=beast,hostname=rub-01-osd.c >>> eph-2.it-services.ruhr-uni-bochum.de,id=rub-rgw.rub-01-osd.ckfkhi,kernel_description=#1 >>> SMP Thu May 18 10:27:05 EDT >>> 2023,kernel_version=4.18.0-477.13.1.el8_8.x86_64,mem_swap_kb=16777212,mem_total_kb=131086224,num_handles=1, >>> os=Linux,pid=8,realm_id=4d725f78-42cc-4ec1-8a0e-feff5800bdfb,realm_name=rub-realm,zone_id=99d273b2-e74c-432f-94de-1a73886419e8,zone_name=rub-z,zonegroup_id=4170a834-cbe5-4cee-b4f9-83a47941c5a3,zonegroup_name=rub-zg} >>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw period pusher: No >>> zones to update >>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>> ERROR: could not sync bucket stats: ret=-5 >>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>> ERROR: failed user stats sync, ret=-5 >>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>> ERROR: sync_owner() failed, owner=rub-fowi-katzen ret=-5 >>> Aug 17 16:43:06 my-hostname radosgw[49228]: rgw object expirer Worker >>> thread: object expiration: stop >>> Aug 17 16:43:22 my-hostname radosgw[49228]: ====== starting new >>> request req=0x7fd2a608b310 >>> request_id=tx0000000924606841dd39d-006a831e0a-885923856-rub-z ===== >> >> >> (set rgw_debug to 5/5 for this) >>> Aug 17 16:49:54 my-hostname radosgw[49228]: beast: 0x7fd2a608b310: >>> 127.0.0.1 - rub-bucket [17/Aug/2026:14:49:54.826 +0000] "GET >>> /sfb1491-f7?max-keys=1000&prefix= HTTP/1.1" 404 255 - >>> "rclone/v1.74.3" - latency=0.001000081s >>> Aug 17 16:49:58 my-hostname radosgw[49228]: ====== starting new >>> request req=0x7fd2a608b310 >>> request_id=tx00000d5116646fe615090-006a831f96-885923856-rub-z ===== >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.000000000s getting op 0 >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.000000000s s3:list_bucket verifying requester >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.001000082s s3:list_bucket normalizing buckets and tenants >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.001000082s s3:list_bucket init permissions >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.001000082s s3:list_bucket op status=0 >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.001000082s s3:list_bucket http status=404 >>> Aug 17 16:49:58 my-hostname radosgw[49228]: ====== req done >>> req=0x7fd2a608b310 op=list_bucket bucket=our-s3-host status=0 >>> http_status=404 latency=0. >> >> >> We are not sure how to remove the user or debug the problem right now >>> # rubgw user rm --uid=rub-fowi-katzen --purge-data >>> could not remove user: unable to remove user, unable to delete user >>> data >>> 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: >>> cls_bucket_list_unordered: error in rgw_rados_operate (bucket list >>> op), r=-2 >>> 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: >>> list_objects_unordered cls_bucket_list_unordered returned -2 for >>> :FOO[99d273b2-e74c-432f-94de-1a73886419e8.791780109.1]) >> >> Any help or hints would be greatly appreciated >> >> Kind regards, >> Thomas -- Thomas Schneider IT.SERVICES Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 Bochum Telefon: +49 234 32 23939 http://www.it-services.rub.de/

1 0

Bucket listing suddenly 404
by Thomas Schneider 17 Aug '26

17 Aug '26

Hey you all, so today our certificates got renewed and we suddenly have the problem that every ls returns a 404 > # rclone ls s3-nichtmuk: > 2026/08/17 16:49:58 NOTICE: Failed to ls: operation error S3: > ListBuckets, https response error StatusCode: 404, RequestID: > tx00000d5116646fe615090-006a831f96-885923856-rub-z, HostID: > 885923856-rub-z-rub-zg, api error NoSuchBucket: UnknownError We are not sure what caused it but one likely culprit is a test with a placement rule we created and removed a while ago. We created a test placement and later removed it, but still got a user that is in that placement and because ceph still things that the user exists we get an error on the rgw startup > Aug 17 16:43:03 my-hostname radosgw[49228]: deferred set uid:gid to > 167:167 (ceph:ceph) > Aug 17 16:43:03 my-hostname radosgw[49228]: ceph version 20.2.0 > (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo), process radosgw, pid 8 > Aug 17 16:43:03 my-hostname radosgw[49228]: framework: beast > Aug 17 16:43:03 my-hostname radosgw[49228]: framework conf key: port, > val: 8001 > Aug 17 16:43:03 my-hostname radosgw[49228]: init_numa not setting numa > affinity > Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Realm: > rub-realm (4d725f78-42cc-4ec1-8a0e-feff5800bdfb) > Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: ZoneGroup: rub-zg > (4170a834-cbe5-4cee-b4f9-83a47941c5a3) > Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Zone: rub-z > (99d273b2-e74c-432f-94de-1a73886419e8) > Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: using period > configuration: bcf5d281-9eb3-4a13-a011-7cc94c040cdf:19 > Aug 17 16:43:04 my-hostname radosgw[49228]: all 8 watchers are set, > enabling cache > Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: garbage > collection: start > Aug 17 16:43:04 my-hostname radosgw[49228]: rgw object expirer Worker > thread: object expiration: start > Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: garbage > collection: stop > Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: > watch_reload::DEDUP_WATCH_OBJ exists > Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: > dedup_bg::watch_reload(): Started watching > DEDUP_WATCH_OBJ::watch_handle=94372625010048 > Aug 17 16:43:04 my-hostname radosgw[49228]: framework: beast > Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: > ssl_certificate, val: config://rgw/cert/$realm/$zone.crt > Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: > ssl_private_key, val: config://rgw/cert/$realm/$zone.key > Aug 17 16:43:04 my-hostname radosgw[49228]: starting handler: beast > Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on > 0.0.0.0:8001 > Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on > [::]:8001 > Aug 17 16:43:05 my-hostname radosgw[49228]: set uid:gid to 167:167 > (ceph:ceph) > Aug 17 16:43:05 my-hostname radosgw[49228]: mgrc > service_daemon_register rgw.885923856 metadata > {arch=x86_64,ceph_release=tentacle,ceph_version=ceph version 20.2.0 > (69f84cc2651aa259a15bc > 192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo),ceph_version_short=20.2.0,container_hostname=my-hostname,container_image=quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b > 6dac688bcc77369b06009b5830fa8d86,cpu=Intel(R) Xeon(R) Silver 4114 CPU @ > 2.20GHz,distro=centos,distro_description=CentOS Stream > 9,distro_version=9,frontend_config#0=beast > port=8001,frontend_type#0=beast,hostname=rub-01-osd.c > eph-2.it-services.ruhr-uni-bochum.de,id=rub-rgw.rub-01-osd.ckfkhi,kernel_description=#1 > SMP Thu May 18 10:27:05 EDT > 2023,kernel_version=4.18.0-477.13.1.el8_8.x86_64,mem_swap_kb=16777212,mem_total_kb=131086224,num_handles=1, > os=Linux,pid=8,realm_id=4d725f78-42cc-4ec1-8a0e-feff5800bdfb,realm_name=rub-realm,zone_id=99d273b2-e74c-432f-94de-1a73886419e8,zone_name=rub-z,zonegroup_id=4170a834-cbe5-4cee-b4f9-83a47941c5a3,zonegroup_name=rub-zg} > Aug 17 16:43:05 my-hostname radosgw[49228]: rgw period pusher: No zones > to update > Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: > ERROR: could not sync bucket stats: ret=-5 > Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: > ERROR: failed user stats sync, ret=-5 > Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: > ERROR: sync_owner() failed, owner=rub-fowi-katzen ret=-5 > Aug 17 16:43:06 my-hostname radosgw[49228]: rgw object expirer Worker > thread: object expiration: stop > Aug 17 16:43:22 my-hostname radosgw[49228]: ====== starting new request > req=0x7fd2a608b310 > request_id=tx0000000924606841dd39d-006a831e0a-885923856-rub-z ===== (set rgw_debug to 5/5 for this) > Aug 17 16:49:54 my-hostname radosgw[49228]: beast: 0x7fd2a608b310: > 127.0.0.1 - rub-bucket [17/Aug/2026:14:49:54.826 +0000] "GET > /sfb1491-f7?max-keys=1000&prefix= HTTP/1.1" 404 255 - "rclone/v1.74.3" > - latency=0.001000081s > Aug 17 16:49:58 my-hostname radosgw[49228]: ====== starting new > request req=0x7fd2a608b310 > request_id=tx00000d5116646fe615090-006a831f96-885923856-rub-z ===== > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.000000000s getting op 0 > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.000000000s s3:list_bucket verifying requester > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.001000082s s3:list_bucket normalizing buckets and tenants > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.001000082s s3:list_bucket init permissions > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.001000082s s3:list_bucket op status=0 > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.001000082s s3:list_bucket http status=404 > Aug 17 16:49:58 my-hostname radosgw[49228]: ====== req done > req=0x7fd2a608b310 op=list_bucket bucket=our-s3-host status=0 > http_status=404 latency=0. We are not sure how to remove the user or debug the problem right now > # rubgw user rm --uid=rub-fowi-katzen --purge-data > could not remove user: unable to remove user, unable to delete user > data > 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: > cls_bucket_list_unordered: error in rgw_rados_operate (bucket list op), > r=-2 > 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: > list_objects_unordered cls_bucket_list_unordered returned -2 for > :FOO[99d273b2-e74c-432f-94de-1a73886419e8.791780109.1]) Any help or hints would be greatly appreciated Kind regards, Thomas -- Thomas Schneider IT.SERVICES Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 Bochum Telefon: +49 234 32 23939 http://www.it-services.rub.de/

1 1

RelWithDebInfo in 20.2.3 Tentacle release (ceph orch package)
by Boris 17 Aug '26

17 Aug '26

Hi, is this intentional? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Ceph Developer Summit Vampire recordings
by Anthony Middleton 14 Aug '26

14 Aug '26

Hey friends, Here are the recordings from the Developer Summit for Vampire: https://www.youtube.com/playlist?list=PLAYbnnO0qf9U Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: [tentacle 20.2.3] How to configure ceph dashboard in a multi subnet deployment
by Alexander Harvey 14 Aug '26

14 Aug '26

I had considerable difficulty getting mgmt-gateway to work here with lots of changes to the YAML and many re-deploys of the dashboard. There are still some problems accessing Prometheus and Grafana that I have not figured out how to solve. The YAML ended up looking like this in the end: --- service_type: ingress service_id: ingress-mgmt-gw service_name: ingress.ingress-mgmt-gw placement: label: mgmt spec: backend_service: mgmt-gateway first_virtual_router_id: 50 keepalive_only: true virtual_ip: .... --- service_type: mgmt-gateway service_name: mgmt-gateway placement: label: mgmt spec: ssl: true ssl_cert: '-----BEGIN CERTIFICATE----- -----END CERTIFICATE----- ' ssl_key: '-----BEGIN PRIVATE KEY----- -----END PRIVATE KEY----- ' --- Alexander Harvey Senior Systems Engineer SAA Infrastructure Team University of Vermont -- From: wodel youchi via ceph-users <ceph-users(a)ceph.io> Date: Thursday, 2026 August 13 at 18:07 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] [tentacle 20.2.3] How to configure ceph dashboard in a multi subnet deployment Hi, In my ceph deployment, I have 3 subnets : mgmt subnet : 172.16.2.0/24 ceph public subnet : 10.10.0.0/16 ceph cluster subnet : 10.20.0.0/16 dashboard is reachable from 10.10.0.0/16 subnet the public ceph. I want to be able to reach it from the mgmt network. I've tried to install keepalived and haproxy to do that, but when I access the mgmt VIP, I'am redirected to the mgr ip on the public subnet. Regards _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: RGW: cannot PUT data into S3-like bucket
by Jan Marek 14 Aug '26

14 Aug '26

Hello, problem was in mtu settings on haproxy host. It has set mtu to 9000, whole CEPH cluster have mtu 9000, but client had mtu 1500. Problem was in the fragmentation. When I set on haproxy host mtu to 1500, problem disappeared. Sincerely Jan Marek Dne st, srp 12, 2026 at 05:12:20 CEST napsal(a) Jan Marek via ceph-users: > Hello, > > I've tried to connect directly to RGW and everythink works fine. > Thus problem is in haproxy, or it's configuration. > > Sincerely > Jan Marek > > Dne st, srp 12, 2026 at 12:51:24 CEST napsal(a) Janne Johansson: > > after upgrade from ceph 19.2.5 to 20.2.2 (and now I'm on 20.2.3), > > I have big problem to write any bigger blocks of data to S3 > > buckets. > > After upgrade to 20.2.2, everythink PUT requests gets 502 HTTP > > code. > > Today I've tried this: > > 1) I've created new bucket > > 2) I set this bucket for rustic (rustic init) with user, which is > > Owner of this buckets. 'rustic init' was fine. > > 3) I've tried to do backup: 'rustic backup'. No one block of > > backup data was written. > > > > I would certainly try a manual S3 client and validate if you can or not put > > objects into buckets, also reading the RGW logs from when this is erroring out > > seems like a very good idea. > > -- > > May the most significant bit of your life be positive. > > -- > Ing. Jan Marek > správce IT > -- Ing. Jan Marek správce IT Jihočeská univerzita v Českých Budějovicích Centum informačních technologií Branišovská 1645/31a, 370 05 České Budějovice T/ +420 38 903 2080 M/ +420 604 660 121 E/ jmarek(a)jcu.cz www.jcu.cz

1 0

[tentacle 20.2.3] How to configure ceph dashboard in a multi subnet deployment
by wodel youchi 13 Aug '26

13 Aug '26

Hi, In my ceph deployment, I have 3 subnets : mgmt subnet : 172.16.2.0/24 ceph public subnet : 10.10.0.0/16 ceph cluster subnet : 10.20.0.0/16 dashboard is reachable from 10.10.0.0/16 subnet the public ceph. I want to be able to reach it from the mgmt network. I've tried to install keepalived and haproxy to do that, but when I access the mgmt VIP, I'am redirected to the mgr ip on the public subnet. Regards

1 0

Re: Performance issue after tentacle upgrade
by Moti 13 Aug '26

13 Aug '26

Yes we do have load balancer, but the service that monitors latency, probes rgw endpoints directly as well as the loadbalancer endpoint, and latency jump is similar for all those endpoints, not only loadbalancer. Also 9 clusters were upgraded from 18 to 20 now, on vastly different times, and they all started latency jump only after upgrade, so it's unlikely to be an independent coincidence in loadbalancer. Two new things I found: - The only consistent latency jump through all 9 clusters is in HEAD method, other methods differ cluster to cluster but HEAD latency is vividly and significantly worse on all of them - I removed and re-created the bucket that prober service is using, it fixed the HEAD latency for several hours and then it again jumped back up. This also happens consistently on all clusters, though with different speeds. I'm not sure what to conclude of that. Something is being corrupted in bucket index? On Mon, 10 Aug 2026, 09:31 Eugen Block via ceph-users, <ceph-users(a)ceph.io> wrote: > Do you have loadbalancers in front of the client traffic? Maybe > something changed there? > > Zitat von Moti <motaharesdq(a)gmail.com>: > > > Hello Eugen, > > I agree that a performance issue this major would've gone noticed by now, > > but I did a load of investigation and still can't find the root cause, it > > might be a non-Ceph issue, but also possible to be a corner case? > > No there haven't been any OS upgrade, or another specific change. Did a > > ceph config dump (including defaults) before/after upgrade, checked all > the > > changed ones, reverted the fishy ones, nothing changed. The previous 3 > > clusters that I was testing on were all bare-metal - all had multi-site > > between 2 zones; separation between ec and replica pool is handled with a > > lua script, no RBD only object store. I also checked on a VM-based > cluster > > (again multi-site, but no hybrid build this time). Same issue there. > > However, when tested on a small single site cluster, that wasn't the case > > (so no visible latency increase there), but it just made things more > > confusing since in all of the other multi-site clusters, the RGWs that > run > > sync threads are separate from the ones that clients connect too, and > it's > > the client latency that I'm monitoring. I did notice this issue: > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… > > but that's not much similar to our case since we don't see any jump in > CPU > > or memory usage. > > Also checked pg ops (including scrubs), some clusters were running scrubs > > and stuff for a while after upgrade but that cleared soon enough, though > > the latency jump seems persistent still after 2 months. > > One correction about my observation on NVMe iops: that started dropping a > > little while "before" upgrade so it's most likely a separate issue, but > > latency jump case stays the same. > > As for other parameters, LC ops do not seem to have changed, there is a > > jump in GC max age but that's minor, cross site replication looks fine, > no > > jump in osd compression. > > Any ideas what else might worth checking/testing? > > > > On Tue, 14 Jul 2026, 12:53 Eugen Block via ceph-users, < > ceph-users(a)ceph.io> > > wrote: > > > >> Hi, > >> > >> I'm not sure if it could be related to summer vacation, but I feel > >> like there should have been several reports similar to yours by now. I > >> don't have upgraded any production cluster to 20 yet, so I can't > >> really deny or confirm anything. > >> Did you upgrade the OS as well? Maybe there were some relevant changes > >> there. Or some non-persistent configs were overwritten, either in the > >> OS or Ceph. > >> Was there more deep-scrubbing after the upgrade which could have > >> impacted the IO? > >> > >> Regards, > >> Eugen > >> > >> Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > >> > >> > Hello everyone! > >> > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid > thing > >> as > >> > well as Tentacle) and latency went ~3-4x more on PUT, even worse on > GET > >> and > >> > HEAD requests. > >> > > >> > There is no resource cap or significant increase in usage is visible. > >> > Neither fast ec nor any other new features are enabled yet, just did > the > >> > upgrade and left it as is. For measuring latency I have a services > that > >> > probes the cluster with consistent load so it's not a difference in > load > >> > either. PG status history does not show any pg operation being > heavily in > >> > process. > >> > > >> > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db > >> block. > >> > Both ec and replicated pools exist. > >> > The strange hint was: NVMe disks' iops went from something around 5-6K > >> > constant io per second to almost zero (less than a hundred per > second, on > >> > all host/OSDs, with some non-frequent high peaks), while the HDD iops > is > >> > not much changed. db blocks on NVMe are still attached to OSDs with no > >> > issue. > >> > > >> > I can't find a clue to explain this. Got the same results on 3 > separate > >> > clusters (all went version 18 --> 20, same hybrid build and configs). > >> > Nothing in config dump changes (changed defaults) catches my > attention to > >> > have caused this. > >> > > >> > Anyone had similar issues? Any tips and ideas on what to check? > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users(a)ceph.io > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: umbrella 21.1.1 RC2 QE validation status
by Brad Hubbard 13 Aug '26

13 Aug '26

On Mon, Aug 10, 2026 at 11:32 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/78413 > > Release Notes - email version for RCs > https://github.com/ceph/ceph/pull/70240#issuecomment-5232852937 @Neha Ojha > <nojha(a)redhat.com> @Patrick Donnelly <pdonnell(a)redhat.com> > LRC upgrade - TBD > > PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Radek, Nitzan, Afreen (dashboard), Travis (rook) > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane, Guillaume > rbd: Ilya > krbd: Ilya > > nvmeof: PASSED > > upgrade: Kamoltat > > upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: PASSED > upgrade-clients:client-upgrade-squid-umbrella-umbrella: PASSED > > perf-basic: PASSED > > crimson-rados: Matan > > powercycle: Brad (same as before) > Approved, waiting on known issue https://tracker.ceph.com/issues/79075 > TIA > -- Cheers, Brad

1 0

Re: RGW: cannot PUT data into S3-like bucket
by Janne Johansson 12 Aug '26

12 Aug '26

> > > after upgrade from ceph 19.2.5 to 20.2.2 (and now I'm on 20.2.3), > I have big problem to write any bigger blocks of data to S3 > buckets. > After upgrade to 20.2.2, everythink PUT requests gets 502 HTTP > code. > Today I've tried this: > > 1) I've created new bucket > 2) I set this bucket for rustic (rustic init) with user, which is > Owner of this buckets. 'rustic init' was fine. > 3) I've tried to do backup: 'rustic backup'. No one block of > backup data was written. I would certainly try a manual S3 client and validate if you can or not put objects into buckets, also reading the RGW logs from when this is erroring out seems like a very good idea. -- May the most significant bit of your life be positive.

2 1

RGW: cannot PUT data into S3-like bucket
by Jan Marek 12 Aug '26

12 Aug '26

Hello, after upgrade from ceph 19.2.5 to 20.2.2 (and now I'm on 20.2.3), I have big problem to write any bigger blocks of data to S3 buckets. My config: CEPH on HDD, everythink works fine. We are using not only S3 RGW, but RADOS for images of virtual machines - without problem. I have 9 host with total 108 HDD. On 4 fast machines I have RGW containers. CEPH cluster is maintained by cephadm (podman). I have haproxy, which do gateway to RGW and maintain Let's Encrypt certificates. After upgrade to 20.2.2, everythink PUT requests gets 502 HTTP code. I've manage Veeam backup of M365 and some rustic backups. Today I've tried this: 1) I've created new bucket 2) I set this bucket for rustic (rustic init) with user, which is Owner of this buckets. 'rustic init' was fine. 3) I've tried to do backup: 'rustic backup'. No one block of backup data was written. Please, can you help me? Many thanks. Sincerely Jan Marek -- Ing. Jan Marek správce IT Jihočeská univerzita v Českých Budějovicích Centum informačních technologií Branišovská 1645/31a, 370 05 České Budějovice T/ +420 38 903 2080 M/ +420 604 660 121 E/ jmarek(a)jcu.cz www.jcu.cz

1 0

Re: Performance issue after tentacle upgrade
by Oguzhan Ozmen (BLOOMBERG/ 120 PARK) 11 Aug '26

11 Aug '26

Hey Moti, Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > I recently upgraded from 18.2.4 to 20.2.1 > > I did notice this issue: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… > but that's not much similar to our case since we don't see any jump in CPU or memory usage. Although the symptoms don't look the same, I'd recommend upgrading to v20.2.3 to get the fix for this issue and the other fixes that went in: https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle After v20.2.3 upgrade, if you still see the same issue, then I'd suggest collecting "perf" output per process per component (rgw, osd mainly) to further analyze the bottleneck. You can also quickly check rgw logs to see what RGW reports as the latency for those PUT/GET/HEAD requests whether it matches the latency you see on your end (your wkld generator); especially for HEAD where no data streaming happens. RGW logs "start" and "end" of a request. This is to rule out whether bottleneck is outside of ceph (e.g., load-balancers). Cheers, Oguzhan

1 0

Announcing go-ceph v0.41.0
by John Mulligan 11 Aug '26

11 Aug '26

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.41.0 Enhancements include additions to cephfs and rgw admin subpackages. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project.

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Stefan Möding 11 Aug '26

11 Aug '26

Hi, thanks for your feedback! Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > that sounds a bit impractical to create a new pool for each tenant > regarding PG and pool management (adding and possibly removing data > pools from the CephFS), depending on how many tenants you have or plan > to have. In our setup each K8S cluster would be on tenant. So the pools would be on the subvolumegroup level and we expect to go ahead with less than ten pools for the foreseeable future. > What I just tested was to manually encrypt a directory in a test > cluster with fscrypt, that seems to work as intended. But I don't have > anything with k8s to test it at the moment. From what I have tested, the problem seems to be located in ceph-csi where the encryption is initially set up. The code tries to allocates an exclusive lock on an object representing the volume to prevent a race condition with multiple clients sharing the volume. The relevant code looks like this: ioctx, err := volOptions.GetConnection().GetIoctx(volOptions.MetadataPool) if err != nil { log.ErrorLog(ctx, "Failed to create ioctx: %s", err) return err } lock := iolock.NewLock(ioctx, objectUUID, lockName, lockCookie, lockDesc, lockDuration) err = lock.LockExclusive(ctx) if err != nil { log.ErrorLog(ctx, "failed to create lock for volume ID %s: %v", volID, err) return err } The error message on the ceph/csi pods clearly shows that the LockExclusive call fails with “Operation not permitted”. I failed to find the place in the relevant code, where the namespace for this operation is set, so it looks like the default namespace is used. But the capabilities only allow a specific namespace and so this could be the root cause. I tried to use the following code to create an object in the metadata pool and lock it. I used the same user with the same capabilities: % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put foobar /dev/null % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock get foobar lock1 --lock-cookie=cookie --lock-duration=60 --lock-type=exclusive % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock info foobar lock1 {"name":"lock1","type":"exclusive","tag":"","lockers":[{"name":"client.2690590","cookie":"cookie","description":"","expiration":"2026-07-31T08:21:36.190835+0200","addr":"172.21.70.10:0/2292729948"}]} So the capabilities seem to be sufficient for the lock operation. But leaving out the namespace when creating the lock will show the exact same error message that the ceph-csi shows: % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put foobar /dev/null % rados --id=k8s-test --pool=cephfs.meta lock get foobar lock1 --lock-cookie=cookie --lock-duration=60 --lock-type=exclusive ERROR: failed locking: (1) Operation not permitted error 1: (1) Operation not permitted -- Stefan

1 0

Re: Survey: Is anyone using 32-bit builds of the CephFS kernel client?
by Xiubo Li 11 Aug '26

11 Aug '26

Hi everyone, Thanks for the feedback. So far, I have received several replies regarding the 32-bit CephFS kernel client usage. Based on the responses, there do not appear to be any active users of 32-bit builds, and I have not received any objections to moving forward with a 64-bit-only implementation. Therefore, I will continue working on the patch to remove 32-bit support and proceed with the 64-bit-only approach. I will make sure the change is properly documented, including the deprecation/removal of 32-bit support in the relevant release notes. Thanks again for the feedback and discussion. Best regards, Xiubo On Thu, 16 Jul 2026 at 16:53, Xiubo Li <xiubo.li(a)clyso.com> wrote: > > Hi everyone, > > We are considering a performance improvement for the CephFS kernel > client (kclient) that would be significantly simpler and cleaner to > implement as 64-bit-only. Supporting 32-bit in this case adds > non-trivial complexity and ongoing maintenance burden, so we'd like to > understand whether continuing 32-bit support is still necessary. > > Before making any decision, we want to hear from the community: > > - Are you currently using, or planning to use, 32-bit (arm/armhf, > i386/i686) builds of the CephFS kernel client? > - If yes, is this for production workloads? Or legacy/testing purposes? > - What is your platform (distribution, kernel version, architecture)? > > If there are no active 32-bit users, we would proceed with a > 64-bit-only implementation and clearly document the deprecation in the > release notes with appropriate warnings. > > If 32-bit support is still needed, we will explore alternative > approaches — for example, keeping the current code path for 32-bit > while the new fast path is 64-bit-only, or finding a way to cleanly > abstract the difference. > > Please reply by August 6, 2026 so we can factor your feedback into the plan. > > Thanks! > Xiubo Li

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Eugen Block 10 Aug '26

10 Aug '26

Hi, that sounds a bit impractical to create a new pool for each tenant regarding PG and pool management (adding and possibly removing data pools from the CephFS), depending on how many tenants you have or plan to have. I would recommend to use this client authorization as a template: ceph fs subvolume authorize cephfs subvol-ebl user2 --group_name=subvolgroup1 --access_level=rw This results in these caps: ceph auth get client.user2 [client.user2] key = XXX caps mds = "allow rw path=/volumes/subvolgroup1/subvol-ebl/2596be92-9168-439f-b39f-61eb092e207b" caps mon = "allow r" caps osd = "allow rw pool=cephfs.cephfs.data namespace=fsvolumens_subvol-ebl" I created the subvolume in the dashboard with isolated namespace (which can be done via CLI as well, of course). And then you also need the "rw" caps (which you already have) for the mgr for k8s to be able to utilize the volume module. But I'm not sure if this will suffice to make it work. What I just tested was to manually encrypt a directory in a test cluster with fscrypt, that seems to work as intended. But I don't have anything with k8s to test it at the moment. IIRC, the caps you specified wrt metadata would indeed allow a client to access other tenant's metadata, but I'm not entirely sure. Hopefully someone else can chime in and has better answers for you. Regards, Eugen Zitat von Stefan Möding via ceph-users <ceph-users(a)ceph.io>: > Hi, > > Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > >> I haven't tried it myself yet so I don't really have an answer. But >> I noticed that your OSD caps may be incomplete wrt x permissions: > >>> osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, >>> allow rw namespace=k8s-test tag cephfs data=root' > >> The second allow statement doesn't have an "x". Maybe that's the issue >> why the workaround fails? > > We tried that, but the error remains. We now consider to use the > following capabilities in combination to creating a dedicated data pool > for each tenant (K8s cluster): > > mon_caps: 'allow r fsname=root' > mgr_caps: 'allow rw' > osd_caps: 'allow rwx pool=cephfs.meta, allow rw pool=cephfs.data, > allow rw pool=cephfs.k8s-test' > mds_caps: 'allow r fsname=root path=/volumes, allow rws fsname=root > path=/volumes/k8s-test' > > CephFS is created with cephfs.meta as metadata pool and cephfs.data as > default data pool while cephfs.k8s-test is the pool for this tenant. > > What are the implications to share the metadata/default data pools and > have no namespace? Is the MDS capability sufficient to separate tenants > or can everyone access the metadata objects of everyone else? > > -- > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Performance issue after tentacle upgrade
by Moti 10 Aug '26

10 Aug '26

Hello Eugen, I agree that a performance issue this major would've gone noticed by now, but I did a load of investigation and still can't find the root cause, it might be a non-Ceph issue, but also possible to be a corner case? No there haven't been any OS upgrade, or another specific change. Did a ceph config dump (including defaults) before/after upgrade, checked all the changed ones, reverted the fishy ones, nothing changed. The previous 3 clusters that I was testing on were all bare-metal - all had multi-site between 2 zones; separation between ec and replica pool is handled with a lua script, no RBD only object store. I also checked on a VM-based cluster (again multi-site, but no hybrid build this time). Same issue there. However, when tested on a small single site cluster, that wasn't the case (so no visible latency increase there), but it just made things more confusing since in all of the other multi-site clusters, the RGWs that run sync threads are separate from the ones that clients connect too, and it's the client latency that I'm monitoring. I did notice this issue: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… but that's not much similar to our case since we don't see any jump in CPU or memory usage. Also checked pg ops (including scrubs), some clusters were running scrubs and stuff for a while after upgrade but that cleared soon enough, though the latency jump seems persistent still after 2 months. One correction about my observation on NVMe iops: that started dropping a little while "before" upgrade so it's most likely a separate issue, but latency jump case stays the same. As for other parameters, LC ops do not seem to have changed, there is a jump in GC max age but that's minor, cross site replication looks fine, no jump in osd compression. Any ideas what else might worth checking/testing? On Tue, 14 Jul 2026, 12:53 Eugen Block via ceph-users, <ceph-users(a)ceph.io> wrote: > Hi, > > I'm not sure if it could be related to summer vacation, but I feel > like there should have been several reports similar to yours by now. I > don't have upgraded any production cluster to 20 yet, so I can't > really deny or confirm anything. > Did you upgrade the OS as well? Maybe there were some relevant changes > there. Or some non-persistent configs were overwritten, either in the > OS or Ceph. > Was there more deep-scrubbing after the upgrade which could have > impacted the IO? > > Regards, > Eugen > > Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > > > Hello everyone! > > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing > as > > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET > and > > HEAD requests. > > > > There is no resource cap or significant increase in usage is visible. > > Neither fast ec nor any other new features are enabled yet, just did the > > upgrade and left it as is. For measuring latency I have a services that > > probes the cluster with consistent load so it's not a difference in load > > either. PG status history does not show any pg operation being heavily in > > process. > > > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db > block. > > Both ec and replicated pools exist. > > The strange hint was: NVMe disks' iops went from something around 5-6K > > constant io per second to almost zero (less than a hundred per second, on > > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > > not much changed. db blocks on NVMe are still attached to OSDs with no > > issue. > > > > I can't find a clue to explain this. Got the same results on 3 separate > > clusters (all went version 18 --> 20, same hybrid build and configs). > > Nothing in config dump changes (changed defaults) catches my attention to > > have caused this. > > > > Anyone had similar issues? Any tips and ideas on what to check? > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by Frédéric Nass 09 Aug '26

09 Aug '26

Ok so here's the clarification: First of all, by "dropping NFSv3 support (by default)" I didn't mean that NFSv3 wouldn't be supported anymore, but that it would be disabled by default and would need to be explicitly enabled in the service specs. Also, this is not a future change, it already happened. v20.2.3 ships PR #70675, so upgrading to v20.2.3 silently disables NFSv3. The related PRs were merged shortly before the tag was cut and are missing from the v20.2.3 release notes and changelog, so there is currently no warning anywhere. We should fix that. Users who need NFSv3 must now explicitly enable it by adding 'enable_nfsv3: true' to the NFS service spec. Since older MGRs reject this field, it can only be applied once MGRs run v20.2.3. The cleanest path is a staggered upgrade to avoid redeploying the NFS service twice: 1. ceph orch upgrade start --image <v20.2.3> --daemon-types mgr 2. add 'enable_nfsv3: true' to the NFS service spec and re-apply it 3. run the upgrade as usual (already upgraded MGRs will be skipped) Users who have already upgraded should simply update the spec and re-apply it to get NFSv3 back. @Wodel Ultimately, upgrading to v20.2.3 should address your original concern. Best regards, Frédéric. -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Le sam. 8 août 2026, 02:36, Frédéric Nass via ceph-users <ceph-users(a)ceph.io> a écrit : > Hi Robert, > > Tentacle should soon be dropping NFSv3 support (by default) as per > https://github.com/ceph/ceph/pull/70675. My understanding so far is that > users will have to add 'enable_nfsv3: true' in the NFS service specs to > keep using NFSv3. I hope that can soon be clarified in the PR and Tentacle > documentation. > > Cheers, > Frédéric > > -- > Frédéric Nass > > Ceph Ambassador France | Senior Ceph Engineer @ CLYSO > > > Frédéric Nass > > Senior Ceph Engineer > > Ceph Ambassador, France > > +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> > > frederic.nass(a)clyso.com > > www.clyso.com > > Hohenzollernstr. 27, 80801 Munich > <https://www.google.com/maps/search/Hohenzollernstr.+27,+80801+Munich?entry=…> > > Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 > > > Le ven. 7 août 2026, 18:07, Robert Sander via ceph-users < > ceph-users(a)ceph.io> > a écrit : > > > Hi, > > > > Am 07.08.26 um 10:59 AM schrieb Frédéric Nass via ceph-users: > > > > > Or, if that's too much of a maintenance burden and you're worried > you'll > > > forget that step, just install rpcbind on the NFS nodes and leave NFSv3 > > > enabled. > > > > Or maybe NFSv3 is made an option for the NFS service spec with the hint > > that then rpcbind has to be installed. > > > > Or the orchestrator installs rpcbind when the NFS service should offer > > NFSv3. > > > > Regards > > -- > > Robert Sander > > Linux Consultant > > > > Heinlein Consulting GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > https://www.heinlein-support.de > > > > Tel: +49 30 405051 - 0 > > Fax: +49 30 405051 - 19 > > > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> frederic.nass(a)clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306

1 0

Re: Ceph Tentacle 20.2.3 Noble Support
by Tarrago, Eli (RIS-BCT) 07 Aug '26

07 Aug '26

Hello Anthony, 1. Our Ceph infrastructure was provisioned by ceph-ansible and utilizes package installation 2. Our Openstack deployment has hypervisors that use packages installed to the hypervisors to allow connectivity between hypervisor and cluster. 3. 20.2.2 Shipped with Noble support, but 20.2.3 did not. It appeared to be an oversight that I wanted to raise. I checked the change logs for the 20.2.3 which made no mention of depreciating the Noble package. As such, I raised it as a potential issue. Thanks, Eli Tarrago From: Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> Date: Friday, August 7, 2026 at 3:20 PM To: Tarrago, Eli (RIS-BCT) <Eli.Tarrago(a)lexisnexisrisk.com> Cc: Anthony Middleton via ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ceph Tentacle 20.2.3 Noble Support ⚠ External email If the sender is unfamiliar, check home.relx.com → Trusted partners to verify them. Why do you care about packages? > On Aug 7, 2026, at 10:10 AM, Tarrago, Eli (RIS-BCT) via ceph-users <ceph-users(a)ceph.io> wrote: > > Good Afternoon Ceph Users, > > I recently became aware of a discrepancy that I wanted to raise: > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdownload.…<https://download.ceph.com/debian-20.2.2/dists/> > > Providers Ubuntu Noble packages, > > Where as, > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdownload.…<https://download.ceph.com/debian-20.2.3/dists/> > > Does not. Sorry it’s Friday and to bring this up with you. I hope noble support can be returned to latest package release. > > Thank you > > Eli Tarrago > > ________________________________ > The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

1 0

Re: Ceph Tentacle 20.2.3 Noble Support
by Anthony D'Atri 07 Aug '26

07 Aug '26

Why do you care about packages? > On Aug 7, 2026, at 10:10 AM, Tarrago, Eli (RIS-BCT) via ceph-users <ceph-users(a)ceph.io> wrote: > > Good Afternoon Ceph Users, > > I recently became aware of a discrepancy that I wanted to raise: > > https://download.ceph.com/debian-20.2.2/dists/ > > Providers Ubuntu Noble packages, > > Where as, > > https://download.ceph.com/debian-20.2.3/dists/ > > Does not. Sorry it’s Friday and to bring this up with you. I hope noble support can be returned to latest package release. > > Thank you > > Eli Tarrago > > ________________________________ > The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Ceph Tentacle 20.2.3 Noble Support
by Yuri Weinstein 07 Aug '26

07 Aug '26

Hi Eli Thank you for your email. Noble was never part of the official tentacle distro support matrix (jammy, centos9, rocky10, bookworm, windows) — it appeared in the 20.2.1 and 20.2.2 builds outside that matrix, which was the actual discrepancy. 20.2.3 reflects the correct, intended matrix, so we won't be rebuilding noble for it. We plan to add Noble support to Tentacle this Fall. We'll clarify this in the release docs going forward to avoid confusion like this. Regards On Fri, Aug 7, 2026 at 10:10 AM Tarrago, Eli (RIS-BCT) via ceph-users < ceph-users(a)ceph.io> wrote: > Good Afternoon Ceph Users, > > I recently became aware of a discrepancy that I wanted to raise: > > https://download.ceph.com/debian-20.2.2/dists/ > > Providers Ubuntu Noble packages, > > Where as, > > https://download.ceph.com/debian-20.2.3/dists/ > > Does not. Sorry it’s Friday and to bring this up with you. I hope noble > support can be returned to latest package release. > > Thank you > > Eli Tarrago > > ________________________________ > The information contained in this e-mail message is intended only for the > personal and confidential use of the recipient(s) named above. This message > may be an attorney-client communication and/or work product and as such is > privileged and confidential. If the reader of this message is not the > intended recipient or an agent responsible for delivering it to the > intended recipient, you are hereby notified that you have received this > document in error and that any review, dissemination, distribution, or > copying of this message is strictly prohibited. If you have received this > communication in error, please notify us immediately by e-mail, and delete > the original message. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by Frédéric Nass 07 Aug '26

07 Aug '26

Hi Robert, Tentacle should soon be dropping NFSv3 support (by default) as per https://github.com/ceph/ceph/pull/70675. My understanding so far is that users will have to add 'enable_nfsv3: true' in the NFS service specs to keep using NFSv3. I hope that can soon be clarified in the PR and Tentacle documentation. Cheers, Frédéric -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> frederic.nass(a)clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Le ven. 7 août 2026, 18:07, Robert Sander via ceph-users <ceph-users(a)ceph.io> a écrit : > Hi, > > Am 07.08.26 um 10:59 AM schrieb Frédéric Nass via ceph-users: > > > Or, if that's too much of a maintenance burden and you're worried you'll > > forget that step, just install rpcbind on the NFS nodes and leave NFSv3 > > enabled. > > Or maybe NFSv3 is made an option for the NFS service spec with the hint > that then rpcbind has to be installed. > > Or the orchestrator installs rpcbind when the NFS service should offer > NFSv3. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: [Ceph-maintainers] Re: Re: v20.2.3 Tentacle released
by Afreen 07 Aug '26

07 Aug '26

On Fri, Aug 7, 2026 at 6:44 PM Salatiel Filho via Ceph-maintainers < ceph-maintainers(a)ceph.io> wrote: > What exactly do you mean by CephFS cloning ? This cluster is used > exclusively to provide RWX volumes to a k8s using cephcsi. Velero does > backup though and creates volume snapshots , but I am not sure if > that's what you mean. > Any commands I can run to confirm if cloning is being used ? > [Afreen] Yes that confirms then . So this bugs is being fixed and it only happens with cephfs subvolums die to a wrong format internally used https://tracker.ceph.com/issues/75619 > "ceph progress clear" did the trick to load the pools page in the > dashboard, but you said: "Its just clears the display events and does > not stop operations." > What operations are still running ? post-upgrade operations? why? > Will them eventually finish ? > [Afreen] What I meant was that running that command was not harmful. And good to know it worked for you. Thanks > > Thanks! > > Atenciosamente/Kind regards, > Salatiel > > On Thu, Aug 6, 2026 at 8:31 PM Afreen <afreen23.git(a)gmail.com> wrote: > > > > Hey, > > > > That seems like some issue in parsing the format and so far what we > found out is that it could happen when CephFS cloning is used , are you > using that ? > > And pools page is failing because its polling /summary API - fetching > progress internally. > > > > You can try clearing progress and then refresh - "ceph progress clear". > Its just clears the display events and does not stop operations. > > > > - Afreen > > > > On Thu, Aug 6, 2026 at 6:42 PM Salatiel Filho via ceph-users < > ceph-users(a)ceph.io> wrote: > >> > >> After upgrading from 20.2.2 to 20.2.3, the dashboard is partially > >> broken. List pools don't work. Error 500. > >> Also get error 500 popup as soon as I login to the dashboard. > >> > >> Logs show this: > >> 2026-08-06 10:04:23 2026-08-06T13:04:23.386+0000 7f5afcd91640 0 > >> [dashboard ERROR dashboard.services.exception] Internal Server Error > >> 2026-08-06 10:04:23 Traceback (most recent call last): > >> 2026-08-06 10:04:23 File > >> "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in > >> dashboard_exception_handler > >> 2026-08-06 10:04:23 return handler(*args, **kwargs) > >> 2026-08-06 10:04:23 File > >> "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in > >> __call__ > >> 2026-08-06 10:04:23 return self.callable(*self.args, > **self.kwargs) > >> 2026-08-06 10:04:23 File > >> "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line > >> 264, in inner > >> 2026-08-06 10:04:23 ret = func(*args, **kwargs) > >> 2026-08-06 10:04:23 File > >> "/usr/share/ceph/mgr/dashboard/controllers/summary.py", line 105, in > >> __call__ > >> 2026-08-06 10:04:23 e, f = progress.get_progress_tasks() > >> 2026-08-06 10:04:23 File > >> "/usr/share/ceph/mgr/dashboard/services/progress.py", line 89, in > >> get_progress_tasks > >> 2026-08-06 10:04:23 > >> finished_t.append(_progress_event_to_dashboard_task(ev, True)) > >> 2026-08-06 10:04:23 File > >> "/usr/share/ceph/mgr/dashboard/services/progress.py", line 60, in > >> _progress_event_to_dashboard_task > >> 2026-08-06 10:04:23 > _progress_event_to_dashboard_task_common(event, task) > >> 2026-08-06 10:04:23 File > >> "/usr/share/ceph/mgr/dashboard/services/progress.py", line 52, in > >> _progress_event_to_dashboard_task_common > >> 2026-08-06 10:04:23 'metadata': dict(event.get('refs', {})), > >> 2026-08-06 10:04:23 ValueError: dictionary update sequence element > >> #0 has length 3; 2 is required > >> 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 > >> [dashboard ERROR dashboard.tools] [::ffff:10.175.143.9:53998] [GET] > >> [500] [0.009s] [admin] [513.0B] /api/summary > >> 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 > >> [dashboard ERROR dashboard.tools] [b'{"status": "500 Internal Server > >> Error", "detail": "The server encountered an unexpected condition > >> which prevented it from fulfilling the request.", "request_id": > >> "c7349113-44d8-4e48-88d6-3a469786397f"} > >> > >> > >> > >> '] > >> 2026-08-06 10:04:23 2026-08-06T13:04:23.494+0000 7f5afed95640 0 > >> [dashboard ERROR dashboard.controllers.frontend_logging] > >> (https://ceph.mydomain/#/pool) Http failure response for > >> https://ceph.mydomain/api/summary: 500 Internal Server Error > >> > >> Atenciosamente/Kind regards, > >> Salatiel > >> > >> On Wed, Aug 5, 2026 at 11:06 AM Yuri Weinstein via ceph-users > >> <ceph-users(a)ceph.io> wrote: > >> > > >> > This is the third minor release in the Tentacle series. > >> > We recommend that all users update to this release. > >> > > >> > Notable Changes > >> > --------------- > >> > > >> > MDS (Metadata Server) > >> > --------------------- > >> > > >> > * Fixed a crash and shutdown hang that could occur when ephemeral > pins were > >> > active and max_mds was set to 0. > >> > * Session reclaim could miss blocklisting an old session; this is now > >> > handled correctly. > >> > * Fixed a case where scrub was unable to identify damage from an > >> > irreparable hard link. > >> > > >> > OSD (Object Storage Daemon) > >> > ---------------------------- > >> > > >> > * BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write > buffer > >> > misalignment. > >> > * Erasure Coding: Fixed truncate+write planning for EC shard sizes. > >> > * Fixed a condition that prevented rolling forward of PG log entries. > >> > * Prevented OSDMap::check_health() from asserting when new OSDs are > found > >> > in a subtree. > >> > * Scrub: "repairing" scrubs are now allowed at all times. > >> > > >> > RGW (RADOS Gateway) > >> > -------------------- > >> > > >> > * Added the ssl_ciphersuites option for the Beast front-end to > support TLS > >> > 1.3 cipher customization. > >> > * Fixed PutObject's canned_acl comparison bug affecting > BlockPublicAcls. > >> > * Fixed crashes occurring on realm reload. > >> > * Implemented CopyObject support for encrypted objects. > >> > * Added mTLS client-certificate authentication support for Kafka > >> > notifications. > >> > * Fixed removal of delete markers in lifecycle processing. > >> > > >> > RADOS / librados / neorados > >> > ---------------------------- > >> > > >> > * neorados: Fixed an infinite trim loop on empty data log shards. > >> > * Fixed cancellation-slot cleanup in librados/asio's associated > executor. > >> > > >> > Dashboard > >> > ----------------- > >> > > >> > * Added a hardware-monitoring dashboard backed by node-proxy metrics. > >> > * Fixed a bind-address regression caused by CherryPy isolation. > >> > * Added support for adding hosts via CSV upload. > >> > * NVMeoF: multiple CLI and UI backports (EC pool support, IO > statistics, > >> > role management). > >> > * Object: The global RGW Roles tab has been removed. Role management > is now > >> > scoped under RGW Accounts, where roles can be listed, created, > edited, and > >> > deleted for a selected account. > >> > > >> > ceph-volume > >> > ----------- > >> > > >> > * Reworked OSD mapper lifecycle handling (LVM + raw) for activate. > >> > * Improved detection of rotational media under dm-crypt to bypass the > >> > workqueue correctly. > >> > * Fixed OSD re-deployment issues with disk-selection filters and DB > devices. > >> > * Added retry handling for lvs when it returns an empty result or a > >> > "devices file is missing" error. > >> > > >> > NVMe-oF > >> > ------- > >> > > >> > * Added configurable delayed failback (default 0, no delay) to > accommodate > >> > initiators that recover paths with a delay. > >> > * Gateways in the DELETING state now ignore beacons and send empty > maps > >> > instead of processing them. > >> > * rbd_with_crc32c is now enabled by default via cephadm. > >> > * Introduced a new NVMeoF mgr module. The module automatically > creates the > >> > ".nvmeof" metadata pool if it does not already exist. Gateway-group > state > >> > files are stored in this pool unless the user specifies another pool. > The > >> > module is enabled automatically on fresh installs; on upgrades from an > >> > older cluster, it must be enabled manually. > >> > > >> > RBD > >> > --- > >> > > >> > * Fixed a use-after-free bug releasing object map locks during deep > copy. > >> > * Fixed memory leaks in PWL discard operations. > >> > * Obsolete primary mirror snapshots are now pruned after relocation. > >> > > >> > We recommend users to update to this release. > >> > For detailed release notes with links & changelog please refer to the > >> > official blog entry at > >> > https://ceph.io/en/news/blog/2026/v20-2-3-tentacle-released/ > >> > > >> > Getting Ceph > >> > ------------ > >> > * Git at git://github.com/ceph/ceph.git > >> > * Tarball at https://download.ceph.com/tarballs/ceph-20.2.3.tar.gz > >> > * Containers at https://quay.io/repository/ceph/ceph > >> > * For packages, see > https://docs.ceph.com/en/latest/install/get-packages/ > >> > * Release git sha1: 06c2f9c35b67055a8a6fb99d1be236b3c4832ace > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users(a)ceph.io > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > Ceph-maintainers mailing list -- ceph-maintainers(a)ceph.io > To unsubscribe send an email to ceph-maintainers-leave(a)ceph.io >

1 1

Ceph Tentacle 20.2.3 Noble Support
by Tarrago, Eli (RIS-BCT) 07 Aug '26

07 Aug '26

Good Afternoon Ceph Users, I recently became aware of a discrepancy that I wanted to raise: https://download.ceph.com/debian-20.2.2/dists/ Providers Ubuntu Noble packages, Where as, https://download.ceph.com/debian-20.2.3/dists/ Does not. Sorry it’s Friday and to bring this up with you. I hope noble support can be returned to latest package release. Thank you Eli Tarrago ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

1 0

New mailing list offered for Ceph job postings
by Patrick Donnelly 07 Aug '26

07 Aug '26

Greetings, A new mailing list is now available for job postings relating to Ceph: https://lists.ceph.io/postorius/lists/careers.ceph.io/ Please subscribe if you want to keep up with job offerings available from employers or would like to post jobs to the list. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Grafana certificate in version 19
by Robert Sander 07 Aug '26

07 Aug '26

Hi, Am 07.08.26 um 3:34 PM schrieb Eugen Block via ceph-users: > > what I noted in my "best practices" doc is this article from RH how to > convert to json format: > > https://access.redhat.com/solutions/7109138 Thanks. This links to https://github.com/rkachach/utils which seems to be a personal repo from Redouane Kachach who also talked about the certmgr in London: https://www.youtube.com/watch?v=AbjZD0pYwxc https://ceph.io/assets/pdfs/events/2026/ceph-days-london/management-in-ceph… So it seems to be valid. But come on guys, please update the documentation. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: Grafana certificate in version 19
by Chris Palmer 07 Aug '26

07 Aug '26

Trying again as something completely reformatted my commands.... I never got to the bottom of this, on either squid or tentacle. I have an externally-generated cert/key. As a workaround I do this after each upgrade (most recently to 20.2.3): cp ~/my_grafana_cert.key /var/lib/ceph/`ceph fsid`/grafana.`hostname -s`/etc/grafana/certs/cert_key cp ~/my_grafana_cert.crt /var/lib/ceph/`ceph fsid`/grafana.`hostname -s`/etc/grafana/certs/cert_file ceph orch daemon restart grafana.`hostname -s` --force It would be nice to know what the right way is. Regards, Chris On 07/08/2026 13:51, Robert Sander via ceph-users wrote: > Hi, > > how do I set the grafana certificate and key in Ceph 19 that is used on > port 3000? > > The documentation on > https://docs.ceph.com/en/squid/cephadm/services/monitoring/#configuring-ssl… > states: > > # ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem > # ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i > $PWD/certificate.pem > # ceph orch reconfig grafana > > But after doing that grafana still serves a self-signed certificate with > the common name containing the hostname of the first MON (which is not > correct). > > Digging into "ceph config-key dump" shows two entries named > mgr/cephadm/cert_store.cert.grafana_cert and > mgr/cephadm/cert_store.key.grafana_key > which seem to cause this. > > AFAIK the certmgr is only available in Ceph 20. But the docs for this > version > look the same: > https://docs.ceph.com/en/tentacle/cephadm/services/monitoring/#configuring-… > > Only the docs for Ceph 21 got an update and mention certmgr: > https://docs.ceph.com/en/umbrella/cephadm/services/monitoring/#configuring-… > > > The orchestrator in Ceph 19 seems to have something called cert-store: > > # ceph orch cert-store > no valid command found; 4 closest matches: > orch cert-store cert ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store key ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store get cert <entity> [--service_name <value>] [--hostname > <value>] [--no-exception-when-missing] > orch cert-store get key <entity> [--service_name <value>] [--hostname > <value>] [--no-exception-when-missing] > Error EINVAL: invalid command > > But there is no help available. > "ceph orch cert-store --help" does not return anything specific. > Searching for "cert-store" in the Ceph 19 docs also does not return > anything. > > Regards

1 0

Re: Grafana certificate in version 19
by Chris Palmer 07 Aug '26

07 Aug '26

I never got to the bottom of this, on either squid or tentacle. I have an externally-generated cert/key. As a workaround I do this after each upgrade (most recently to 20.2.3): |cp ~/my_grafana_cert.key /var/lib/ceph/`ceph fsid`/grafana.`hostname -s`/etc/grafana/certs/cert_key cp ~/||my_grafana_cert||.crt /var/lib/ceph/`ceph fsid`/grafana.||`hostname -s`||/etc/grafana/certs/cert_file ceph orch daemon restart grafana.||`hostname -s`||--force| It would be nice to know what the right way is. Regards, Chris On 07/08/2026 13:51, Robert Sander via ceph-users wrote: > Hi, > > how do I set the grafana certificate and key in Ceph 19 that is used on > port 3000? > > The documentation on > https://docs.ceph.com/en/squid/cephadm/services/monitoring/#configuring-ssl… > states: > > # ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem > # ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i > $PWD/certificate.pem > # ceph orch reconfig grafana > > But after doing that grafana still serves a self-signed certificate with > the common name containing the hostname of the first MON (which is not > correct). > > Digging into "ceph config-key dump" shows two entries named > mgr/cephadm/cert_store.cert.grafana_cert and > mgr/cephadm/cert_store.key.grafana_key > which seem to cause this. > > AFAIK the certmgr is only available in Ceph 20. But the docs for this > version > look the same: > https://docs.ceph.com/en/tentacle/cephadm/services/monitoring/#configuring-… > > Only the docs for Ceph 21 got an update and mention certmgr: > https://docs.ceph.com/en/umbrella/cephadm/services/monitoring/#configuring-… > > > The orchestrator in Ceph 19 seems to have something called cert-store: > > # ceph orch cert-store > no valid command found; 4 closest matches: > orch cert-store cert ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store key ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store get cert <entity> [--service_name <value>] [--hostname > <value>] [--no-exception-when-missing] > orch cert-store get key <entity> [--service_name <value>] [--hostname > <value>] [--no-exception-when-missing] > Error EINVAL: invalid command > > But there is no help available. > "ceph orch cert-store --help" does not return anything specific. > Searching for "cert-store" in the Ceph 19 docs also does not return > anything. > > Regards

1 0

Re: Grafana certificate in version 19
by Eugen Block 07 Aug '26

07 Aug '26

Hi, what I noted in my "best practices" doc is this article from RH how to convert to json format: https://access.redhat.com/solutions/7109138 And after conversion the certificate and key are imported like this: ceph config-key set mgr/cephadm/cert_store.cert.grafana_cert -i grafana_certs.json ceph config-key set mgr/cephadm/cert_store.cert.grafana_key -i grafana_key.json I vaguely remember a thread here on this list about it as well, but I don't know if this workaround is still required or not. I'll probably have to when the current cert expires. :-D Regards, Eugen Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Hi, > > how do I set the grafana certificate and key in Ceph 19 that is used on > port 3000? > > The documentation on > https://docs.ceph.com/en/squid/cephadm/services/monitoring/#configuring-ssl… > states: > > # ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem > # ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i > $PWD/certificate.pem > # ceph orch reconfig grafana > > But after doing that grafana still serves a self-signed certificate with > the common name containing the hostname of the first MON (which is > not correct). > > Digging into "ceph config-key dump" shows two entries named > mgr/cephadm/cert_store.cert.grafana_cert and > mgr/cephadm/cert_store.key.grafana_key > which seem to cause this. > > AFAIK the certmgr is only available in Ceph 20. But the docs for this version > look the same: > https://docs.ceph.com/en/tentacle/cephadm/services/monitoring/#configuring-… > > Only the docs for Ceph 21 got an update and mention certmgr: > https://docs.ceph.com/en/umbrella/cephadm/services/monitoring/#configuring-… > > The orchestrator in Ceph 19 seems to have something called cert-store: > > # ceph orch cert-store > no valid command found; 4 closest matches: > orch cert-store cert ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store key ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store get cert <entity> [--service_name <value>] > [--hostname <value>] [--no-exception-when-missing] > orch cert-store get key <entity> [--service_name <value>] > [--hostname <value>] [--no-exception-when-missing] > Error EINVAL: invalid command > > But there is no help available. > "ceph orch cert-store --help" does not return anything specific. > Searching for "cert-store" in the Ceph 19 docs also does not return anything. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Grafana certificate in version 19
by Robert Sander 07 Aug '26

07 Aug '26

Hi, how do I set the grafana certificate and key in Ceph 19 that is used on port 3000? The documentation on https://docs.ceph.com/en/squid/cephadm/services/monitoring/#configuring-ssl… states: # ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem # ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i $PWD/certificate.pem # ceph orch reconfig grafana But after doing that grafana still serves a self-signed certificate with the common name containing the hostname of the first MON (which is not correct). Digging into "ceph config-key dump" shows two entries named mgr/cephadm/cert_store.cert.grafana_cert and mgr/cephadm/cert_store.key.grafana_key which seem to cause this. AFAIK the certmgr is only available in Ceph 20. But the docs for this version look the same: https://docs.ceph.com/en/tentacle/cephadm/services/monitoring/#configuring-… Only the docs for Ceph 21 got an update and mention certmgr: https://docs.ceph.com/en/umbrella/cephadm/services/monitoring/#configuring-… The orchestrator in Ceph 19 seems to have something called cert-store: # ceph orch cert-store no valid command found; 4 closest matches: orch cert-store cert ls [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] orch cert-store key ls [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] orch cert-store get cert <entity> [--service_name <value>] [--hostname <value>] [--no-exception-when-missing] orch cert-store get key <entity> [--service_name <value>] [--hostname <value>] [--no-exception-when-missing] Error EINVAL: invalid command But there is no help available. "ceph orch cert-store --help" does not return anything specific. Searching for "cert-store" in the Ceph 19 docs also does not return anything. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: v20.2.3 Tentacle released
by Afreen 07 Aug '26

07 Aug '26

Hey, That seems like some issue in parsing the format and so far what we found out is that it could happen when CephFS cloning is used , are you using that ? And pools page is failing because its polling /summary API - fetching progress internally. You can try clearing progress and then refresh - "ceph progress clear". Its just clears the display events and does not stop operations. - Afreen On Thu, Aug 6, 2026 at 6:42 PM Salatiel Filho via ceph-users < ceph-users(a)ceph.io> wrote: > After upgrading from 20.2.2 to 20.2.3, the dashboard is partially > broken. List pools don't work. Error 500. > Also get error 500 popup as soon as I login to the dashboard. > > Logs show this: > 2026-08-06 10:04:23 2026-08-06T13:04:23.386+0000 7f5afcd91640 0 > [dashboard ERROR dashboard.services.exception] Internal Server Error > 2026-08-06 10:04:23 Traceback (most recent call last): > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in > dashboard_exception_handler > 2026-08-06 10:04:23 return handler(*args, **kwargs) > 2026-08-06 10:04:23 File > "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in > __call__ > 2026-08-06 10:04:23 return self.callable(*self.args, **self.kwargs) > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line > 264, in inner > 2026-08-06 10:04:23 ret = func(*args, **kwargs) > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/controllers/summary.py", line 105, in > __call__ > 2026-08-06 10:04:23 e, f = progress.get_progress_tasks() > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/services/progress.py", line 89, in > get_progress_tasks > 2026-08-06 10:04:23 > finished_t.append(_progress_event_to_dashboard_task(ev, True)) > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/services/progress.py", line 60, in > _progress_event_to_dashboard_task > 2026-08-06 10:04:23 _progress_event_to_dashboard_task_common(event, > task) > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/services/progress.py", line 52, in > _progress_event_to_dashboard_task_common > 2026-08-06 10:04:23 'metadata': dict(event.get('refs', {})), > 2026-08-06 10:04:23 ValueError: dictionary update sequence element > #0 has length 3; 2 is required > 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 > [dashboard ERROR dashboard.tools] [::ffff:10.175.143.9:53998] [GET] > [500] [0.009s] [admin] [513.0B] /api/summary > 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 > [dashboard ERROR dashboard.tools] [b'{"status": "500 Internal Server > Error", "detail": "The server encountered an unexpected condition > which prevented it from fulfilling the request.", "request_id": > "c7349113-44d8-4e48-88d6-3a469786397f"} > > > > '] > 2026-08-06 10:04:23 2026-08-06T13:04:23.494+0000 7f5afed95640 0 > [dashboard ERROR dashboard.controllers.frontend_logging] > (https://ceph.mydomain/#/pool) Http failure response for > https://ceph.mydomain/api/summary: 500 Internal Server Error > > Atenciosamente/Kind regards, > Salatiel > > On Wed, Aug 5, 2026 at 11:06 AM Yuri Weinstein via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > This is the third minor release in the Tentacle series. > > We recommend that all users update to this release. > > > > Notable Changes > > --------------- > > > > MDS (Metadata Server) > > --------------------- > > > > * Fixed a crash and shutdown hang that could occur when ephemeral pins > were > > active and max_mds was set to 0. > > * Session reclaim could miss blocklisting an old session; this is now > > handled correctly. > > * Fixed a case where scrub was unable to identify damage from an > > irreparable hard link. > > > > OSD (Object Storage Daemon) > > ---------------------------- > > > > * BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write > buffer > > misalignment. > > * Erasure Coding: Fixed truncate+write planning for EC shard sizes. > > * Fixed a condition that prevented rolling forward of PG log entries. > > * Prevented OSDMap::check_health() from asserting when new OSDs are found > > in a subtree. > > * Scrub: "repairing" scrubs are now allowed at all times. > > > > RGW (RADOS Gateway) > > -------------------- > > > > * Added the ssl_ciphersuites option for the Beast front-end to support > TLS > > 1.3 cipher customization. > > * Fixed PutObject's canned_acl comparison bug affecting BlockPublicAcls. > > * Fixed crashes occurring on realm reload. > > * Implemented CopyObject support for encrypted objects. > > * Added mTLS client-certificate authentication support for Kafka > > notifications. > > * Fixed removal of delete markers in lifecycle processing. > > > > RADOS / librados / neorados > > ---------------------------- > > > > * neorados: Fixed an infinite trim loop on empty data log shards. > > * Fixed cancellation-slot cleanup in librados/asio's associated executor. > > > > Dashboard > > ----------------- > > > > * Added a hardware-monitoring dashboard backed by node-proxy metrics. > > * Fixed a bind-address regression caused by CherryPy isolation. > > * Added support for adding hosts via CSV upload. > > * NVMeoF: multiple CLI and UI backports (EC pool support, IO statistics, > > role management). > > * Object: The global RGW Roles tab has been removed. Role management is > now > > scoped under RGW Accounts, where roles can be listed, created, edited, > and > > deleted for a selected account. > > > > ceph-volume > > ----------- > > > > * Reworked OSD mapper lifecycle handling (LVM + raw) for activate. > > * Improved detection of rotational media under dm-crypt to bypass the > > workqueue correctly. > > * Fixed OSD re-deployment issues with disk-selection filters and DB > devices. > > * Added retry handling for lvs when it returns an empty result or a > > "devices file is missing" error. > > > > NVMe-oF > > ------- > > > > * Added configurable delayed failback (default 0, no delay) to > accommodate > > initiators that recover paths with a delay. > > * Gateways in the DELETING state now ignore beacons and send empty maps > > instead of processing them. > > * rbd_with_crc32c is now enabled by default via cephadm. > > * Introduced a new NVMeoF mgr module. The module automatically creates > the > > ".nvmeof" metadata pool if it does not already exist. Gateway-group state > > files are stored in this pool unless the user specifies another pool. The > > module is enabled automatically on fresh installs; on upgrades from an > > older cluster, it must be enabled manually. > > > > RBD > > --- > > > > * Fixed a use-after-free bug releasing object map locks during deep copy. > > * Fixed memory leaks in PWL discard operations. > > * Obsolete primary mirror snapshots are now pruned after relocation. > > > > We recommend users to update to this release. > > For detailed release notes with links & changelog please refer to the > > official blog entry at > > https://ceph.io/en/news/blog/2026/v20-2-3-tentacle-released/ > > > > Getting Ceph > > ------------ > > * Git at git://github.com/ceph/ceph.git > > * Tarball at https://download.ceph.com/tarballs/ceph-20.2.3.tar.gz > > * Containers at https://quay.io/repository/ceph/ceph > > * For packages, see > https://docs.ceph.com/en/latest/install/get-packages/ > > * Release git sha1: 06c2f9c35b67055a8a6fb99d1be236b3c4832ace > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: RGW Account Quota in Multisite Setup
by Casey Bodley 07 Aug '26

07 Aug '26

On Thu, Aug 6, 2026 at 3:44 AM Danila Reznichuk via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I have a question about account quotas in an RGW multisite setup. > > My configuration is fairly simple: > > Main zonegroup with the master zone. > Archive zonegroup with two zones: A and B. > > I created an account on the master zone of the main zonegroup and set > the account quota (--quota-scope account) to 1 GiB. As expected, this > change was synchronized to all other zones in all zonegroups. > > Then, by mistake, I changed the quota for the same account on zone A of > the archive zonegroup to 2 GiB. This change was not synchronized to the > other zones. That seems reasonable to me and could even be considered a > feature, since different zonegroups may represent different locations > with different storage policies or usage patterns. > > After some testing, I found that updating the account quota again on > the master zone of the main zonegroup overwrites the locally modified > quota on zone A. > > So I have a few questions: > > Is this the intended behavior? > Is there a supported way to configure different account quotas for > different zones or zonegroups? > If not, is the account quota expected to be globally consistent across > the entire multisite deployment? the intent of multisite is to keep all metadata (accounts, users, buckets, etc) consistent across zones, with the metadata master zone as the source of truth. there are several radosgw-admin commands that can modify metadata on secondary zones, but they're supposed to print a warning message and require --yes-i-really-mean-it. that logic was missing from the quota commands, so i've raised https://tracker.ceph.com/issues/79120 to track the fix and its backports > > Thanks! > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by Robert Sander 07 Aug '26

07 Aug '26

Hi, Am 07.08.26 um 10:59 AM schrieb Frédéric Nass via ceph-users: > Or, if that's too much of a maintenance burden and you're worried you'll > forget that step, just install rpcbind on the NFS nodes and leave NFSv3 > enabled. Or maybe NFSv3 is made an option for the NFS service spec with the hint that then rpcbind has to be installed. Or the orchestrator installs rpcbind when the NFS service should offer NFSv3. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by Frédéric Nass 07 Aug '26

07 Aug '26

Hi everyone, Unfortunately, the custom config won't work. The custom config is loaded after ganesha.conf, and by then NFS_CORE_PARAM has already been parsed from the generated config. Ganesha doesn't merge settings between two blocks of the same type. The later is simply ignored, so your entire NFS_CORE_PARAM block from custom config would have no effect. We should probably update the documentation to list which blocks can actually be defined in the custom config, or just mention the above. As a workaround, download the Ganesha Jinja2 template that matches your Ceph version: curl -O https://github.com/ceph/ceph/blob/v20.2.2/src/pybind/mgr/cephadm/templates/… Edit it to disable NFSv3, store it as a cephadm override: ceph config-key set mgr/cephadm/services/nfs/ganesha.conf -i ./ganesha.conf.j2 Then redeploy the daemons: ceph orch redeploy nfs.<service_id> Make sure to update the template before upgrading the Ceph cluster. This file has changed across Ceph releases and is expected to keep changing in the future. Or, if that's too much of a maintenance burden and you're worried you'll forget that step, just install rpcbind on the NFS nodes and leave NFSv3 enabled. Hope that helps. Best regards, Frédéric -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> frederic.nass(a)clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Le mer. 5 août 2026, 15:40, Eugen Block via ceph-users <ceph-users(a)ceph.io> a écrit : > That's what the docs say, yes. > > > https://docs.ceph.com/en/latest/mgr/nfs/#set-customized-nfs-ganesha-configu… > > Zitat von wodel youchi <wodel.youchi(a)gmail.com>: > > > Hi, > > > > Thanks, to remove nfsv3 from ganesha's config, should I use : > > > > ceph nfs cluster config set <cluster_id> -i <config_file> > > > > > > with for example : > > > > vim <config_file> > > > > NFS_CORE_PARAM { > > * Protocols = 4; > > } > > > > Then > > # ceph orch restart nfs.tentacle-nfs > > > > Regards. > > > > Le mar. 4 août 2026 à 11:33, Eugen Block via ceph-users < > ceph-users(a)ceph.io> > > a écrit : > > > >> Hi, > >> > >> look at this thread [0] and the corresponding links provided in there > >> [1], [2]. > >> > >> Regards, > >> Eugen > >> > >> [0] > >> > >> > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LG2EGDZMIW… > >> [1] https://tracker.ceph.com/issues/74307 > >> [2] https://tracker.ceph.com/issues/74191 > >> > >> Zitat von wodel youchi via ceph-users <ceph-users(a)ceph.io>: > >> > >> > Hi, > >> > > >> > I'm using tentacle 20.2.2 with podman on Rocky Linux 9. > >> > I'm trying to configure nfs-ganesha, but the nfs-tentacle containers > >> won't > >> > stay up. > >> > I've 3 nodes, all services are collocated. > >> > > >> >> > >> >> # ceph nfs cluster create tentacle-nfs > "srvceph01,srvceph02,srvceph03" > >> > --ingress --virtual_ip 10.200.24.89 --ingress-mode haproxy-standard > >> --port > >> > 12049 > >> > > >> > > >> > When I create the nfs cluster (with HA, haproxy + keepalived). 3 > >> > nfs-tentacle-nfs containers are created, one on each node, but they do > >> not > >> > stay alive, they start then disappear, again and again, and in the end > >> just > >> > one is created, but it does not really work, HAProxy shows that all > >> > backends are down. > >> > > >> > What I've got from the vanished containers is that the nfs service is > >> > looking for rpc-bind which does not exist because ganesha's config > file > >> > enables both nfs v3 and v4. > >> > > >> > # ceph orch ps --service_name nfs.tentacle-nfs > >> >> NAME HOST PORTS STATUS > >> >> REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID > CONTAINER ID > >> >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown > >> >> 5m ago 6m - - <unknown> <unknown> <unknown> > >> >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running > (5m) > >> >> 5m ago 6m 15.0M - 5.9 c22719cb0d76 > dcb1eab045cd > >> >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running > (4m) > >> >> 4m ago 6m 14.6M - 5.9 c22719cb0d76 > b11d32f4b0cc > >> > > >> > > >> > > >> > # ceph orch ps > >> >> NAME HOST PORTS > >> >> STATUS REFRESHED AGE MEM USE MEM LIM VERSION > IMAGE > >> ID > >> >> CONTAINER ID > >> >> ... > >> >> haproxy.nfs.tentacle-nfs.srvceph01.uvueop srvceph01 > *:12049,19049 > >> >> running (6m) 4m ago 6m 11.5M - 2.3.17-d1c9119 > >> >> e85424b0d443 d4a840202ae0 > >> >> haproxy.nfs.tentacle-nfs.srvceph02.xujdir srvceph02 > *:12049,19049 > >> >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > >> >> e85424b0d443 747cf2052cd5 > >> >> haproxy.nfs.tentacle-nfs.srvceph03.snapkl srvceph03 > *:12049,19049 > >> >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > >> >> e85424b0d443 3c17cb2a4757 > >> >> keepalived.nfs.tentacle-nfs.srvceph01.phscfe srvceph01 > >> >> running (6m) 4m ago 6m 19.9M - 2.2.4 > >> >> 4a3a1ff181d9 d053e0b91210 > >> >> keepalived.nfs.tentacle-nfs.srvceph02.ubwgoo srvceph02 > >> >> running (6m) 5m ago 6m 20.1M - 2.2.4 > >> >> 4a3a1ff181d9 6ad75ccccaa0 > >> >> keepalived.nfs.tentacle-nfs.srvceph03.phbgis srvceph03 > >> >> running (6m) 5m ago 6m 20.1M - 2.2.4 > >> >> 4a3a1ff181d9 9efebd8d5ba5 > >> >> mds.cephfs.srvceph01.tyrask srvceph01 > >> >> running (17m) 4m ago 5d 32.4M - 20.2.2 > >> >> c22719cb0d76 2f89b9d86343 > >> >> mds.cephfs.srvceph02.ebyvdg srvceph02 > >> >> running (17m) 5m ago 5d 33.1M - 20.2.2 > >> >> c22719cb0d76 5afe19ec45b8 > >> >> mds.cephfs.srvceph03.miznhi srvceph03 > >> >> running (17m) 5m ago 5d 146M - 20.2.2 > >> >> c22719cb0d76 6ee9a906d887 > >> >> ... > >> >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 > >> >> unknown 5m ago 7m - - <unknown> > >> <unknown> > >> >> <unknown> > >> >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 > >> >> running (5m) 5m ago 7m 15.0M - 5.9 > >> >> c22719cb0d76 dcb1eab045cd > >> >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 > >> >> running (4m) 4m ago 7m 14.6M - 5.9 > >> >> c22719cb0d76 b11d32f4b0cc > >> > > >> > ... > >> > > >> > > >> > # podman logs nfs-tentacle-container > >> > > >> > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :grace reload client info completed from > >> > backend > >> > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > >> > count(0) > >> > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > rados_cluster_grace_enforcing :CLIENT ID :EVENT > >> > :rados_cluster_grace_enforcing: ret=-45 > >> > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > >> > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :grace reload client info completed from > >> > backend > >> > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > >> > count(0) > >> > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > rados_cluster_grace_enforcing :CLIENT ID :EVENT > >> > :rados_cluster_grace_enforcing: ret=-45 > >> > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > >> > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :grace reload client info completed from > >> > backend > >> > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > >> > count(0) > >> > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > rados_cluster_grace_enforcing :CLIENT ID :EVENT > >> > :rados_cluster_grace_enforcing: ret=-45 > >> > > >> > > >> > > >> > After > >> > > >> >> # ceph orch restart nfs.tentacle-nfs > >> > > >> > > >> > I got this > >> > > >> >> # podman logs nfs-tentacle-container > >> > > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > init_logging :LOG :NULL :LOG: Setting log level for all components to > >> > NIV_EVENT > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file > >> > successfully parsed > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed > limit > >> > on FDs to 1048576. > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully > initialized. > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > >> > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :grace reload client info completed from > >> > backend > >> > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > >> > count(0) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > rados_cluster_grace_enforcing :CLIENT ID :EVENT > >> > :rados_cluster_grace_enforcing: ret=0 > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > main :NFS STARTUP :WARN :No export entries found in configuration file > >> !!! > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > config_errs_to_log :CONFIG :WARN :Config File > >> > (/etc/ganesha/ganesha.conf:9): Unknown parameter > >> (allow_set_io_flusher_fail) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > config_errs_to_log :CONFIG :WARN :Config File > >> > (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > config_errs_to_log :CONFIG :WARN :Config File > >> > (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully > >> > removed for proper quota management in FSAL > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: > >> > > >> > cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect > to > >> > socket /run/dbus/system_bus_socket: No such file or directory) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > gsh_dbus_register_path :DBUS :CRIT > :dbus_connection_register_object_path > >> > called with no DBUS connection > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > gsh_dbus_register_path :DBUS :CRIT > :dbus_connection_register_object_path > >> > called with no DBUS connection > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > gsh_dbus_register_path :DBUS :CRIT > :dbus_connection_register_object_path > >> > called with no DBUS connection > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs > >> > *04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP > >> > <--------------------------* > >> > > >> > > >> > > >> > podman exec -it f2e2c668e0e3 bash > >> >> [root@srvceph01 /]# cat etc/ganesha/ganesha.conf > >> >> # This file is generated by cephadm. > >> >> NFS_CORE_PARAM { > >> >> Enable_NLM = false; > >> >> Enable_RQUOTA = false; > >> >> * Protocols = 3, 4; > >> >> <---------------------------------------------------* > >> >> mount_path_pseudo = true; > >> >> Enable_UDP = false; > >> >> NFS_Port = 22049; > >> >> allow_set_io_flusher_fail = true; > >> >> Monitoring_Port = 9587; > >> >> } > >> >> NFSv4 { > >> >> Delegations = false; > >> >> RecoveryBackend = "rados_cluster"; > >> >> Minor_Versions = 1, 2; > >> >> Server_Scope = > >> "bcf27a48-8a8d-11f1-bdd3-fc607038078a-tentacle-nfs"; > >> >> IdmapConf = "/etc/ganesha/idmap.conf"; > >> >> } > >> >> RADOS_KV { > >> >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > >> >> nodeid = 2; > >> >> pool = ".nfs"; > >> >> namespace = "tentacle-nfs"; > >> >> } > >> >> RADOS_URLS { > >> >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > >> >> watch_url = > "rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs"; > >> >> } > >> >> RGW { > >> >> cluster = "ceph"; > >> >> name = "client.nfs.tentacle-nfs.2.0.srvceph01.ahcayv-rgw"; > >> >> } > >> >> %url rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs > >> > > >> > > >> > > >> > I am not sure if this is the cause of the problem, so I thought, why > not > >> > enable nfsv3 even if I'm not going to use it? but... > >> > Ceph's documentation mentions (tentacle version selected) : > >> > https://docs.ceph.com/en/tentacle/mgr/nfs/#create-nfs-ganesha-cluster > >> > Create NFS Ganesha Cluster > >> > > >> > ceph nfs cluster create <cluster_id> .... [--enable-nfsv3] > >> > > >> > The --enable-nfsv3 is not recognized, I've tried to use it to enable > >> > nfsv3 (as a workaround), but it is not recognized. > >> > > >> > I've also tried to use this from > >> > > https://docs.ceph.com/en/tentacle/cephadm/services/nfs/#cephadm-ha-nfs > >> > : > >> > > >> > Service Specification > >> > > >> > service_type: nfsservice_id: mynfsplacement: hosts: - host1 - > >> > host2spec: port: 12345 monitoring_port: 567 enable_nfsv3: true > >> > <------------------- not recognized > >> > > >> > The enable_nfsv3 is not recognized. > >> > > >> > If this is the cause of the problem, then I'm stuck, I don't know how > to > >> > disable nfsv3 on ganesha nor how to enable nfsv3 with cephadm !!!!! > >> > > >> > > >> > Regards. > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users(a)ceph.io > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: RGW Account Quota in Multisite Setup
by Natanam Sethu 07 Aug '26

07 Aug '26

Hi Casey, If you are using the bi-directional replication. When ever you modify the quota on Zone A. It will modify the master Zone as well. If you one use way replication. I.e that is from the master zone to zone A, it will not Regards Natanam Sethu On Thu, 6 Aug 2026 at 18:49, Casey Bodley via ceph-users <ceph-users(a)ceph.io> wrote: > On Thu, Aug 6, 2026 at 3:44 AM Danila Reznichuk via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > Hi, > > > > I have a question about account quotas in an RGW multisite setup. > > > > My configuration is fairly simple: > > > > Main zonegroup with the master zone. > > Archive zonegroup with two zones: A and B. > > > > I created an account on the master zone of the main zonegroup and set > > the account quota (--quota-scope account) to 1 GiB. As expected, this > > change was synchronized to all other zones in all zonegroups. > > > > Then, by mistake, I changed the quota for the same account on zone A of > > the archive zonegroup to 2 GiB. This change was not synchronized to the > > other zones. That seems reasonable to me and could even be considered a > > feature, since different zonegroups may represent different locations > > with different storage policies or usage patterns. > > > > After some testing, I found that updating the account quota again on > > the master zone of the main zonegroup overwrites the locally modified > > quota on zone A. > > > > So I have a few questions: > > > > Is this the intended behavior? > > Is there a supported way to configure different account quotas for > > different zones or zonegroups? > > If not, is the account quota expected to be globally consistent across > > the entire multisite deployment? > > the intent of multisite is to keep all metadata (accounts, users, > buckets, etc) consistent across zones, with the metadata master zone > as the source of truth. there are several radosgw-admin commands that > can modify metadata on secondary zones, but they're supposed to print > a warning message and require --yes-i-really-mean-it. that logic was > missing from the quota commands, so i've raised > https://tracker.ceph.com/issues/79120 to track the fix and its > backports > > > > > Thanks! > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: v20.2.3 Tentacle released
by Salatiel Filho 06 Aug '26

06 Aug '26

After upgrading from 20.2.2 to 20.2.3, the dashboard is partially broken. List pools don't work. Error 500. Also get error 500 popup as soon as I login to the dashboard. Logs show this: 2026-08-06 10:04:23 2026-08-06T13:04:23.386+0000 7f5afcd91640 0 [dashboard ERROR dashboard.services.exception] Internal Server Error 2026-08-06 10:04:23 Traceback (most recent call last): 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler 2026-08-06 10:04:23 return handler(*args, **kwargs) 2026-08-06 10:04:23 File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ 2026-08-06 10:04:23 return self.callable(*self.args, **self.kwargs) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 264, in inner 2026-08-06 10:04:23 ret = func(*args, **kwargs) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/controllers/summary.py", line 105, in __call__ 2026-08-06 10:04:23 e, f = progress.get_progress_tasks() 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 89, in get_progress_tasks 2026-08-06 10:04:23 finished_t.append(_progress_event_to_dashboard_task(ev, True)) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 60, in _progress_event_to_dashboard_task 2026-08-06 10:04:23 _progress_event_to_dashboard_task_common(event, task) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 52, in _progress_event_to_dashboard_task_common 2026-08-06 10:04:23 'metadata': dict(event.get('refs', {})), 2026-08-06 10:04:23 ValueError: dictionary update sequence element #0 has length 3; 2 is required 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 [dashboard ERROR dashboard.tools] [::ffff:10.175.143.9:53998] [GET] [500] [0.009s] [admin] [513.0B] /api/summary 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 [dashboard ERROR dashboard.tools] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "c7349113-44d8-4e48-88d6-3a469786397f"} '] 2026-08-06 10:04:23 2026-08-06T13:04:23.494+0000 7f5afed95640 0 [dashboard ERROR dashboard.controllers.frontend_logging] (https://ceph.mydomain/#/pool) Http failure response for https://ceph.mydomain/api/summary: 500 Internal Server Error Atenciosamente/Kind regards, Salatiel On Wed, Aug 5, 2026 at 11:06 AM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> wrote: > > This is the third minor release in the Tentacle series. > We recommend that all users update to this release. > > Notable Changes > --------------- > > MDS (Metadata Server) > --------------------- > > * Fixed a crash and shutdown hang that could occur when ephemeral pins were > active and max_mds was set to 0. > * Session reclaim could miss blocklisting an old session; this is now > handled correctly. > * Fixed a case where scrub was unable to identify damage from an > irreparable hard link. > > OSD (Object Storage Daemon) > ---------------------------- > > * BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write buffer > misalignment. > * Erasure Coding: Fixed truncate+write planning for EC shard sizes. > * Fixed a condition that prevented rolling forward of PG log entries. > * Prevented OSDMap::check_health() from asserting when new OSDs are found > in a subtree. > * Scrub: "repairing" scrubs are now allowed at all times. > > RGW (RADOS Gateway) > -------------------- > > * Added the ssl_ciphersuites option for the Beast front-end to support TLS > 1.3 cipher customization. > * Fixed PutObject's canned_acl comparison bug affecting BlockPublicAcls. > * Fixed crashes occurring on realm reload. > * Implemented CopyObject support for encrypted objects. > * Added mTLS client-certificate authentication support for Kafka > notifications. > * Fixed removal of delete markers in lifecycle processing. > > RADOS / librados / neorados > ---------------------------- > > * neorados: Fixed an infinite trim loop on empty data log shards. > * Fixed cancellation-slot cleanup in librados/asio's associated executor. > > Dashboard > ----------------- > > * Added a hardware-monitoring dashboard backed by node-proxy metrics. > * Fixed a bind-address regression caused by CherryPy isolation. > * Added support for adding hosts via CSV upload. > * NVMeoF: multiple CLI and UI backports (EC pool support, IO statistics, > role management). > * Object: The global RGW Roles tab has been removed. Role management is now > scoped under RGW Accounts, where roles can be listed, created, edited, and > deleted for a selected account. > > ceph-volume > ----------- > > * Reworked OSD mapper lifecycle handling (LVM + raw) for activate. > * Improved detection of rotational media under dm-crypt to bypass the > workqueue correctly. > * Fixed OSD re-deployment issues with disk-selection filters and DB devices. > * Added retry handling for lvs when it returns an empty result or a > "devices file is missing" error. > > NVMe-oF > ------- > > * Added configurable delayed failback (default 0, no delay) to accommodate > initiators that recover paths with a delay. > * Gateways in the DELETING state now ignore beacons and send empty maps > instead of processing them. > * rbd_with_crc32c is now enabled by default via cephadm. > * Introduced a new NVMeoF mgr module. The module automatically creates the > ".nvmeof" metadata pool if it does not already exist. Gateway-group state > files are stored in this pool unless the user specifies another pool. The > module is enabled automatically on fresh installs; on upgrades from an > older cluster, it must be enabled manually. > > RBD > --- > > * Fixed a use-after-free bug releasing object map locks during deep copy. > * Fixed memory leaks in PWL discard operations. > * Obsolete primary mirror snapshots are now pruned after relocation. > > We recommend users to update to this release. > For detailed release notes with links & changelog please refer to the > official blog entry at > https://ceph.io/en/news/blog/2026/v20-2-3-tentacle-released/ > > Getting Ceph > ------------ > * Git at git://github.com/ceph/ceph.git > * Tarball at https://download.ceph.com/tarballs/ceph-20.2.3.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * Release git sha1: 06c2f9c35b67055a8a6fb99d1be236b3c4832ace > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

RGW Account Quota in Multisite Setup
by dreznichuk＠gmail.com 06 Aug '26

06 Aug '26

Hi, I have a question about account quotas in an RGW multisite setup. My configuration is fairly simple: Main zonegroup with the master zone. Archive zonegroup with two zones: A and B. I created an account on the master zone of the main zonegroup and set the account quota (--quota-scope account) to 1 GiB. As expected, this change was synchronized to all other zones in all zonegroups. Then, by mistake, I changed the quota for the same account on zone A of the archive zonegroup to 2 GiB. This change was not synchronized to the other zones. That seems reasonable to me and could even be considered a feature, since different zonegroups may represent different locations with different storage policies or usage patterns. After some testing, I found that updating the account quota again on the master zone of the main zonegroup overwrites the locally modified quota on zone A. So I have a few questions: Is this the intended behavior? Is there a supported way to configure different account quotas for different zones or zonegroups? If not, is the account quota expected to be globally consistent across the entire multisite deployment? Thanks!

1 0

Affects of high pg count on OSD peering
by Elias Carter 05 Aug '26

05 Aug '26

We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total PGs and upwards). Something that I have noticed is that CRUSH changes (marking an OSD as out, or a new OSD joining the cluster) can cause many hundreds of PGs to remap and peer. During this time we get PG_AVAILABILITY health check warnings as the OSDs need to peer with their new PGs before they can serve IOs. RADOS tail latency suffers because IOs sent to a peering PG are blocked until peering completes. I believe that the more PGs we have, the more PGs need to peer, and the longer IOs are blocked for the pool. Does anyone have suggestions on how to deal with PG peering blocking IOs? I was thinking about gradually re-weighting OSDs instead of marking them out or in, but that can result in multiple redundant data moves. Elias

1 0

v20.2.3 Tentacle released
by Yuri Weinstein 05 Aug '26

05 Aug '26

This is the third minor release in the Tentacle series. We recommend that all users update to this release. Notable Changes --------------- MDS (Metadata Server) --------------------- * Fixed a crash and shutdown hang that could occur when ephemeral pins were active and max_mds was set to 0. * Session reclaim could miss blocklisting an old session; this is now handled correctly. * Fixed a case where scrub was unable to identify damage from an irreparable hard link. OSD (Object Storage Daemon) ---------------------------- * BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write buffer misalignment. * Erasure Coding: Fixed truncate+write planning for EC shard sizes. * Fixed a condition that prevented rolling forward of PG log entries. * Prevented OSDMap::check_health() from asserting when new OSDs are found in a subtree. * Scrub: "repairing" scrubs are now allowed at all times. RGW (RADOS Gateway) -------------------- * Added the ssl_ciphersuites option for the Beast front-end to support TLS 1.3 cipher customization. * Fixed PutObject's canned_acl comparison bug affecting BlockPublicAcls. * Fixed crashes occurring on realm reload. * Implemented CopyObject support for encrypted objects. * Added mTLS client-certificate authentication support for Kafka notifications. * Fixed removal of delete markers in lifecycle processing. RADOS / librados / neorados ---------------------------- * neorados: Fixed an infinite trim loop on empty data log shards. * Fixed cancellation-slot cleanup in librados/asio's associated executor. Dashboard ----------------- * Added a hardware-monitoring dashboard backed by node-proxy metrics. * Fixed a bind-address regression caused by CherryPy isolation. * Added support for adding hosts via CSV upload. * NVMeoF: multiple CLI and UI backports (EC pool support, IO statistics, role management). * Object: The global RGW Roles tab has been removed. Role management is now scoped under RGW Accounts, where roles can be listed, created, edited, and deleted for a selected account. ceph-volume ----------- * Reworked OSD mapper lifecycle handling (LVM + raw) for activate. * Improved detection of rotational media under dm-crypt to bypass the workqueue correctly. * Fixed OSD re-deployment issues with disk-selection filters and DB devices. * Added retry handling for lvs when it returns an empty result or a "devices file is missing" error. NVMe-oF ------- * Added configurable delayed failback (default 0, no delay) to accommodate initiators that recover paths with a delay. * Gateways in the DELETING state now ignore beacons and send empty maps instead of processing them. * rbd_with_crc32c is now enabled by default via cephadm. * Introduced a new NVMeoF mgr module. The module automatically creates the ".nvmeof" metadata pool if it does not already exist. Gateway-group state files are stored in this pool unless the user specifies another pool. The module is enabled automatically on fresh installs; on upgrades from an older cluster, it must be enabled manually. RBD --- * Fixed a use-after-free bug releasing object map locks during deep copy. * Fixed memory leaks in PWL discard operations. * Obsolete primary mirror snapshots are now pruned after relocation. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-3-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.2.3.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 06c2f9c35b67055a8a6fb99d1be236b3c4832ace

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Stefan Möding 05 Aug '26

05 Aug '26

Hi, Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > I haven't tried it myself yet so I don't really have an answer. But > I noticed that your OSD caps may be incomplete wrt x permissions: >> osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, >> allow rw namespace=k8s-test tag cephfs data=root' > The second allow statement doesn't have an "x". Maybe that's the issue > why the workaround fails? We tried that, but the error remains. We now consider to use the following capabilities in combination to creating a dedicated data pool for each tenant (K8s cluster): mon_caps: 'allow r fsname=root' mgr_caps: 'allow rw' osd_caps: 'allow rwx pool=cephfs.meta, allow rw pool=cephfs.data, allow rw pool=cephfs.k8s-test' mds_caps: 'allow r fsname=root path=/volumes, allow rws fsname=root path=/volumes/k8s-test' CephFS is created with cephfs.meta as metadata pool and cephfs.data as default data pool while cephfs.k8s-test is the pool for this tenant. What are the implications to share the metadata/default data pools and have no namespace? Is the MDS capability sufficient to separate tenants or can everyone access the metadata objects of everyone else? -- Stefan

1 0

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by wodel youchi 05 Aug '26

05 Aug '26

Hi, Thanks, to remove nfsv3 from ganesha's config, should I use : ceph nfs cluster config set <cluster_id> -i <config_file> with for example : vim <config_file> NFS_CORE_PARAM { * Protocols = 4; } Then # ceph orch restart nfs.tentacle-nfs Regards. Le mar. 4 août 2026 à 11:33, Eugen Block via ceph-users <ceph-users(a)ceph.io> a écrit : > Hi, > > look at this thread [0] and the corresponding links provided in there > [1], [2]. > > Regards, > Eugen > > [0] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LG2EGDZMIW… > [1] https://tracker.ceph.com/issues/74307 > [2] https://tracker.ceph.com/issues/74191 > > Zitat von wodel youchi via ceph-users <ceph-users(a)ceph.io>: > > > Hi, > > > > I'm using tentacle 20.2.2 with podman on Rocky Linux 9. > > I'm trying to configure nfs-ganesha, but the nfs-tentacle containers > won't > > stay up. > > I've 3 nodes, all services are collocated. > > > >> > >> # ceph nfs cluster create tentacle-nfs "srvceph01,srvceph02,srvceph03" > > --ingress --virtual_ip 10.200.24.89 --ingress-mode haproxy-standard > --port > > 12049 > > > > > > When I create the nfs cluster (with HA, haproxy + keepalived). 3 > > nfs-tentacle-nfs containers are created, one on each node, but they do > not > > stay alive, they start then disappear, again and again, and in the end > just > > one is created, but it does not really work, HAProxy shows that all > > backends are down. > > > > What I've got from the vanished containers is that the nfs service is > > looking for rpc-bind which does not exist because ganesha's config file > > enables both nfs v3 and v4. > > > > # ceph orch ps --service_name nfs.tentacle-nfs > >> NAME HOST PORTS STATUS > >> REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID > >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown > >> 5m ago 6m - - <unknown> <unknown> <unknown> > >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running (5m) > >> 5m ago 6m 15.0M - 5.9 c22719cb0d76 dcb1eab045cd > >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running (4m) > >> 4m ago 6m 14.6M - 5.9 c22719cb0d76 b11d32f4b0cc > > > > > > > > # ceph orch ps > >> NAME HOST PORTS > >> STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE > ID > >> CONTAINER ID > >> ... > >> haproxy.nfs.tentacle-nfs.srvceph01.uvueop srvceph01 *:12049,19049 > >> running (6m) 4m ago 6m 11.5M - 2.3.17-d1c9119 > >> e85424b0d443 d4a840202ae0 > >> haproxy.nfs.tentacle-nfs.srvceph02.xujdir srvceph02 *:12049,19049 > >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > >> e85424b0d443 747cf2052cd5 > >> haproxy.nfs.tentacle-nfs.srvceph03.snapkl srvceph03 *:12049,19049 > >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > >> e85424b0d443 3c17cb2a4757 > >> keepalived.nfs.tentacle-nfs.srvceph01.phscfe srvceph01 > >> running (6m) 4m ago 6m 19.9M - 2.2.4 > >> 4a3a1ff181d9 d053e0b91210 > >> keepalived.nfs.tentacle-nfs.srvceph02.ubwgoo srvceph02 > >> running (6m) 5m ago 6m 20.1M - 2.2.4 > >> 4a3a1ff181d9 6ad75ccccaa0 > >> keepalived.nfs.tentacle-nfs.srvceph03.phbgis srvceph03 > >> running (6m) 5m ago 6m 20.1M - 2.2.4 > >> 4a3a1ff181d9 9efebd8d5ba5 > >> mds.cephfs.srvceph01.tyrask srvceph01 > >> running (17m) 4m ago 5d 32.4M - 20.2.2 > >> c22719cb0d76 2f89b9d86343 > >> mds.cephfs.srvceph02.ebyvdg srvceph02 > >> running (17m) 5m ago 5d 33.1M - 20.2.2 > >> c22719cb0d76 5afe19ec45b8 > >> mds.cephfs.srvceph03.miznhi srvceph03 > >> running (17m) 5m ago 5d 146M - 20.2.2 > >> c22719cb0d76 6ee9a906d887 > >> ... > >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 > >> unknown 5m ago 7m - - <unknown> > <unknown> > >> <unknown> > >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 > >> running (5m) 5m ago 7m 15.0M - 5.9 > >> c22719cb0d76 dcb1eab045cd > >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 > >> running (4m) 4m ago 7m 14.6M - 5.9 > >> c22719cb0d76 b11d32f4b0cc > > > > ... > > > > > > # podman logs nfs-tentacle-container > > > > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :grace reload client info completed from > > backend > > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > > count(0) > > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > rados_cluster_grace_enforcing :CLIENT ID :EVENT > > :rados_cluster_grace_enforcing: ret=-45 > > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :grace reload client info completed from > > backend > > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > > count(0) > > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > rados_cluster_grace_enforcing :CLIENT ID :EVENT > > :rados_cluster_grace_enforcing: ret=-45 > > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :grace reload client info completed from > > backend > > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > > count(0) > > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > rados_cluster_grace_enforcing :CLIENT ID :EVENT > > :rados_cluster_grace_enforcing: ret=-45 > > > > > > > > After > > > >> # ceph orch restart nfs.tentacle-nfs > > > > > > I got this > > > >> # podman logs nfs-tentacle-container > > > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > init_logging :LOG :NULL :LOG: Setting log level for all components to > > NIV_EVENT > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file > > successfully parsed > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit > > on FDs to 1048576. > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :grace reload client info completed from > > backend > > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > > count(0) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > rados_cluster_grace_enforcing :CLIENT ID :EVENT > > :rados_cluster_grace_enforcing: ret=0 > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > main :NFS STARTUP :WARN :No export entries found in configuration file > !!! > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > config_errs_to_log :CONFIG :WARN :Config File > > (/etc/ganesha/ganesha.conf:9): Unknown parameter > (allow_set_io_flusher_fail) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > config_errs_to_log :CONFIG :WARN :Config File > > (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > config_errs_to_log :CONFIG :WARN :Config File > > (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully > > removed for proper quota management in FSAL > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: > > > cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to > > socket /run/dbus/system_bus_socket: No such file or directory) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > > called with no DBUS connection > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > > called with no DBUS connection > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > > called with no DBUS connection > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs > > *04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP > > <--------------------------* > > > > > > > > podman exec -it f2e2c668e0e3 bash > >> [root@srvceph01 /]# cat etc/ganesha/ganesha.conf > >> # This file is generated by cephadm. > >> NFS_CORE_PARAM { > >> Enable_NLM = false; > >> Enable_RQUOTA = false; > >> * Protocols = 3, 4; > >> <---------------------------------------------------* > >> mount_path_pseudo = true; > >> Enable_UDP = false; > >> NFS_Port = 22049; > >> allow_set_io_flusher_fail = true; > >> Monitoring_Port = 9587; > >> } > >> NFSv4 { > >> Delegations = false; > >> RecoveryBackend = "rados_cluster"; > >> Minor_Versions = 1, 2; > >> Server_Scope = > "bcf27a48-8a8d-11f1-bdd3-fc607038078a-tentacle-nfs"; > >> IdmapConf = "/etc/ganesha/idmap.conf"; > >> } > >> RADOS_KV { > >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > >> nodeid = 2; > >> pool = ".nfs"; > >> namespace = "tentacle-nfs"; > >> } > >> RADOS_URLS { > >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > >> watch_url = "rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs"; > >> } > >> RGW { > >> cluster = "ceph"; > >> name = "client.nfs.tentacle-nfs.2.0.srvceph01.ahcayv-rgw"; > >> } > >> %url rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs > > > > > > > > I am not sure if this is the cause of the problem, so I thought, why not > > enable nfsv3 even if I'm not going to use it? but... > > Ceph's documentation mentions (tentacle version selected) : > > https://docs.ceph.com/en/tentacle/mgr/nfs/#create-nfs-ganesha-cluster > > Create NFS Ganesha Cluster > > > > ceph nfs cluster create <cluster_id> .... [--enable-nfsv3] > > > > The --enable-nfsv3 is not recognized, I've tried to use it to enable > > nfsv3 (as a workaround), but it is not recognized. > > > > I've also tried to use this from > > https://docs.ceph.com/en/tentacle/cephadm/services/nfs/#cephadm-ha-nfs > > : > > > > Service Specification > > > > service_type: nfsservice_id: mynfsplacement: hosts: - host1 - > > host2spec: port: 12345 monitoring_port: 567 enable_nfsv3: true > > <------------------- not recognized > > > > The enable_nfsv3 is not recognized. > > > > If this is the cause of the problem, then I'm stuck, I don't know how to > > disable nfsv3 on ganesha nor how to enable nfsv3 with cephadm !!!!! > > > > > > Regards. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by Eugen Block 04 Aug '26

04 Aug '26

Hi, look at this thread [0] and the corresponding links provided in there [1], [2]. Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LG2EGDZMIW… [1] https://tracker.ceph.com/issues/74307 [2] https://tracker.ceph.com/issues/74191 Zitat von wodel youchi via ceph-users <ceph-users(a)ceph.io>: > Hi, > > I'm using tentacle 20.2.2 with podman on Rocky Linux 9. > I'm trying to configure nfs-ganesha, but the nfs-tentacle containers won't > stay up. > I've 3 nodes, all services are collocated. > >> >> # ceph nfs cluster create tentacle-nfs "srvceph01,srvceph02,srvceph03" > --ingress --virtual_ip 10.200.24.89 --ingress-mode haproxy-standard --port > 12049 > > > When I create the nfs cluster (with HA, haproxy + keepalived). 3 > nfs-tentacle-nfs containers are created, one on each node, but they do not > stay alive, they start then disappear, again and again, and in the end just > one is created, but it does not really work, HAProxy shows that all > backends are down. > > What I've got from the vanished containers is that the nfs service is > looking for rpc-bind which does not exist because ganesha's config file > enables both nfs v3 and v4. > > # ceph orch ps --service_name nfs.tentacle-nfs >> NAME HOST PORTS STATUS >> REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown >> 5m ago 6m - - <unknown> <unknown> <unknown> >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running (5m) >> 5m ago 6m 15.0M - 5.9 c22719cb0d76 dcb1eab045cd >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running (4m) >> 4m ago 6m 14.6M - 5.9 c22719cb0d76 b11d32f4b0cc > > > > # ceph orch ps >> NAME HOST PORTS >> STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID >> CONTAINER ID >> ... >> haproxy.nfs.tentacle-nfs.srvceph01.uvueop srvceph01 *:12049,19049 >> running (6m) 4m ago 6m 11.5M - 2.3.17-d1c9119 >> e85424b0d443 d4a840202ae0 >> haproxy.nfs.tentacle-nfs.srvceph02.xujdir srvceph02 *:12049,19049 >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 >> e85424b0d443 747cf2052cd5 >> haproxy.nfs.tentacle-nfs.srvceph03.snapkl srvceph03 *:12049,19049 >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 >> e85424b0d443 3c17cb2a4757 >> keepalived.nfs.tentacle-nfs.srvceph01.phscfe srvceph01 >> running (6m) 4m ago 6m 19.9M - 2.2.4 >> 4a3a1ff181d9 d053e0b91210 >> keepalived.nfs.tentacle-nfs.srvceph02.ubwgoo srvceph02 >> running (6m) 5m ago 6m 20.1M - 2.2.4 >> 4a3a1ff181d9 6ad75ccccaa0 >> keepalived.nfs.tentacle-nfs.srvceph03.phbgis srvceph03 >> running (6m) 5m ago 6m 20.1M - 2.2.4 >> 4a3a1ff181d9 9efebd8d5ba5 >> mds.cephfs.srvceph01.tyrask srvceph01 >> running (17m) 4m ago 5d 32.4M - 20.2.2 >> c22719cb0d76 2f89b9d86343 >> mds.cephfs.srvceph02.ebyvdg srvceph02 >> running (17m) 5m ago 5d 33.1M - 20.2.2 >> c22719cb0d76 5afe19ec45b8 >> mds.cephfs.srvceph03.miznhi srvceph03 >> running (17m) 5m ago 5d 146M - 20.2.2 >> c22719cb0d76 6ee9a906d887 >> ... >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 >> unknown 5m ago 7m - - <unknown> <unknown> >> <unknown> >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 >> running (5m) 5m ago 7m 15.0M - 5.9 >> c22719cb0d76 dcb1eab045cd >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 >> running (4m) 4m ago 7m 14.6M - 5.9 >> c22719cb0d76 b11d32f4b0cc > > ... > > > # podman logs nfs-tentacle-container > > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :grace reload client info completed from > backend > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > count(0) > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > rados_cluster_grace_enforcing :CLIENT ID :EVENT > :rados_cluster_grace_enforcing: ret=-45 > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :grace reload client info completed from > backend > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > count(0) > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > rados_cluster_grace_enforcing :CLIENT ID :EVENT > :rados_cluster_grace_enforcing: ret=-45 > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :grace reload client info completed from > backend > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > count(0) > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > rados_cluster_grace_enforcing :CLIENT ID :EVENT > :rados_cluster_grace_enforcing: ret=-45 > > > > After > >> # ceph orch restart nfs.tentacle-nfs > > > I got this > >> # podman logs nfs-tentacle-container > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > init_logging :LOG :NULL :LOG: Setting log level for all components to > NIV_EVENT > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file > successfully parsed > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit > on FDs to 1048576. > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :grace reload client info completed from > backend > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > count(0) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > rados_cluster_grace_enforcing :CLIENT ID :EVENT > :rados_cluster_grace_enforcing: ret=0 > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > main :NFS STARTUP :WARN :No export entries found in configuration file !!! > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > config_errs_to_log :CONFIG :WARN :Config File > (/etc/ganesha/ganesha.conf:9): Unknown parameter (allow_set_io_flusher_fail) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > config_errs_to_log :CONFIG :WARN :Config File > (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > config_errs_to_log :CONFIG :WARN :Config File > (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully > removed for proper quota management in FSAL > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: > cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to > socket /run/dbus/system_bus_socket: No such file or directory) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > called with no DBUS connection > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > called with no DBUS connection > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > called with no DBUS connection > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs > *04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP > <--------------------------* > > > > podman exec -it f2e2c668e0e3 bash >> [root@srvceph01 /]# cat etc/ganesha/ganesha.conf >> # This file is generated by cephadm. >> NFS_CORE_PARAM { >> Enable_NLM = false; >> Enable_RQUOTA = false; >> * Protocols = 3, 4; >> <---------------------------------------------------* >> mount_path_pseudo = true; >> Enable_UDP = false; >> NFS_Port = 22049; >> allow_set_io_flusher_fail = true; >> Monitoring_Port = 9587; >> } >> NFSv4 { >> Delegations = false; >> RecoveryBackend = "rados_cluster"; >> Minor_Versions = 1, 2; >> Server_Scope = "bcf27a48-8a8d-11f1-bdd3-fc607038078a-tentacle-nfs"; >> IdmapConf = "/etc/ganesha/idmap.conf"; >> } >> RADOS_KV { >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; >> nodeid = 2; >> pool = ".nfs"; >> namespace = "tentacle-nfs"; >> } >> RADOS_URLS { >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; >> watch_url = "rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs"; >> } >> RGW { >> cluster = "ceph"; >> name = "client.nfs.tentacle-nfs.2.0.srvceph01.ahcayv-rgw"; >> } >> %url rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs > > > > I am not sure if this is the cause of the problem, so I thought, why not > enable nfsv3 even if I'm not going to use it? but... > Ceph's documentation mentions (tentacle version selected) : > https://docs.ceph.com/en/tentacle/mgr/nfs/#create-nfs-ganesha-cluster > Create NFS Ganesha Cluster > > ceph nfs cluster create <cluster_id> .... [--enable-nfsv3] > > The --enable-nfsv3 is not recognized, I've tried to use it to enable > nfsv3 (as a workaround), but it is not recognized. > > I've also tried to use this from > https://docs.ceph.com/en/tentacle/cephadm/services/nfs/#cephadm-ha-nfs > : > > Service Specification > > service_type: nfsservice_id: mynfsplacement: hosts: - host1 - > host2spec: port: 12345 monitoring_port: 567 enable_nfsv3: true > <------------------- not recognized > > The enable_nfsv3 is not recognized. > > If this is the cause of the problem, then I'm stuck, I don't know how to > disable nfsv3 on ganesha nor how to enable nfsv3 with cephadm !!!!! > > > Regards. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

CERTIFICATE_VERIFY_FAILED ?
by listy 04 Aug '26

04 Aug '26

Hi guys I'm thinking I've my alerts configured a ok yet clusters warns: ... CEPH status is: HEALTH_WARN unable to send alert email [WRN] ALERTS_SMTP_ERROR: unable to send alert email [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:998) ... Do you know what I might be missing or getting something wrong? many thanks, L.

1 0

[Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by wodel youchi 04 Aug '26

04 Aug '26

Hi, I'm using tentacle 20.2.2 with podman on Rocky Linux 9. I'm trying to configure nfs-ganesha, but the nfs-tentacle containers won't stay up. I've 3 nodes, all services are collocated. > > # ceph nfs cluster create tentacle-nfs "srvceph01,srvceph02,srvceph03" --ingress --virtual_ip 10.200.24.89 --ingress-mode haproxy-standard --port 12049 When I create the nfs cluster (with HA, haproxy + keepalived). 3 nfs-tentacle-nfs containers are created, one on each node, but they do not stay alive, they start then disappear, again and again, and in the end just one is created, but it does not really work, HAProxy shows that all backends are down. What I've got from the vanished containers is that the nfs service is looking for rpc-bind which does not exist because ganesha's config file enables both nfs v3 and v4. # ceph orch ps --service_name nfs.tentacle-nfs > NAME HOST PORTS STATUS > REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID > nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown > 5m ago 6m - - <unknown> <unknown> <unknown> > nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running (5m) > 5m ago 6m 15.0M - 5.9 c22719cb0d76 dcb1eab045cd > nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running (4m) > 4m ago 6m 14.6M - 5.9 c22719cb0d76 b11d32f4b0cc # ceph orch ps > NAME HOST PORTS > STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID > CONTAINER ID > ... > haproxy.nfs.tentacle-nfs.srvceph01.uvueop srvceph01 *:12049,19049 > running (6m) 4m ago 6m 11.5M - 2.3.17-d1c9119 > e85424b0d443 d4a840202ae0 > haproxy.nfs.tentacle-nfs.srvceph02.xujdir srvceph02 *:12049,19049 > running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > e85424b0d443 747cf2052cd5 > haproxy.nfs.tentacle-nfs.srvceph03.snapkl srvceph03 *:12049,19049 > running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > e85424b0d443 3c17cb2a4757 > keepalived.nfs.tentacle-nfs.srvceph01.phscfe srvceph01 > running (6m) 4m ago 6m 19.9M - 2.2.4 > 4a3a1ff181d9 d053e0b91210 > keepalived.nfs.tentacle-nfs.srvceph02.ubwgoo srvceph02 > running (6m) 5m ago 6m 20.1M - 2.2.4 > 4a3a1ff181d9 6ad75ccccaa0 > keepalived.nfs.tentacle-nfs.srvceph03.phbgis srvceph03 > running (6m) 5m ago 6m 20.1M - 2.2.4 > 4a3a1ff181d9 9efebd8d5ba5 > mds.cephfs.srvceph01.tyrask srvceph01 > running (17m) 4m ago 5d 32.4M - 20.2.2 > c22719cb0d76 2f89b9d86343 > mds.cephfs.srvceph02.ebyvdg srvceph02 > running (17m) 5m ago 5d 33.1M - 20.2.2 > c22719cb0d76 5afe19ec45b8 > mds.cephfs.srvceph03.miznhi srvceph03 > running (17m) 5m ago 5d 146M - 20.2.2 > c22719cb0d76 6ee9a906d887 > ... > nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 > unknown 5m ago 7m - - <unknown> <unknown> > <unknown> > nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 > running (5m) 5m ago 7m 15.0M - 5.9 > c22719cb0d76 dcb1eab045cd > nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 > running (4m) 4m ago 7m 14.6M - 5.9 > c22719cb0d76 b11d32f4b0cc ... # podman logs nfs-tentacle-container 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45 After > # ceph orch restart nfs.tentacle-nfs I got this > # podman logs nfs-tentacle-container 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576. 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=0 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:9): Unknown parameter (allow_set_io_flusher_fail) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs *04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP <--------------------------* podman exec -it f2e2c668e0e3 bash > [root@srvceph01 /]# cat etc/ganesha/ganesha.conf > # This file is generated by cephadm. > NFS_CORE_PARAM { > Enable_NLM = false; > Enable_RQUOTA = false; > * Protocols = 3, 4; > <---------------------------------------------------* > mount_path_pseudo = true; > Enable_UDP = false; > NFS_Port = 22049; > allow_set_io_flusher_fail = true; > Monitoring_Port = 9587; > } > NFSv4 { > Delegations = false; > RecoveryBackend = "rados_cluster"; > Minor_Versions = 1, 2; > Server_Scope = "bcf27a48-8a8d-11f1-bdd3-fc607038078a-tentacle-nfs"; > IdmapConf = "/etc/ganesha/idmap.conf"; > } > RADOS_KV { > UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > nodeid = 2; > pool = ".nfs"; > namespace = "tentacle-nfs"; > } > RADOS_URLS { > UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > watch_url = "rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs"; > } > RGW { > cluster = "ceph"; > name = "client.nfs.tentacle-nfs.2.0.srvceph01.ahcayv-rgw"; > } > %url rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs I am not sure if this is the cause of the problem, so I thought, why not enable nfsv3 even if I'm not going to use it? but... Ceph's documentation mentions (tentacle version selected) : https://docs.ceph.com/en/tentacle/mgr/nfs/#create-nfs-ganesha-cluster Create NFS Ganesha Cluster ceph nfs cluster create <cluster_id> .... [--enable-nfsv3] The --enable-nfsv3 is not recognized, I've tried to use it to enable nfsv3 (as a workaround), but it is not recognized. I've also tried to use this from https://docs.ceph.com/en/tentacle/cephadm/services/nfs/#cephadm-ha-nfs : Service Specification service_type: nfsservice_id: mynfsplacement: hosts: - host1 - host2spec: port: 12345 monitoring_port: 567 enable_nfsv3: true <------------------- not recognized The enable_nfsv3 is not recognized. If this is the cause of the problem, then I'm stuck, I don't know how to disable nfsv3 on ganesha nor how to enable nfsv3 with cephadm !!!!! Regards.

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Eugen Block 04 Aug '26

04 Aug '26

Hi, I haven't tried it myself yet so I don't really have an answer. But I noticed that your OSD caps may be incomplete wrt x permissions: > osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, > allow rw namespace=k8s-test tag cephfs data=root' The second allow statement doesn't have an "x". Maybe that's the issue why the workaround fails? Regards, Eugen Zitat von Stefan Möding via ceph-users <ceph-users(a)ceph.io>: > Hi, > > we are trying to use Ceph (Tentacle 20.2.2) as storage provider for > multiple Kubernetes clusters. We are trying to implement the following > security measures: > > * Each Kubernetes cluster should use a dedicated RADOS namespace for > its objects. > * The volumes should use fscrypt encryption. > > We can deploy pods successfully as long as we don’t try to enable > fscrypt. With encryption enabled the pod shows the following logs: > > AttachVolume.Attach succeeded for volume > "pvc-0923605d-3993-4d99-a4a3-d41749ead831" > MountVolume.MountDevice failed for volume > "pvc-0923605d-3993-4d99-a4a3-d41749ead831" : rpc error: code = > Internal desc = failed to lock volume ID > 64ab25ab-9d66-4508-be74-13c02825f40e: rados: ret=-1, Operation not > permitted > > There is a closed issue that mentions a length limitation when creating > a lock to initialize the encryption: > https://github.com/ceph/ceph-csi/issues/5419 > > I also tried the mentioned workaround but that does not help. > > There has also been a discussion that the ’x’ permission is needed in > the OSD capabilities, which we have set: > > mon_caps: 'allow r fsname=root' > mgr_caps: 'allow rw' > osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, > allow rw namespace=k8s-test tag cephfs data=root' > mds_caps: 'allow r fsname=root path=/volumes, allow rws fsname=root > path=/volumes/k8s-test' > > Does anybody have a working setup with encryption and RADOS namespaces? > > -- > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph Steering Committee Meeting 2026-08-03
by Ilya Dryomov 03 Aug '26

03 Aug '26

Hello, - first public foundation board meeting recording is available (Neha) - https://www.youtube.com/watch?v=gPyu9KLVD64 - CDS is starting this week! (Anthony) - https://ceph.io/en/community/events/2026/ceph-developer-summit-vampire/ - performance session led by Mark Nelson is yet to be scheduled, the post will be updated when that happens - upcoming Ceph Days PSA (Patrick) - Zurich: https://ceph.io/en/community/events/2026/ceph-days-zurich/ - Chicago: https://ceph.io/en/community/events/2026/ceph-days-chicago/ - merge window process (Ilya, Patrick) - the date in the calendar is the "branch closes on" not the "PR must be ready for QA by" date - this is in large part due to the fact that the time that it takes for the PR to be picked up into a QA run and the time that it takes for the test results to be reviewed are currently outside of the release manager's control - Patrick will be introducing a new needs-qa alike automated label for backport PRs to formalize the "PR is ready for QA" requirements and make such PRs more discoverable - component leads are free to do their own QA runs when needed - 21.1.1 status - merge window closes in two days - barring negative user feedback or last minute blockers this would be the last umbrella RC - "happy path" tentative release date remains to be Aug 31 - squid EOL plan - at least one merge window/release after umbrella is out Thanks, Ilya

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Wannes Smet 03 Aug '26

03 Aug '26

Hi, Also make sure every node has an _admin label. (ceph orch host label add <hostname> _admin. Verify with ceph orch host ls. Not proud of it, but I once chown-ed /var/lib/ceph on mulptiple nodes. Due to that, our Ceph cluster lost quorum because all the containers crashed. A learning experience I'll never forget #facepalm. I'll definitely think twice before I manupulate /var/lib/ceph ever again ;) . Only if your cluster is 100% test with no production workload on it: Why not try to shut down one node and see what happens on the surviving nodes. Do you get back to HEALTH_OK after you restart the downed node? It should and if it doesn't you're lucky to find out before you went in production. Also, do a "rolling" reboot of all of your nodes. Do you get back to HEALTH_OK after a reboot has finished? This for all the nodes? Anything else breaks if you shut down one node? Wannes. ________________________________ From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, July 21, 2026 22:31 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: How to administer Ceph from any node without bootstrap dependency Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of the admin keyring via orchestrator and the _admin label [0]. That enables you to manage the cluster from any admin node. Not sure what you mean by "bootstrap file" though... If you'd synced the /var/lib/ceph directory, you'd overwrite all the daemons but one, including keyring etc. So please don't do that, that would break your cluster. Regards, Eugen [0] https://urldefense.com/v3/__https://docs.ceph.com/en/latest/cephadm/operati… Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > Hi all, > > I am currently deploying a Ceph cluster with cephadm on 3 nodes with > MON, MGR, and MDS roles. OSDs will be added later. > > My goal is for these 3 nodes to be fully interchangeable, with the > ability to administer the cluster from any of them without risk if a > node fails. > > I have identified that the solution involves synchronizing > /var/lib/ceph/ (excluding the bootstrap file) across all nodes. > > However: > > - Is this the right approach to achieve this goal? > > - Which tool should I use for reliable bidirectional synchronization? > > - What are the potential risks (conflicts, corruption) and precautions > to take? > > Have you implemented such a configuration in production before? > > Thank you for your advice. > > -- > Michel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Guidance Required for Changing Ceph Squid Development Version to Stable Release
by Eugen Block 03 Aug '26

03 Aug '26

Hi, my first recommendation would be to not use a dev release in a production cluster, but that's not really your question. ;-) I haven't used 19.3.0 ever so I'm not sure how it's going to react if you tried to adopt with 19.2.3. Since downgrading Ceph never has been supported this might either just fail or maybe even cause an inconsistency, hard to tell. So my recommendation would be to upgrade to 20 first, then adopt. Regards, Eugen Zitat von Kathirvel T via ceph-users <ceph-users(a)ceph.io>: > Hello Ceph Community, > > We are looking for guidance regarding our existing Ceph cluster. > > Our cluster was deployed using ceph-ansible and is currently running > the Squid development release (v19.3.0). We are planning to adopt > the cluster into cephadm using the cephadm-adopt playbook provided > by ceph-ansible. > > Before proceeding, we would like to understand the recommended > approach for changing our Ceph version from the Squid development > release (v19.3.0) to the stable Squid release (v19.2.3). > > Cluster Details > > Deployment: ceph-ansible > > Current Version: Squid Development (v19.3.0) > > Health: HEALTH_OK > > Hosts: 4 > > MONs: 3 > > MGRs: 1 Active, 2 Standby > > OSDs: 15 (15 Up / 15 In) > > RGW: 1 daemon > > Pools: 15 > > Objects: ~1.64 million > > Usage: 17 TiB used, 9.2 TiB / 26 TiB avail > > > > > We have the following questions: > > Is it possible to change from the Squid development version > (v19.3.0) to the stable Squid release (v19.2.3)? > > Is this a supported approach, or is there a recommended alternative? > > Should we change to the stable release before adopting the cluster > into cephadm, or is it better to perform the cephadm adoption first > and then update the version? > > Are there any known limitations, prerequisites, or best practices we > should be aware of before proceeding? > > > Any guidance, recommendations, or references to relevant > documentation would be greatly appreciated. > > > > Thanks & Regards, > > > > > > > > > > > > > > Kathirvel T > > Associate R&D Engineer mailto:manoj.kumar@zybisys.com > > kathirvel.t(a)zybisys.com > > https://www.zybisys.com > > > > > > https://in.linkedin.com/company/zybisys > https://www.facebook.com/Zybisys/ > https://www.instagram.com/ZyBiSys/ > https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ > Disclaimer : The content of this email and any files transmitted > with it are confidential and intended solely for the use of the > individual or entity to which they are addressed. If you have > received this email in error, please notify the sender and remove > the messages from your system. If you are not the named addressee, > it is strictly forbidden for you to share, circulate, distribute or > copy any part of this e-mail to any third party without the written > consent of the sender. > > > > E-mail transmission cannot be guaranteed to be secured or error free > as information could be intercepted, corrupted, lost, destroyed, > arrive late, incomplete, or may contain viruses. Therefore, we do > not accept liability for any errors or omissions in the contents of > this message, which arise as a result of e-mail transmission. The > recipient should check this e-mail and any attachments for the > presence of viruses. The company accepts no liability for any damage > caused by any virus transmitted by this email." > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Marek Szuba 02 Aug '26

02 Aug '26

Thank you for all the responses! OK, so it's about possible resource contention and not some sort of scary data-corruption problem. That's fine by me, as we only use CephFS mounted directly on storage nodes for operations requiring elevated privileges - nothing resource-intensive. Cheers, -- MS

1 0