- ceph-users - lists.ceph.io

Re: Grafana certificate in version 19
by Chris Palmer 07 Aug '26

07 Aug '26

I never got to the bottom of this, on either squid or tentacle. I have an externally-generated cert/key. As a workaround I do this after each upgrade (most recently to 20.2.3): |cp ~/my_grafana_cert.key /var/lib/ceph/`ceph fsid`/grafana.`hostname -s`/etc/grafana/certs/cert_key cp ~/||my_grafana_cert||.crt /var/lib/ceph/`ceph fsid`/grafana.||`hostname -s`||/etc/grafana/certs/cert_file ceph orch daemon restart grafana.||`hostname -s`||--force| It would be nice to know what the right way is. Regards, Chris On 07/08/2026 13:51, Robert Sander via ceph-users wrote: > Hi, > > how do I set the grafana certificate and key in Ceph 19 that is used on > port 3000? > > The documentation on > https://docs.ceph.com/en/squid/cephadm/services/monitoring/#configuring-ssl… > states: > > # ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem > # ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i > $PWD/certificate.pem > # ceph orch reconfig grafana > > But after doing that grafana still serves a self-signed certificate with > the common name containing the hostname of the first MON (which is not > correct). > > Digging into "ceph config-key dump" shows two entries named > mgr/cephadm/cert_store.cert.grafana_cert and > mgr/cephadm/cert_store.key.grafana_key > which seem to cause this. > > AFAIK the certmgr is only available in Ceph 20. But the docs for this > version > look the same: > https://docs.ceph.com/en/tentacle/cephadm/services/monitoring/#configuring-… > > Only the docs for Ceph 21 got an update and mention certmgr: > https://docs.ceph.com/en/umbrella/cephadm/services/monitoring/#configuring-… > > > The orchestrator in Ceph 19 seems to have something called cert-store: > > # ceph orch cert-store > no valid command found; 4 closest matches: > orch cert-store cert ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store key ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store get cert <entity> [--service_name <value>] [--hostname > <value>] [--no-exception-when-missing] > orch cert-store get key <entity> [--service_name <value>] [--hostname > <value>] [--no-exception-when-missing] > Error EINVAL: invalid command > > But there is no help available. > "ceph orch cert-store --help" does not return anything specific. > Searching for "cert-store" in the Ceph 19 docs also does not return > anything. > > Regards

1 0

Re: Grafana certificate in version 19
by Eugen Block 07 Aug '26

07 Aug '26

Hi, what I noted in my "best practices" doc is this article from RH how to convert to json format: https://access.redhat.com/solutions/7109138 And after conversion the certificate and key are imported like this: ceph config-key set mgr/cephadm/cert_store.cert.grafana_cert -i grafana_certs.json ceph config-key set mgr/cephadm/cert_store.cert.grafana_key -i grafana_key.json I vaguely remember a thread here on this list about it as well, but I don't know if this workaround is still required or not. I'll probably have to when the current cert expires. :-D Regards, Eugen Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Hi, > > how do I set the grafana certificate and key in Ceph 19 that is used on > port 3000? > > The documentation on > https://docs.ceph.com/en/squid/cephadm/services/monitoring/#configuring-ssl… > states: > > # ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem > # ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i > $PWD/certificate.pem > # ceph orch reconfig grafana > > But after doing that grafana still serves a self-signed certificate with > the common name containing the hostname of the first MON (which is > not correct). > > Digging into "ceph config-key dump" shows two entries named > mgr/cephadm/cert_store.cert.grafana_cert and > mgr/cephadm/cert_store.key.grafana_key > which seem to cause this. > > AFAIK the certmgr is only available in Ceph 20. But the docs for this version > look the same: > https://docs.ceph.com/en/tentacle/cephadm/services/monitoring/#configuring-… > > Only the docs for Ceph 21 got an update and mention certmgr: > https://docs.ceph.com/en/umbrella/cephadm/services/monitoring/#configuring-… > > The orchestrator in Ceph 19 seems to have something called cert-store: > > # ceph orch cert-store > no valid command found; 4 closest matches: > orch cert-store cert ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store key ls [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] > orch cert-store get cert <entity> [--service_name <value>] > [--hostname <value>] [--no-exception-when-missing] > orch cert-store get key <entity> [--service_name <value>] > [--hostname <value>] [--no-exception-when-missing] > Error EINVAL: invalid command > > But there is no help available. > "ceph orch cert-store --help" does not return anything specific. > Searching for "cert-store" in the Ceph 19 docs also does not return anything. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Grafana certificate in version 19
by Robert Sander 07 Aug '26

07 Aug '26

Hi, how do I set the grafana certificate and key in Ceph 19 that is used on port 3000? The documentation on https://docs.ceph.com/en/squid/cephadm/services/monitoring/#configuring-ssl… states: # ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem # ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i $PWD/certificate.pem # ceph orch reconfig grafana But after doing that grafana still serves a self-signed certificate with the common name containing the hostname of the first MON (which is not correct). Digging into "ceph config-key dump" shows two entries named mgr/cephadm/cert_store.cert.grafana_cert and mgr/cephadm/cert_store.key.grafana_key which seem to cause this. AFAIK the certmgr is only available in Ceph 20. But the docs for this version look the same: https://docs.ceph.com/en/tentacle/cephadm/services/monitoring/#configuring-… Only the docs for Ceph 21 got an update and mention certmgr: https://docs.ceph.com/en/umbrella/cephadm/services/monitoring/#configuring-… The orchestrator in Ceph 19 seems to have something called cert-store: # ceph orch cert-store no valid command found; 4 closest matches: orch cert-store cert ls [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] orch cert-store key ls [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] orch cert-store get cert <entity> [--service_name <value>] [--hostname <value>] [--no-exception-when-missing] orch cert-store get key <entity> [--service_name <value>] [--hostname <value>] [--no-exception-when-missing] Error EINVAL: invalid command But there is no help available. "ceph orch cert-store --help" does not return anything specific. Searching for "cert-store" in the Ceph 19 docs also does not return anything. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: v20.2.3 Tentacle released
by Afreen 07 Aug '26

07 Aug '26

Hey, That seems like some issue in parsing the format and so far what we found out is that it could happen when CephFS cloning is used , are you using that ? And pools page is failing because its polling /summary API - fetching progress internally. You can try clearing progress and then refresh - "ceph progress clear". Its just clears the display events and does not stop operations. - Afreen On Thu, Aug 6, 2026 at 6:42 PM Salatiel Filho via ceph-users < ceph-users(a)ceph.io> wrote: > After upgrading from 20.2.2 to 20.2.3, the dashboard is partially > broken. List pools don't work. Error 500. > Also get error 500 popup as soon as I login to the dashboard. > > Logs show this: > 2026-08-06 10:04:23 2026-08-06T13:04:23.386+0000 7f5afcd91640 0 > [dashboard ERROR dashboard.services.exception] Internal Server Error > 2026-08-06 10:04:23 Traceback (most recent call last): > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in > dashboard_exception_handler > 2026-08-06 10:04:23 return handler(*args, **kwargs) > 2026-08-06 10:04:23 File > "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in > __call__ > 2026-08-06 10:04:23 return self.callable(*self.args, **self.kwargs) > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line > 264, in inner > 2026-08-06 10:04:23 ret = func(*args, **kwargs) > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/controllers/summary.py", line 105, in > __call__ > 2026-08-06 10:04:23 e, f = progress.get_progress_tasks() > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/services/progress.py", line 89, in > get_progress_tasks > 2026-08-06 10:04:23 > finished_t.append(_progress_event_to_dashboard_task(ev, True)) > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/services/progress.py", line 60, in > _progress_event_to_dashboard_task > 2026-08-06 10:04:23 _progress_event_to_dashboard_task_common(event, > task) > 2026-08-06 10:04:23 File > "/usr/share/ceph/mgr/dashboard/services/progress.py", line 52, in > _progress_event_to_dashboard_task_common > 2026-08-06 10:04:23 'metadata': dict(event.get('refs', {})), > 2026-08-06 10:04:23 ValueError: dictionary update sequence element > #0 has length 3; 2 is required > 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 > [dashboard ERROR dashboard.tools] [::ffff:10.175.143.9:53998] [GET] > [500] [0.009s] [admin] [513.0B] /api/summary > 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 > [dashboard ERROR dashboard.tools] [b'{"status": "500 Internal Server > Error", "detail": "The server encountered an unexpected condition > which prevented it from fulfilling the request.", "request_id": > "c7349113-44d8-4e48-88d6-3a469786397f"} > > > > '] > 2026-08-06 10:04:23 2026-08-06T13:04:23.494+0000 7f5afed95640 0 > [dashboard ERROR dashboard.controllers.frontend_logging] > (https://ceph.mydomain/#/pool) Http failure response for > https://ceph.mydomain/api/summary: 500 Internal Server Error > > Atenciosamente/Kind regards, > Salatiel > > On Wed, Aug 5, 2026 at 11:06 AM Yuri Weinstein via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > This is the third minor release in the Tentacle series. > > We recommend that all users update to this release. > > > > Notable Changes > > --------------- > > > > MDS (Metadata Server) > > --------------------- > > > > * Fixed a crash and shutdown hang that could occur when ephemeral pins > were > > active and max_mds was set to 0. > > * Session reclaim could miss blocklisting an old session; this is now > > handled correctly. > > * Fixed a case where scrub was unable to identify damage from an > > irreparable hard link. > > > > OSD (Object Storage Daemon) > > ---------------------------- > > > > * BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write > buffer > > misalignment. > > * Erasure Coding: Fixed truncate+write planning for EC shard sizes. > > * Fixed a condition that prevented rolling forward of PG log entries. > > * Prevented OSDMap::check_health() from asserting when new OSDs are found > > in a subtree. > > * Scrub: "repairing" scrubs are now allowed at all times. > > > > RGW (RADOS Gateway) > > -------------------- > > > > * Added the ssl_ciphersuites option for the Beast front-end to support > TLS > > 1.3 cipher customization. > > * Fixed PutObject's canned_acl comparison bug affecting BlockPublicAcls. > > * Fixed crashes occurring on realm reload. > > * Implemented CopyObject support for encrypted objects. > > * Added mTLS client-certificate authentication support for Kafka > > notifications. > > * Fixed removal of delete markers in lifecycle processing. > > > > RADOS / librados / neorados > > ---------------------------- > > > > * neorados: Fixed an infinite trim loop on empty data log shards. > > * Fixed cancellation-slot cleanup in librados/asio's associated executor. > > > > Dashboard > > ----------------- > > > > * Added a hardware-monitoring dashboard backed by node-proxy metrics. > > * Fixed a bind-address regression caused by CherryPy isolation. > > * Added support for adding hosts via CSV upload. > > * NVMeoF: multiple CLI and UI backports (EC pool support, IO statistics, > > role management). > > * Object: The global RGW Roles tab has been removed. Role management is > now > > scoped under RGW Accounts, where roles can be listed, created, edited, > and > > deleted for a selected account. > > > > ceph-volume > > ----------- > > > > * Reworked OSD mapper lifecycle handling (LVM + raw) for activate. > > * Improved detection of rotational media under dm-crypt to bypass the > > workqueue correctly. > > * Fixed OSD re-deployment issues with disk-selection filters and DB > devices. > > * Added retry handling for lvs when it returns an empty result or a > > "devices file is missing" error. > > > > NVMe-oF > > ------- > > > > * Added configurable delayed failback (default 0, no delay) to > accommodate > > initiators that recover paths with a delay. > > * Gateways in the DELETING state now ignore beacons and send empty maps > > instead of processing them. > > * rbd_with_crc32c is now enabled by default via cephadm. > > * Introduced a new NVMeoF mgr module. The module automatically creates > the > > ".nvmeof" metadata pool if it does not already exist. Gateway-group state > > files are stored in this pool unless the user specifies another pool. The > > module is enabled automatically on fresh installs; on upgrades from an > > older cluster, it must be enabled manually. > > > > RBD > > --- > > > > * Fixed a use-after-free bug releasing object map locks during deep copy. > > * Fixed memory leaks in PWL discard operations. > > * Obsolete primary mirror snapshots are now pruned after relocation. > > > > We recommend users to update to this release. > > For detailed release notes with links & changelog please refer to the > > official blog entry at > > https://ceph.io/en/news/blog/2026/v20-2-3-tentacle-released/ > > > > Getting Ceph > > ------------ > > * Git at git://github.com/ceph/ceph.git > > * Tarball at https://download.ceph.com/tarballs/ceph-20.2.3.tar.gz > > * Containers at https://quay.io/repository/ceph/ceph > > * For packages, see > https://docs.ceph.com/en/latest/install/get-packages/ > > * Release git sha1: 06c2f9c35b67055a8a6fb99d1be236b3c4832ace > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: RGW Account Quota in Multisite Setup
by Casey Bodley 07 Aug '26

07 Aug '26

On Thu, Aug 6, 2026 at 3:44 AM Danila Reznichuk via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I have a question about account quotas in an RGW multisite setup. > > My configuration is fairly simple: > > Main zonegroup with the master zone. > Archive zonegroup with two zones: A and B. > > I created an account on the master zone of the main zonegroup and set > the account quota (--quota-scope account) to 1 GiB. As expected, this > change was synchronized to all other zones in all zonegroups. > > Then, by mistake, I changed the quota for the same account on zone A of > the archive zonegroup to 2 GiB. This change was not synchronized to the > other zones. That seems reasonable to me and could even be considered a > feature, since different zonegroups may represent different locations > with different storage policies or usage patterns. > > After some testing, I found that updating the account quota again on > the master zone of the main zonegroup overwrites the locally modified > quota on zone A. > > So I have a few questions: > > Is this the intended behavior? > Is there a supported way to configure different account quotas for > different zones or zonegroups? > If not, is the account quota expected to be globally consistent across > the entire multisite deployment? the intent of multisite is to keep all metadata (accounts, users, buckets, etc) consistent across zones, with the metadata master zone as the source of truth. there are several radosgw-admin commands that can modify metadata on secondary zones, but they're supposed to print a warning message and require --yes-i-really-mean-it. that logic was missing from the quota commands, so i've raised https://tracker.ceph.com/issues/79120 to track the fix and its backports > > Thanks! > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by Robert Sander 07 Aug '26

07 Aug '26

Hi, Am 07.08.26 um 10:59 AM schrieb Frédéric Nass via ceph-users: > Or, if that's too much of a maintenance burden and you're worried you'll > forget that step, just install rpcbind on the NFS nodes and leave NFSv3 > enabled. Or maybe NFSv3 is made an option for the NFS service spec with the hint that then rpcbind has to be installed. Or the orchestrator installs rpcbind when the NFS service should offer NFSv3. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by Frédéric Nass 07 Aug '26

07 Aug '26

Hi everyone, Unfortunately, the custom config won't work. The custom config is loaded after ganesha.conf, and by then NFS_CORE_PARAM has already been parsed from the generated config. Ganesha doesn't merge settings between two blocks of the same type. The later is simply ignored, so your entire NFS_CORE_PARAM block from custom config would have no effect. We should probably update the documentation to list which blocks can actually be defined in the custom config, or just mention the above. As a workaround, download the Ganesha Jinja2 template that matches your Ceph version: curl -O https://github.com/ceph/ceph/blob/v20.2.2/src/pybind/mgr/cephadm/templates/… Edit it to disable NFSv3, store it as a cephadm override: ceph config-key set mgr/cephadm/services/nfs/ganesha.conf -i ./ganesha.conf.j2 Then redeploy the daemons: ceph orch redeploy nfs.<service_id> Make sure to update the template before upgrading the Ceph cluster. This file has changed across Ceph releases and is expected to keep changing in the future. Or, if that's too much of a maintenance burden and you're worried you'll forget that step, just install rpcbind on the NFS nodes and leave NFSv3 enabled. Hope that helps. Best regards, Frédéric -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> frederic.nass(a)clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Le mer. 5 août 2026, 15:40, Eugen Block via ceph-users <ceph-users(a)ceph.io> a écrit : > That's what the docs say, yes. > > > https://docs.ceph.com/en/latest/mgr/nfs/#set-customized-nfs-ganesha-configu… > > Zitat von wodel youchi <wodel.youchi(a)gmail.com>: > > > Hi, > > > > Thanks, to remove nfsv3 from ganesha's config, should I use : > > > > ceph nfs cluster config set <cluster_id> -i <config_file> > > > > > > with for example : > > > > vim <config_file> > > > > NFS_CORE_PARAM { > > * Protocols = 4; > > } > > > > Then > > # ceph orch restart nfs.tentacle-nfs > > > > Regards. > > > > Le mar. 4 août 2026 à 11:33, Eugen Block via ceph-users < > ceph-users(a)ceph.io> > > a écrit : > > > >> Hi, > >> > >> look at this thread [0] and the corresponding links provided in there > >> [1], [2]. > >> > >> Regards, > >> Eugen > >> > >> [0] > >> > >> > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LG2EGDZMIW… > >> [1] https://tracker.ceph.com/issues/74307 > >> [2] https://tracker.ceph.com/issues/74191 > >> > >> Zitat von wodel youchi via ceph-users <ceph-users(a)ceph.io>: > >> > >> > Hi, > >> > > >> > I'm using tentacle 20.2.2 with podman on Rocky Linux 9. > >> > I'm trying to configure nfs-ganesha, but the nfs-tentacle containers > >> won't > >> > stay up. > >> > I've 3 nodes, all services are collocated. > >> > > >> >> > >> >> # ceph nfs cluster create tentacle-nfs > "srvceph01,srvceph02,srvceph03" > >> > --ingress --virtual_ip 10.200.24.89 --ingress-mode haproxy-standard > >> --port > >> > 12049 > >> > > >> > > >> > When I create the nfs cluster (with HA, haproxy + keepalived). 3 > >> > nfs-tentacle-nfs containers are created, one on each node, but they do > >> not > >> > stay alive, they start then disappear, again and again, and in the end > >> just > >> > one is created, but it does not really work, HAProxy shows that all > >> > backends are down. > >> > > >> > What I've got from the vanished containers is that the nfs service is > >> > looking for rpc-bind which does not exist because ganesha's config > file > >> > enables both nfs v3 and v4. > >> > > >> > # ceph orch ps --service_name nfs.tentacle-nfs > >> >> NAME HOST PORTS STATUS > >> >> REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID > CONTAINER ID > >> >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown > >> >> 5m ago 6m - - <unknown> <unknown> <unknown> > >> >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running > (5m) > >> >> 5m ago 6m 15.0M - 5.9 c22719cb0d76 > dcb1eab045cd > >> >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running > (4m) > >> >> 4m ago 6m 14.6M - 5.9 c22719cb0d76 > b11d32f4b0cc > >> > > >> > > >> > > >> > # ceph orch ps > >> >> NAME HOST PORTS > >> >> STATUS REFRESHED AGE MEM USE MEM LIM VERSION > IMAGE > >> ID > >> >> CONTAINER ID > >> >> ... > >> >> haproxy.nfs.tentacle-nfs.srvceph01.uvueop srvceph01 > *:12049,19049 > >> >> running (6m) 4m ago 6m 11.5M - 2.3.17-d1c9119 > >> >> e85424b0d443 d4a840202ae0 > >> >> haproxy.nfs.tentacle-nfs.srvceph02.xujdir srvceph02 > *:12049,19049 > >> >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > >> >> e85424b0d443 747cf2052cd5 > >> >> haproxy.nfs.tentacle-nfs.srvceph03.snapkl srvceph03 > *:12049,19049 > >> >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > >> >> e85424b0d443 3c17cb2a4757 > >> >> keepalived.nfs.tentacle-nfs.srvceph01.phscfe srvceph01 > >> >> running (6m) 4m ago 6m 19.9M - 2.2.4 > >> >> 4a3a1ff181d9 d053e0b91210 > >> >> keepalived.nfs.tentacle-nfs.srvceph02.ubwgoo srvceph02 > >> >> running (6m) 5m ago 6m 20.1M - 2.2.4 > >> >> 4a3a1ff181d9 6ad75ccccaa0 > >> >> keepalived.nfs.tentacle-nfs.srvceph03.phbgis srvceph03 > >> >> running (6m) 5m ago 6m 20.1M - 2.2.4 > >> >> 4a3a1ff181d9 9efebd8d5ba5 > >> >> mds.cephfs.srvceph01.tyrask srvceph01 > >> >> running (17m) 4m ago 5d 32.4M - 20.2.2 > >> >> c22719cb0d76 2f89b9d86343 > >> >> mds.cephfs.srvceph02.ebyvdg srvceph02 > >> >> running (17m) 5m ago 5d 33.1M - 20.2.2 > >> >> c22719cb0d76 5afe19ec45b8 > >> >> mds.cephfs.srvceph03.miznhi srvceph03 > >> >> running (17m) 5m ago 5d 146M - 20.2.2 > >> >> c22719cb0d76 6ee9a906d887 > >> >> ... > >> >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 > >> >> unknown 5m ago 7m - - <unknown> > >> <unknown> > >> >> <unknown> > >> >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 > >> >> running (5m) 5m ago 7m 15.0M - 5.9 > >> >> c22719cb0d76 dcb1eab045cd > >> >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 > >> >> running (4m) 4m ago 7m 14.6M - 5.9 > >> >> c22719cb0d76 b11d32f4b0cc > >> > > >> > ... > >> > > >> > > >> > # podman logs nfs-tentacle-container > >> > > >> > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :grace reload client info completed from > >> > backend > >> > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > >> > count(0) > >> > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > rados_cluster_grace_enforcing :CLIENT ID :EVENT > >> > :rados_cluster_grace_enforcing: ret=-45 > >> > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > >> > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :grace reload client info completed from > >> > backend > >> > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > >> > count(0) > >> > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > rados_cluster_grace_enforcing :CLIENT ID :EVENT > >> > :rados_cluster_grace_enforcing: ret=-45 > >> > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > >> > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :grace reload client info completed from > >> > backend > >> > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > >> > count(0) > >> > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : > ganesha.nfsd-2[main] > >> > rados_cluster_grace_enforcing :CLIENT ID :EVENT > >> > :rados_cluster_grace_enforcing: ret=-45 > >> > > >> > > >> > > >> > After > >> > > >> >> # ceph orch restart nfs.tentacle-nfs > >> > > >> > > >> > I got this > >> > > >> >> # podman logs nfs-tentacle-container > >> > > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > init_logging :LOG :NULL :LOG: Setting log level for all components to > >> > NIV_EVENT > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file > >> > successfully parsed > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed > limit > >> > on FDs to 1048576. > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully > initialized. > >> > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > >> > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_start_grace :STATE :EVENT :grace reload client info completed from > >> > backend > >> > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > >> > count(0) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > rados_cluster_grace_enforcing :CLIENT ID :EVENT > >> > :rados_cluster_grace_enforcing: ret=0 > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > main :NFS STARTUP :WARN :No export entries found in configuration file > >> !!! > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > config_errs_to_log :CONFIG :WARN :Config File > >> > (/etc/ganesha/ganesha.conf:9): Unknown parameter > >> (allow_set_io_flusher_fail) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > config_errs_to_log :CONFIG :WARN :Config File > >> > (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > config_errs_to_log :CONFIG :WARN :Config File > >> > (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully > >> > removed for proper quota management in FSAL > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: > >> > > >> > cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect > to > >> > socket /run/dbus/system_bus_socket: No such file or directory) > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > gsh_dbus_register_path :DBUS :CRIT > :dbus_connection_register_object_path > >> > called with no DBUS connection > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > gsh_dbus_register_path :DBUS :CRIT > :dbus_connection_register_object_path > >> > called with no DBUS connection > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > gsh_dbus_register_path :DBUS :CRIT > :dbus_connection_register_object_path > >> > called with no DBUS connection > >> > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs > >> > *04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : > ganesha.nfsd-2[main] > >> > __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP > >> > <--------------------------* > >> > > >> > > >> > > >> > podman exec -it f2e2c668e0e3 bash > >> >> [root@srvceph01 /]# cat etc/ganesha/ganesha.conf > >> >> # This file is generated by cephadm. > >> >> NFS_CORE_PARAM { > >> >> Enable_NLM = false; > >> >> Enable_RQUOTA = false; > >> >> * Protocols = 3, 4; > >> >> <---------------------------------------------------* > >> >> mount_path_pseudo = true; > >> >> Enable_UDP = false; > >> >> NFS_Port = 22049; > >> >> allow_set_io_flusher_fail = true; > >> >> Monitoring_Port = 9587; > >> >> } > >> >> NFSv4 { > >> >> Delegations = false; > >> >> RecoveryBackend = "rados_cluster"; > >> >> Minor_Versions = 1, 2; > >> >> Server_Scope = > >> "bcf27a48-8a8d-11f1-bdd3-fc607038078a-tentacle-nfs"; > >> >> IdmapConf = "/etc/ganesha/idmap.conf"; > >> >> } > >> >> RADOS_KV { > >> >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > >> >> nodeid = 2; > >> >> pool = ".nfs"; > >> >> namespace = "tentacle-nfs"; > >> >> } > >> >> RADOS_URLS { > >> >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > >> >> watch_url = > "rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs"; > >> >> } > >> >> RGW { > >> >> cluster = "ceph"; > >> >> name = "client.nfs.tentacle-nfs.2.0.srvceph01.ahcayv-rgw"; > >> >> } > >> >> %url rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs > >> > > >> > > >> > > >> > I am not sure if this is the cause of the problem, so I thought, why > not > >> > enable nfsv3 even if I'm not going to use it? but... > >> > Ceph's documentation mentions (tentacle version selected) : > >> > https://docs.ceph.com/en/tentacle/mgr/nfs/#create-nfs-ganesha-cluster > >> > Create NFS Ganesha Cluster > >> > > >> > ceph nfs cluster create <cluster_id> .... [--enable-nfsv3] > >> > > >> > The --enable-nfsv3 is not recognized, I've tried to use it to enable > >> > nfsv3 (as a workaround), but it is not recognized. > >> > > >> > I've also tried to use this from > >> > > https://docs.ceph.com/en/tentacle/cephadm/services/nfs/#cephadm-ha-nfs > >> > : > >> > > >> > Service Specification > >> > > >> > service_type: nfsservice_id: mynfsplacement: hosts: - host1 - > >> > host2spec: port: 12345 monitoring_port: 567 enable_nfsv3: true > >> > <------------------- not recognized > >> > > >> > The enable_nfsv3 is not recognized. > >> > > >> > If this is the cause of the problem, then I'm stuck, I don't know how > to > >> > disable nfsv3 on ganesha nor how to enable nfsv3 with cephadm !!!!! > >> > > >> > > >> > Regards. > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users(a)ceph.io > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: RGW Account Quota in Multisite Setup
by Natanam Sethu 07 Aug '26

07 Aug '26

Hi Casey, If you are using the bi-directional replication. When ever you modify the quota on Zone A. It will modify the master Zone as well. If you one use way replication. I.e that is from the master zone to zone A, it will not Regards Natanam Sethu On Thu, 6 Aug 2026 at 18:49, Casey Bodley via ceph-users <ceph-users(a)ceph.io> wrote: > On Thu, Aug 6, 2026 at 3:44 AM Danila Reznichuk via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > Hi, > > > > I have a question about account quotas in an RGW multisite setup. > > > > My configuration is fairly simple: > > > > Main zonegroup with the master zone. > > Archive zonegroup with two zones: A and B. > > > > I created an account on the master zone of the main zonegroup and set > > the account quota (--quota-scope account) to 1 GiB. As expected, this > > change was synchronized to all other zones in all zonegroups. > > > > Then, by mistake, I changed the quota for the same account on zone A of > > the archive zonegroup to 2 GiB. This change was not synchronized to the > > other zones. That seems reasonable to me and could even be considered a > > feature, since different zonegroups may represent different locations > > with different storage policies or usage patterns. > > > > After some testing, I found that updating the account quota again on > > the master zone of the main zonegroup overwrites the locally modified > > quota on zone A. > > > > So I have a few questions: > > > > Is this the intended behavior? > > Is there a supported way to configure different account quotas for > > different zones or zonegroups? > > If not, is the account quota expected to be globally consistent across > > the entire multisite deployment? > > the intent of multisite is to keep all metadata (accounts, users, > buckets, etc) consistent across zones, with the metadata master zone > as the source of truth. there are several radosgw-admin commands that > can modify metadata on secondary zones, but they're supposed to print > a warning message and require --yes-i-really-mean-it. that logic was > missing from the quota commands, so i've raised > https://tracker.ceph.com/issues/79120 to track the fix and its > backports > > > > > Thanks! > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: v20.2.3 Tentacle released
by Salatiel Filho 06 Aug '26

06 Aug '26

After upgrading from 20.2.2 to 20.2.3, the dashboard is partially broken. List pools don't work. Error 500. Also get error 500 popup as soon as I login to the dashboard. Logs show this: 2026-08-06 10:04:23 2026-08-06T13:04:23.386+0000 7f5afcd91640 0 [dashboard ERROR dashboard.services.exception] Internal Server Error 2026-08-06 10:04:23 Traceback (most recent call last): 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler 2026-08-06 10:04:23 return handler(*args, **kwargs) 2026-08-06 10:04:23 File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ 2026-08-06 10:04:23 return self.callable(*self.args, **self.kwargs) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 264, in inner 2026-08-06 10:04:23 ret = func(*args, **kwargs) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/controllers/summary.py", line 105, in __call__ 2026-08-06 10:04:23 e, f = progress.get_progress_tasks() 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 89, in get_progress_tasks 2026-08-06 10:04:23 finished_t.append(_progress_event_to_dashboard_task(ev, True)) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 60, in _progress_event_to_dashboard_task 2026-08-06 10:04:23 _progress_event_to_dashboard_task_common(event, task) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 52, in _progress_event_to_dashboard_task_common 2026-08-06 10:04:23 'metadata': dict(event.get('refs', {})), 2026-08-06 10:04:23 ValueError: dictionary update sequence element #0 has length 3; 2 is required 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 [dashboard ERROR dashboard.tools] [::ffff:10.175.143.9:53998] [GET] [500] [0.009s] [admin] [513.0B] /api/summary 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 [dashboard ERROR dashboard.tools] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "c7349113-44d8-4e48-88d6-3a469786397f"} '] 2026-08-06 10:04:23 2026-08-06T13:04:23.494+0000 7f5afed95640 0 [dashboard ERROR dashboard.controllers.frontend_logging] (https://ceph.mydomain/#/pool) Http failure response for https://ceph.mydomain/api/summary: 500 Internal Server Error Atenciosamente/Kind regards, Salatiel On Wed, Aug 5, 2026 at 11:06 AM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> wrote: > > This is the third minor release in the Tentacle series. > We recommend that all users update to this release. > > Notable Changes > --------------- > > MDS (Metadata Server) > --------------------- > > * Fixed a crash and shutdown hang that could occur when ephemeral pins were > active and max_mds was set to 0. > * Session reclaim could miss blocklisting an old session; this is now > handled correctly. > * Fixed a case where scrub was unable to identify damage from an > irreparable hard link. > > OSD (Object Storage Daemon) > ---------------------------- > > * BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write buffer > misalignment. > * Erasure Coding: Fixed truncate+write planning for EC shard sizes. > * Fixed a condition that prevented rolling forward of PG log entries. > * Prevented OSDMap::check_health() from asserting when new OSDs are found > in a subtree. > * Scrub: "repairing" scrubs are now allowed at all times. > > RGW (RADOS Gateway) > -------------------- > > * Added the ssl_ciphersuites option for the Beast front-end to support TLS > 1.3 cipher customization. > * Fixed PutObject's canned_acl comparison bug affecting BlockPublicAcls. > * Fixed crashes occurring on realm reload. > * Implemented CopyObject support for encrypted objects. > * Added mTLS client-certificate authentication support for Kafka > notifications. > * Fixed removal of delete markers in lifecycle processing. > > RADOS / librados / neorados > ---------------------------- > > * neorados: Fixed an infinite trim loop on empty data log shards. > * Fixed cancellation-slot cleanup in librados/asio's associated executor. > > Dashboard > ----------------- > > * Added a hardware-monitoring dashboard backed by node-proxy metrics. > * Fixed a bind-address regression caused by CherryPy isolation. > * Added support for adding hosts via CSV upload. > * NVMeoF: multiple CLI and UI backports (EC pool support, IO statistics, > role management). > * Object: The global RGW Roles tab has been removed. Role management is now > scoped under RGW Accounts, where roles can be listed, created, edited, and > deleted for a selected account. > > ceph-volume > ----------- > > * Reworked OSD mapper lifecycle handling (LVM + raw) for activate. > * Improved detection of rotational media under dm-crypt to bypass the > workqueue correctly. > * Fixed OSD re-deployment issues with disk-selection filters and DB devices. > * Added retry handling for lvs when it returns an empty result or a > "devices file is missing" error. > > NVMe-oF > ------- > > * Added configurable delayed failback (default 0, no delay) to accommodate > initiators that recover paths with a delay. > * Gateways in the DELETING state now ignore beacons and send empty maps > instead of processing them. > * rbd_with_crc32c is now enabled by default via cephadm. > * Introduced a new NVMeoF mgr module. The module automatically creates the > ".nvmeof" metadata pool if it does not already exist. Gateway-group state > files are stored in this pool unless the user specifies another pool. The > module is enabled automatically on fresh installs; on upgrades from an > older cluster, it must be enabled manually. > > RBD > --- > > * Fixed a use-after-free bug releasing object map locks during deep copy. > * Fixed memory leaks in PWL discard operations. > * Obsolete primary mirror snapshots are now pruned after relocation. > > We recommend users to update to this release. > For detailed release notes with links & changelog please refer to the > official blog entry at > https://ceph.io/en/news/blog/2026/v20-2-3-tentacle-released/ > > Getting Ceph > ------------ > * Git at git://github.com/ceph/ceph.git > * Tarball at https://download.ceph.com/tarballs/ceph-20.2.3.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * Release git sha1: 06c2f9c35b67055a8a6fb99d1be236b3c4832ace > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

RGW Account Quota in Multisite Setup
by dreznichuk＠gmail.com 06 Aug '26

06 Aug '26

Hi, I have a question about account quotas in an RGW multisite setup. My configuration is fairly simple: Main zonegroup with the master zone. Archive zonegroup with two zones: A and B. I created an account on the master zone of the main zonegroup and set the account quota (--quota-scope account) to 1 GiB. As expected, this change was synchronized to all other zones in all zonegroups. Then, by mistake, I changed the quota for the same account on zone A of the archive zonegroup to 2 GiB. This change was not synchronized to the other zones. That seems reasonable to me and could even be considered a feature, since different zonegroups may represent different locations with different storage policies or usage patterns. After some testing, I found that updating the account quota again on the master zone of the main zonegroup overwrites the locally modified quota on zone A. So I have a few questions: Is this the intended behavior? Is there a supported way to configure different account quotas for different zones or zonegroups? If not, is the account quota expected to be globally consistent across the entire multisite deployment? Thanks!

1 0