- ceph-users - lists.ceph.io

Release notes update: don't (try to) upgrade from v19.2.6 to Tentacle < 20.2.4
by Eugen Block 03 Sep '26

03 Sep '26

Hi, I know it might be unlikely, but not all users are (yet) subscribed to this list or for some reason can't catch up. I feel like the docs should clearly state that you can't upgrade from 19.2.6 to Tentacle < 20.2.4. I'm almost sure that there will be users tying to do exactly that just because 20 > 19. :-) Regards, Eugen

1 0

Orch stops refreshing daemon list
by Chris Palmer 02 Sep '26

02 Sep '26

I have a problem following a 20.2.3 -> 20.2.4 upgrade. This has happened once on a virtualised test cluster, and now on a small production cluster. After it happened on test, I rolled all nodes back to snapshots and tried again, and it worked, so I assumed finger trouble and proceeded to the production cluster. But now it is stuck... - The upgrade itself is completely successful. Repeated "ceph orch ps --refresh" looks good and shows the "refreshed" value returning to zero each time. - I then rotate the SSH keys, and everything at first appears ok. "check-host" works between every pair of hosts in each direction. - But... "ceph orch ps --refresh" has now stopped refreshing. And any attempts to restart or redeploy a daemon all say it is scheduled but nothing happens. I've tried failing MGRs, rebooting a node then failing back to the MGR on it. There's nothing odd that I can spot in the MGR logs. After the reboot, "ceph orch ps --refresh" still shows the OLD container IDs, and an ever-increasing last-refresh time. I've also rotated the client.admin key on the production cluster without any problems. I don't think that is causing this orch problem, as the test cluster originally failed before client.admin had been rotated. Can anyone suggest how I may troubleshoot this? Regards, Chris

1 0

Re: Tentacle v20.2.4 packages build for Ubuntu Noble (24.04)
by Patrick Donnelly 02 Sep '26

02 Sep '26

On Wed, Sep 2, 2026 at 7:07 AM Eugen Block <ceph-users(a)ceph.io> wrote: > > Hi Patrick, > > thanks for the efforts to bring 20.2.4 to Noble! > > I just installed those packages on a virtual machine that I use for > testing. I can create a cluster successfully (package based), but the > MGR ends up in a respawn loop, reporting these error messages: > > ---snip--- > Sep 02 10:27:10 tentacle-ubuntu24 ceph-mgr[3289]: Failed to import > NVMeoFClient and related components: cannot import name 'NVMeoFClient' > from 'dashboard.services.nvmeof_client' > (/usr/share/ceph/mgr/dashboard/services/nvmeof_client.py) > Sep 02 10:27:10 tentacle-ubuntu24 ceph-mgr[3289]: ERROR:root:Module > 'xmltodict' is not installed. > ---snip--- > > The xmltodict error can easily be fixed by installing > python3-xmltodict, but it probably should be a dependency then when > installing the ceph-mgr package, I assume? Correct. That is an example of a change that will need backported for v20.2.5. > The NVMeoFClient message doesn't seem to be critical, at least I can't > notice anything obvious right now, but I also have no nvme service > deployed or anything. The dashboard also works fine. > > Should I create tracker for the xmltodict error? No need, I'll take care of it. Thank you for testing and the report! -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Grafana certificate in version 19
by Redouane Kachach 02 Sep '26

02 Sep '26

I hope this helps: https://github.com/ceph/ceph/pull/66348#issuecomment-3641268398 Best, Redouane. On Fri, Aug 7, 2026 at 3:51 PM Chris Palmer via ceph-users < ceph-users(a)ceph.io> wrote: > I never got to the bottom of this, on either squid or tentacle. I have > an externally-generated cert/key. As a workaround I do this after each > upgrade (most recently to 20.2.3): > > |cp ~/my_grafana_cert.key /var/lib/ceph/`ceph fsid`/grafana.`hostname > -s`/etc/grafana/certs/cert_key cp ~/||my_grafana_cert||.crt > /var/lib/ceph/`ceph fsid`/grafana.||`hostname > -s`||/etc/grafana/certs/cert_file ceph orch daemon restart > grafana.||`hostname -s`||--force| > > It would be nice to know what the right way is. > > Regards, Chris > > > On 07/08/2026 13:51, Robert Sander via ceph-users wrote: > > Hi, > > > > how do I set the grafana certificate and key in Ceph 19 that is used on > > port 3000? > > > > The documentation on > > > https://docs.ceph.com/en/squid/cephadm/services/monitoring/#configuring-ssl… > > states: > > > > # ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem > > # ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i > > $PWD/certificate.pem > > # ceph orch reconfig grafana > > > > But after doing that grafana still serves a self-signed certificate with > > the common name containing the hostname of the first MON (which is not > > correct). > > > > Digging into "ceph config-key dump" shows two entries named > > mgr/cephadm/cert_store.cert.grafana_cert and > > mgr/cephadm/cert_store.key.grafana_key > > which seem to cause this. > > > > AFAIK the certmgr is only available in Ceph 20. But the docs for this > > version > > look the same: > > > https://docs.ceph.com/en/tentacle/cephadm/services/monitoring/#configuring-… > > > > Only the docs for Ceph 21 got an update and mention certmgr: > > > https://docs.ceph.com/en/umbrella/cephadm/services/monitoring/#configuring-… > > > > > > The orchestrator in Ceph 19 seems to have something called cert-store: > > > > # ceph orch cert-store > > no valid command found; 4 closest matches: > > orch cert-store cert ls [--format > > {plain|json|json-pretty|yaml|xml-pretty|xml}] > > orch cert-store key ls [--format > > {plain|json|json-pretty|yaml|xml-pretty|xml}] > > orch cert-store get cert <entity> [--service_name <value>] [--hostname > > <value>] [--no-exception-when-missing] > > orch cert-store get key <entity> [--service_name <value>] [--hostname > > <value>] [--no-exception-when-missing] > > Error EINVAL: invalid command > > > > But there is no help available. > > "ceph orch cert-store --help" does not return anything specific. > > Searching for "cert-store" in the Ceph 19 docs also does not return > > anything. > > > > Regards > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: Tentacle v20.2.4 packages build for Ubuntu Noble (24.04)
by Eugen Block 02 Sep '26

02 Sep '26

Hi Patrick, thanks for the efforts to bring 20.2.4 to Noble! I just installed those packages on a virtual machine that I use for testing. I can create a cluster successfully (package based), but the MGR ends up in a respawn loop, reporting these error messages: ---snip--- Sep 02 10:27:10 tentacle-ubuntu24 ceph-mgr[3289]: Failed to import NVMeoFClient and related components: cannot import name 'NVMeoFClient' from 'dashboard.services.nvmeof_client' (/usr/share/ceph/mgr/dashboard/services/nvmeof_client.py) Sep 02 10:27:10 tentacle-ubuntu24 ceph-mgr[3289]: ERROR:root:Module 'xmltodict' is not installed. ---snip--- The xmltodict error can easily be fixed by installing python3-xmltodict, but it probably should be a dependency then when installing the ceph-mgr package, I assume? The NVMeoFClient message doesn't seem to be critical, at least I can't notice anything obvious right now, but I also have no nvme service deployed or anything. The dashboard also works fine. Should I create tracker for the xmltodict error? Thanks! Eugen Zitat von Patrick Donnelly <ceph-users(a)ceph.io>: > Hi everyone, > > Due to an internal miscommunication, packages were inadvertently built > for Ubuntu Noble in Tentacle v20.2.1 [1] and v20.2.2 [2]. Following > those releases, I had published a roadmap matrix of distribution > support [3] which marked Noble support as "Upcoming" for Tentacle, as > testing had not yet been configured for it. Subsequently, the build > team used that roadmap as the authoritative source of support, so > Noble was not built or packaged for v20.2.3 or v20.2.4. > > We recognize this puts community clusters that were using Noble > packages for v20.2.1 and v20.2.2 in a difficult position due to the > CVE fixes. So we've generated signed Ubuntu 24.04 packages for v20.2.4 > (not v20.2.3). Please understand that these packages have not been > tested in any capacity beyond basic validation that they install. > > As I've mentioned in other threads on the ceph-users list, I expect > v20.2.5 [4] to have full and tested support for Ubuntu Noble. > > [1] https://download.ceph.com/debian-20.2.1/dists/noble/ > [2] https://download.ceph.com/debian-20.2.2/dists/noble/ > [3] https://docs.ceph.com/en/latest/start/os-recommendations/ > [4] https://tracker.ceph.com/issues/79030 > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Tentacle v20.2.4 packages build for Ubuntu Noble (24.04)
by Patrick Donnelly 01 Sep '26

01 Sep '26

Hi everyone, Due to an internal miscommunication, packages were inadvertently built for Ubuntu Noble in Tentacle v20.2.1 [1] and v20.2.2 [2]. Following those releases, I had published a roadmap matrix of distribution support [3] which marked Noble support as "Upcoming" for Tentacle, as testing had not yet been configured for it. Subsequently, the build team used that roadmap as the authoritative source of support, so Noble was not built or packaged for v20.2.3 or v20.2.4. We recognize this puts community clusters that were using Noble packages for v20.2.1 and v20.2.2 in a difficult position due to the CVE fixes. So we've generated signed Ubuntu 24.04 packages for v20.2.4 (not v20.2.3). Please understand that these packages have not been tested in any capacity beyond basic validation that they install. As I've mentioned in other threads on the ceph-users list, I expect v20.2.5 [4] to have full and tested support for Ubuntu Noble. [1] https://download.ceph.com/debian-20.2.1/dists/noble/ [2] https://download.ceph.com/debian-20.2.2/dists/noble/ [3] https://docs.ceph.com/en/latest/start/os-recommendations/ [4] https://tracker.ceph.com/issues/79030 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Eugen Block 01 Sep '26

01 Sep '26

If you're suspecting ceph-csi to be the issue here, I recommend to create an issue here (if you haven't already): https://github.com/ceph/ceph-csi/ Zitat von Stefan Möding via ceph-users <ceph-users(a)ceph.io>: > Hi, > > thanks for your feedback! > > Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > >> that sounds a bit impractical to create a new pool for each tenant >> regarding PG and pool management (adding and possibly removing data >> pools from the CephFS), depending on how many tenants you have or plan >> to have. > > In our setup each K8S cluster would be on tenant. So the pools would be > on the subvolumegroup level and we expect to go ahead with less than ten > pools for the foreseeable future. > > >> What I just tested was to manually encrypt a directory in a test >> cluster with fscrypt, that seems to work as intended. But I don't have >> anything with k8s to test it at the moment. > > From what I have tested, the problem seems to be located in ceph-csi > where the encryption is initially set up. The code tries to allocates > an exclusive lock on an object representing the volume to prevent a race > condition with multiple clients sharing the volume. The relevant code > looks like this: > > ioctx, err := volOptions.GetConnection().GetIoctx(volOptions.MetadataPool) > if err != nil { > log.ErrorLog(ctx, "Failed to create ioctx: %s", err) > return err > } > lock := iolock.NewLock(ioctx, objectUUID, lockName, lockCookie, > lockDesc, lockDuration) > err = lock.LockExclusive(ctx) > if err != nil { > log.ErrorLog(ctx, "failed to create lock for volume ID %s: %v", volID, err) > return err > } > > The error message on the ceph/csi pods clearly shows that the > LockExclusive call fails with “Operation not permitted”. I failed to > find the place in the relevant code, where the namespace for this > operation is set, so it looks like the default namespace is used. But > the capabilities only allow a specific namespace and so this could be > the root cause. > > I tried to use the following code to create an object in the metadata > pool and lock it. I used the same user with the same capabilities: > > % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put > foobar /dev/null > % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock > get foobar lock1 --lock-cookie=cookie --lock-duration=60 > --lock-type=exclusive > % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock > info foobar lock1 > {"name":"lock1","type":"exclusive","tag":"","lockers":[{"name":"client.2690590","cookie":"cookie","description":"","expiration":"2026-07-31T08:21:36.190835+0200","addr":"172.21.70.10:0/2292729948"}]} > > So the capabilities seem to be sufficient for the lock operation. But > leaving out the namespace when creating the lock will show the exact > same error message that the ceph-csi shows: > > % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put > foobar /dev/null > % rados --id=k8s-test --pool=cephfs.meta lock get foobar lock1 > --lock-cookie=cookie --lock-duration=60 --lock-type=exclusive > ERROR: failed locking: (1) Operation not permitted > error 1: (1) Operation not permitted > > -- > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Python version for cephadm orchestrator
by Robert Sander 31 Aug '26

31 Aug '26

Hi, is there a possibility to set the Python interpreter path that the cephadm orchestrator uses on the nodes? We currently deal with a cluster that runs on SLES 15.7 where the default Python version is still 3.6 and /usr/bin/python3 points to that version. Python 3.11 is available as /usr/bin/python3.11 Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

19.2.6 garbage collection: RGWGC::send_split_chain - send chain returned error: -28
by Manuel Rios - EDH 31 Aug '26

31 Aug '26

Hi , Since last patch 19.2.6 squid (stable), when we delete a bucket with the common cli: radosgw-admin bucket rm --bucket=XXXXX --purge-objects We get this error: 2026-08-31T08:10:45.237+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:10:47.394+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:10:59.289+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:11:21.806+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:11:23.422+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:11:31.995+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 Know error or new bug? ADVERTENCIA LEGAL: Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado. Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. [Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Flor ecoTECH]No imprimas si no es necesario. Protejamos el Medio Ambiente.

1 0

Re: Affects of high pg count on OSD peering
by Elias Carter 31 Aug '26

31 Aug '26

Regarding osd_heartbeat_interval, I set it to 1 in our production clusters and did some tests with marking OSDs as out/in. From my tests, I did not see any noticeable difference on client side IOs between osd_heartbeat_interval=6 and osd_heartbeat_interval=1. In both cases p99 RBD latency increased by about 40% for ~10 seconds after marking OSDs as out/in. Elias On Wed, Aug 19, 2026 at 9:17 AM Elias Carter <ceph-users(a)ceph.io> wrote: > > Can you share these values? > > bash-5.1$ ceph config get osd osd_max_pg_per_osd_hard_ratio > 3.000000 > bash-5.1$ ceph config get mon mon_max_pg_per_osd > 600 > > On Tue, Aug 18, 2026 at 11:00 PM Eugen Block <eblock(a)nde.ag> wrote: > > > Can you share these values? > > > > osd_max_pg_per_osd_hard_ratio > > mon_max_pg_per_osd > > > > Zitat von Elias Carter <elias(a)dropbox.com>: > > > > >> Oh, in your first message you wrote you have 300 PGs per OSD, which I > > > didn’t find that high. Now you wrote 600 PGs per OSD, which is it? > > > > > > My mistake, our target is 300 PGs per OSD and the maximum is 600 PGs > per > > > OSD. Currently, our clusters are hovering around the target of 300 PGs > > per > > > OSD. > > > > > >> Can you share some details about the OSDs (size)? > > > > > > We are running 14TiB NVMe SSD drives (PCIe 4). Our largest cluster > right > > > now has approximately 400 OSDs. > > > > > > Regarding tuning osd_heartbeat_interval, I did gradually lower it from > 6 > > to > > > 1 and found that packets per second increased by about 50% on the OSDs. > > > Although we are only seeing tens of thousands of packets per second per > > > OSD, these nodes have NICs rated for 100 million packets per second, so > > we > > > are not even close to hitting limits there. So far I don't see any > > cluster > > > instability caused by setting osd_heartbeat_interval to 1 in our > > clusters. > > > My assumption is that the number of heartbeat packets will increase > > > quadratically with the number of OSDs in the cluster (given 300 PGs/OSD > > > there is a good likelihood that any one OSD peers with almost every > other > > > OSD?). > > > > > > Elias > > > > > > On Tue, Aug 18, 2026 at 1:15 PM Eugen Block <eblock(a)nde.ag> wrote: > > > > > >> Oh, in your first message you wrote you have 300 PGs per OSD, which I > > >> didn’t find that high. Now you wrote 600 PGs per OSD, which is it? 😁 > > >> 600 sounds like a lot indeed, that could be one of the downsides of > > >> having too many PGs per OSD. Some time ago there was a thread here on > > >> the list about the ancient recommendation of 100 per OSD and nobody > > >> seemed to know the exact reason. Several people (including me) > > >> mentioned good experiences with around 300 to 400 PGs per OSD. Maybe > > >> you found the max value 😅 > > >> > > >> Can you share some details about the OSDs (size)? > > >> > > >> Zitat von Elias Carter <elias(a)dropbox.com>: > > >> > > >> >> the IO should not be blocked for the entire pool but only for those > > >> > repeering PGs > > >> > > > >> > Yes, the issue is that when running 600 PGs per OSD a high fraction > of > > >> PGs > > >> > re-peer simultaneously. > > >> > > > >> >> Gradually reweighting instead of setting an OSD "out" directly is a > > >> > common practice among ceph users, there should be plenty of threads > on > > >> > this list about that. The downside is of course a lot more manual > > >> > intervention and possibly (some) redundant data movement. > > >> > > > >> > Thank you for the confirmation. We run Ceph within kubernetes (rook) > > and > > >> my > > >> > plan is to write an operator which will handle gradually reweighting > > OSDs > > >> > as they are added/removed from the cluster. Rook allows setting the > > >> initial > > >> > OSD weight, so my plan is to set the initial OSD weights to 0 and > then > > >> have > > >> > the operator gradually reweight the OSDs to 1. > > >> > > > >> >> One knob to turn could be to decrease osd_heartbeat_interval > (number > > >> > of seconds between peer pings) from 6 seconds to a lower value (1 > > >> > second is the minimum, but I probably would start with 5 or 4 > > >> > seconds). > > >> > > > >> > Thank you; I will try that out. We have a mix of 100G and 400G > > networks, > > >> so > > >> > I think the increased ping frequency will be ok. I will try it out > on > > our > > >> > staging clusters first and report back if there are any interesting > > >> > findings. > > >> > > > >> > Elias > > >> > > > >> > On Tue, Aug 18, 2026 at 2:21 AM Eugen Block <ceph-users(a)ceph.io> > > wrote: > > >> > > > >> >> Hi, > > >> >> > > >> >> the IO should not be blocked for the entire pool but only for those > > >> >> repeering PGs. Warnings during that phase are expected. > > >> >> Gradually reweighting instead of setting an OSD "out" directly is a > > >> >> common practice among ceph users, there should be plenty of threads > > on > > >> >> this list about that. The downside is of course a lot more manual > > >> >> intervention and possibly (some) redundant data movement. > > >> >> > > >> >> One knob to turn could be to decrease osd_heartbeat_interval > (number > > >> >> of seconds between peer pings) from 6 seconds to a lower value (1 > > >> >> second is the minimum, but I probably would start with 5 or 4 > > >> >> seconds). That could (or probably would) increase network traffic > > >> >> significantly, so it's a trade-off. If you have a test cluster with > > >> >> similar behavior, test it there first. > > >> >> > > >> >> Regards, > > >> >> Eugen > > >> >> > > >> >> Zitat von Elias Carter via ceph-users <ceph-users(a)ceph.io>: > > >> >> > > >> >> > We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total > > PGs > > >> and > > >> >> > upwards). > > >> >> > > > >> >> > Something that I have noticed is that CRUSH changes (marking an > > OSD as > > >> >> out, > > >> >> > or a new OSD joining the cluster) can cause many hundreds of PGs > to > > >> remap > > >> >> > and peer. During this time we get PG_AVAILABILITY health check > > >> warnings > > >> >> as > > >> >> > the OSDs need to peer with their new PGs before they can serve > IOs. > > >> RADOS > > >> >> > tail latency suffers because IOs sent to a peering PG are blocked > > >> until > > >> >> > peering completes. > > >> >> > > > >> >> > I believe that the more PGs we have, the more PGs need to peer, > and > > >> the > > >> >> > longer IOs are blocked for the pool. > > >> >> > > > >> >> > Does anyone have suggestions on how to deal with PG peering > > blocking > > >> IOs? > > >> >> > > > >> >> > I was thinking about gradually re-weighting OSDs instead of > marking > > >> them > > >> >> > out or in, but that can result in multiple redundant data moves. > > >> >> > > > >> >> > Elias > > >> >> > _______________________________________________ > > >> >> > ceph-users mailing list -- ceph-users(a)ceph.io > > >> >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> >> > > >> >> > > >> >> _______________________________________________ > > >> >> ceph-users mailing list -- ceph-users(a)ceph.io > > >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> >> > > >> > > >> > > >> > > >> > > > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1