April 2026 - ceph-users - lists.ceph.io

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 15 Apr '26

15 Apr '26

Hi, some comments inline. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi Eugen! > > Thanks for the support, my response is inline > > On 14/04/26 13:03, Eugen Block via ceph-users wrote: >> Hi, >> >> I don't have a solution (yet?), but a couple of points to clarify. >> >> - I assume your cluster isn't managed by cephadm, correct? > > Correct! It's an old cluster initialy deployed with ceph-deploy in > the Hammer days and upgraded since then. Right now is manually > managed (but one of the targets at the end of the upgrade is to > switch to cephadm). > > >> - Can you share the output of 'ceph mon dump'? > > Sure! > > # ceph mon dump > epoch 18 > fsid 5bed1bf6-123f-4597-aaaa-d77931228548 > last_changed 2026-04-08T20:23:59.781295+0200 > created 2018-01-09T10:20:36.351967+0100 > min_mon_release 15 (octopus) > election_strategy: 1 > 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 > 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 > 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 > dumped monmap epoch 18 So your cluster currently has quorum with 2 out of 3 MONs on Quincy already, correct? Restarting the mon on node3 shouldn't be dangerous, I think. You'd still have quorum with node4 and node5, even if node3 wouldn't be able to join. If it doesn't join as Octopus, you could still upgrade it and retry with Quincy. What I'm wondering about, why did you remove the MONs you wanted to upgrade from Ceph? Is that recommended somewhere? Maybe you need to run 'ceph mon add <name> <addr>' as well to let node1 or node2 join the monmap again? I'm not sure, it has been a long time doing this kind of stuff. ;-) But I don't think I ever removed a MON before upgrading it, not even when reinstalling the OS. Just for completeness sake, could you also share 'ceph -s' and 'ceph versions'? >> - There were a couple of reports back at O/P versions about MONs >> not joining the quorum. Sometimes it worked after hours, sometimes >> it didn't, and I'm still not aware of a real solution. One of our >> customers has been facing severe MON sync delay, sometimes taking >> around 40 minutes until a MON joined quorum. It turned out to be >> caused by some paxos default settings, requring the MONs to do a >> full sync instead of a diff sync. This has to do with millions of >> snapshots they have in their cluster. What made it worse was the >> MON's filesystems were on HDDs, increasing the sync time as well. > > We didn't wait for a long time before stopping the new MON a regain > "control" of the cluster, maybe 10 minutes max... > >> - Inspecting debug logs on the syncing MON could help finding the >> root cause. > > With "syncing MON" you mean the MON which is just added to the > cluster, right? Collecting this kind of logs is on the to-do list, > but blocked because the "log problem" mentioned bellow. Yes, that's what I meant. The missing logs are a problem, of course. It does sound familiar, but I don't have a good idea right now except for looking at directory and file permissions, apparmor/selinux things, dmesg, stuff like that. >> >> Some more comments inline. >> > > [cut] > >>> >>> We taught, among other things, to: >>> >>> - Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is >>> the only daemon which wasn't restarted and when queried ("ceph >>> daemon mon.node1 mon_status") has in "extra_probe_peers" the IP >>> addresses of the original removed nodes, which are the same IPs as >>> the nodes we want to add. We aren't sure if this prevent the >>> ceph-mon to join, because we don't understand when/what >>> "extra_probe_peers" is doing. >> >> According to the output above node1 is already out of the cluster, >> are you referring to node3? >> >>> - Remove the node1 from the cluster, set min_mon_release to Quincy >>> and then try to add the new monitors. > > Yes, sorry for the misunderstanding, I was referring to node3 in > both instances. > > >>> >>> We are unsure about both approaches because if they don't work we >>> will have a "unstable" cluster with only 2 monitors and no much >>> space to operate. >>> >>> To "complicate" the matter we notice that on node4 and node5 the >>> ceph- mon is not logging anymore to "/var/log/ceph/ceph*.log" even >>> tough a "mon_cluster_log_to_file: true". If I search for open FD >>> there is nothing in /var/log/ceph for the running ceph-mon... >> >> Is there something in journalctl? > > We increased the 'debug_mon' to 20 ('ceph daemon mon.node4 config > set debug_mon 20') and we check it with 'ceph daemon mon.node4 > config show' but the output of 'journalctl -u ceph-mon(a)mon.node4` is > not more verbose then before, the last line is from 2:00 this > morning and it's related to sudo: > > apr 14 02:35:40 node4 sudo[881890]: ceph : PWD=/ ; USER=root ; > COMMAND=/usr/sbin/smartctl -x --json=o /dev/md127 > > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Malte Stroem 14 Apr '26

14 Apr '26

Hi Marek, add storage to the cluster. Please show us: ceph -s Best, Malte On 4/14/26 17:53, Marek Szuba via ceph-users wrote: > Hello, > > After the latest run of system updates and reboots on our prototype Ceph > cluster (Ubuntu Jammy + cephadm-managed Ceph 20.2.0 on Podman), we have > ended up with something seriously weird. Specifically, at a glance the > cluster appears to be in service: > > * Ceph reports the health of the cluster as OK > * according to both 'ceph -s' and 'ceph orch ps', all the daemons which > should be up and running ARE up and running. In particular, all the > expected OSDs appear to be operational > * 'ceph -s' shows no recovery I/O > > However: > > * despite all of the above, 'ceph -s' reports 1448 remapped PGs and > about 24% objects classified as misplaced > * 'ceph pg ls remapped' does report a long list of PGs with misplaced > objects > * on said list, most if not all PGs seem (I haven't checked all 1448 > lines) to have 3 OSDs each listed under ACTING but only either 1 or 2 > listed under UP - but not in a consistent way, in that an OSD missing > from the UP list for one PG does appear on the UP lists of other PGs > > Any idea what might be going on here, and how to make it stop? We were > about to upgrade that cluster to 20.2.1 following the system updates, > however we are very much hesitant to do so while things look so odd in > case the upgrade makes things worse. >

1 0

Ceph User + Dev April Meetup
by Anthony Middleton 14 Apr '26

14 Apr '26

Hey all, The next Ceph User+Dev meetup is tomorrow, April 15th, at 14:00 UTC. Add your discussion topics here: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes I hope to see more people there! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Eugen Block 14 Apr '26

14 Apr '26

First thing coming to mind is: ceph mgr fail If you already did that something else is going on. Zitat von Marek Szuba via ceph-users <ceph-users(a)ceph.io>: > Hello, > > After the latest run of system updates and reboots on our prototype Ceph > cluster (Ubuntu Jammy + cephadm-managed Ceph 20.2.0 on Podman), we have > ended up with something seriously weird. Specifically, at a glance the > cluster appears to be in service: > > * Ceph reports the health of the cluster as OK > * according to both 'ceph -s' and 'ceph orch ps', all the daemons which > should be up and running ARE up and running. In particular, all the > expected OSDs appear to be operational > * 'ceph -s' shows no recovery I/O > > However: > > * despite all of the above, 'ceph -s' reports 1448 remapped PGs and > about 24% objects classified as misplaced > * 'ceph pg ls remapped' does report a long list of PGs with misplaced > objects > * on said list, most if not all PGs seem (I haven't checked all 1448 > lines) to have 3 OSDs each listed under ACTING but only either 1 or 2 > listed under UP - but not in a consistent way, in that an OSD missing > from the UP list for one PG does appear on the UP lists of other PGs > > Any idea what might be going on here, and how to make it stop? We were > about to upgrade that cluster to 20.2.1 following the system updates, > however we are very much hesitant to do so while things look so odd in > case the upgrade makes things worse. > > -- > MS > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Marek Szuba 14 Apr '26

14 Apr '26

Hello, After the latest run of system updates and reboots on our prototype Ceph cluster (Ubuntu Jammy + cephadm-managed Ceph 20.2.0 on Podman), we have ended up with something seriously weird. Specifically, at a glance the cluster appears to be in service: * Ceph reports the health of the cluster as OK * according to both 'ceph -s' and 'ceph orch ps', all the daemons which should be up and running ARE up and running. In particular, all the expected OSDs appear to be operational * 'ceph -s' shows no recovery I/O However: * despite all of the above, 'ceph -s' reports 1448 remapped PGs and about 24% objects classified as misplaced * 'ceph pg ls remapped' does report a long list of PGs with misplaced objects * on said list, most if not all PGs seem (I haven't checked all 1448 lines) to have 3 OSDs each listed under ACTING but only either 1 or 2 listed under UP - but not in a consistent way, in that an OSD missing from the UP list for one PG does appear on the UP lists of other PGs Any idea what might be going on here, and how to make it stop? We were about to upgrade that cluster to 20.2.1 following the system updates, however we are very much hesitant to do so while things look so odd in case the upgrade makes things worse. -- MS

1 0

Bluestore deferred writes (Squid)
by Magnus Grönlund 14 Apr '26

14 Apr '26

Hi All, We are running a 5 year old 15 node Ceph cluster with spinners for data and NVMe for DB/Wal. After the recent upgrade to Squid (from Reef) some users reported issues with fluctuating (often bad) disk performance especially affecting k8s etcd. They point to the change to bluestore_prefer_deferred_size_hdd claiming that it's detrimental to the "synchronous write patter" of escd. They also point to submitted_deferred_writes and issued_deferred_writes claiming that there is a massive and growing backlog of deferred writes. I'm not 100% correct that this is the issue but I'm having a hard time finding any relevant information (bad google-fu?) especially about what submitted and issued deferred writes actually represents and why the difference between issued - submitted keeps growing. Anyone got some insights to share? Thanks in advance Magnus

1 0

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 14 Apr '26

14 Apr '26

Hi Eugen! Thanks for the support, my response is inline On 14/04/26 13:03, Eugen Block via ceph-users wrote: > Hi, > > I don't have a solution (yet?), but a couple of points to clarify. > > - I assume your cluster isn't managed by cephadm, correct? Correct! It's an old cluster initialy deployed with ceph-deploy in the Hammer days and upgraded since then. Right now is manually managed (but one of the targets at the end of the upgrade is to switch to cephadm). > - Can you share the output of 'ceph mon dump'? Sure! # ceph mon dump epoch 18 fsid 5bed1bf6-123f-4597-aaaa-d77931228548 last_changed 2026-04-08T20:23:59.781295+0200 created 2018-01-09T10:20:36.351967+0100 min_mon_release 15 (octopus) election_strategy: 1 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 dumped monmap epoch 18 > - There were a couple of reports back at O/P versions about MONs not > joining the quorum. Sometimes it worked after hours, sometimes it > didn't, and I'm still not aware of a real solution. One of our customers > has been facing severe MON sync delay, sometimes taking around 40 > minutes until a MON joined quorum. It turned out to be caused by some > paxos default settings, requring the MONs to do a full sync instead of a > diff sync. This has to do with millions of snapshots they have in their > cluster. What made it worse was the MON's filesystems were on HDDs, > increasing the sync time as well. We didn't wait for a long time before stopping the new MON a regain "control" of the cluster, maybe 10 minutes max... > - Inspecting debug logs on the syncing MON could help finding the root > cause. With "syncing MON" you mean the MON which is just added to the cluster, right? Collecting this kind of logs is on the to-do list, but blocked because the "log problem" mentioned bellow. > > Some more comments inline. > [cut] >> >> We taught, among other things, to: >> >> - Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is the >> only daemon which wasn't restarted and when queried ("ceph daemon >> mon.node1 mon_status") has in "extra_probe_peers" the IP addresses of >> the original removed nodes, which are the same IPs as the nodes we >> want to add. We aren't sure if this prevent the ceph-mon to join, >> because we don't understand when/what "extra_probe_peers" is doing. > > According to the output above node1 is already out of the cluster, are > you referring to node3? > >> - Remove the node1 from the cluster, set min_mon_release to Quincy and >> then try to add the new monitors. Yes, sorry for the misunderstanding, I was referring to node3 in both instances. >> >> We are unsure about both approaches because if they don't work we will >> have a "unstable" cluster with only 2 monitors and no much space to >> operate. >> >> To "complicate" the matter we notice that on node4 and node5 the ceph- >> mon is not logging anymore to "/var/log/ceph/ceph*.log" even tough a >> "mon_cluster_log_to_file: true". If I search for open FD there is >> nothing in /var/log/ceph for the running ceph-mon... > > Is there something in journalctl? We increased the 'debug_mon' to 20 ('ceph daemon mon.node4 config set debug_mon 20') and we check it with 'ceph daemon mon.node4 config show' but the output of 'journalctl -u ceph-mon(a)mon.node4` is not more verbose then before, the last line is from 2:00 this morning and it's related to sudo: apr 14 02:35:40 node4 sudo[881890]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/md127 Iztok

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 14 Apr '26

14 Apr '26

Hi, I don't have a solution (yet?), but a couple of points to clarify. - I assume your cluster isn't managed by cephadm, correct? - Can you share the output of 'ceph mon dump'? - There were a couple of reports back at O/P versions about MONs not joining the quorum. Sometimes it worked after hours, sometimes it didn't, and I'm still not aware of a real solution. One of our customers has been facing severe MON sync delay, sometimes taking around 40 minutes until a MON joined quorum. It turned out to be caused by some paxos default settings, requring the MONs to do a full sync instead of a diff sync. This has to do with millions of snapshots they have in their cluster. What made it worse was the MON's filesystems were on HDDs, increasing the sync time as well. - Inspecting debug logs on the syncing MON could help finding the root cause. Some more comments inline. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi to all! > > I'm writing this (long) email to seek the wisdom of the community > about a problem we are facing in this moment. > > As I wrote before we would like to upgrade one of our Ceph cluster > from Octopus to Quincy (and then to Squid) and, at the same time, > upgrade the O.S. from centos7/rocky8 to rocky9. > > Our plan is (was?) to start from the MON/MGR nodes, bring those > nodes to Quincy and rocky9 and then re-install all the OSD servers > (one-by-one). Our initial setup for the MON/MGR nodes was: > > - node1: centos7+ceph15.2.16 > - node2: centos7+ceph15.2.16 > - node3: centos7+ceph15.2.16 > - node4: rocky8+ceph15.2.17 > - node5: rocky8+ceph15.2.17 > > To begin with we removed node1 and node2 from the cluster ("ceph mon > remove mon.node1") and reinstall it with rocky9+ceph17.2.9. > > As next steps we taught to: > 1. upgrade Ceph in-place on node4 and node5 to ceph17.2.6 > 2. add the freshly installed node1 and node2 to the cluster (with ceph17.2.9) > 3. remove, reinstall and re-add node3 to the cluster. > 4. remove, reinstall and re-add node4 and node5 to the cluster (to > have all rocky9 nodes). > > The first point went smoothly, but unfortunately the second did not > (and we are stopped there). > > We exported the monmap from the running cluster and created on the > freshly (re)installed hosts (node1, node2) the mon data directory. > Then we started the ceph-mon daemon on host node1. Once the daemon > started it got stuck in probing/electing, the commands to the > cluster (like "ceph -s") took a long time to return. If the ceph-mon > on node1 was stopped the command returned to work as usual and the > cluster reported (as it should) a warning about a monitor down (the > node1 we just stopped). Removing "mon.node1" from the cluster > returned the health to OK. > > Clearly everything screams about "network" and/or > "time-synchronization". But the networks seems fine, no firewall is > in place, ping between nodes is fine (checked also jumbo-frames) and > the services are able to communicate with each other. The same about > time, the clocks are all synchronized and no CLOCK_SKEW is detected. > > The current situation from the monitor standpoint is the following: > > in the cluster: > - node3: centos7+ceph15.2.16 > - node4: rocky8+ceph17.2.6 > - node5: rocky8+ceph17.2.6 > out-of-the-cluster: > - node1: rocky9+ceph17.2.9 > - node2: rocky9+ceph17.2.9 > > We are unable to decide what is our best course of action so we are > searching the wisdom from the community :-) > > We taught, among other things, to: > > - Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is the > only daemon which wasn't restarted and when queried ("ceph daemon > mon.node1 mon_status") has in "extra_probe_peers" the IP addresses > of the original removed nodes, which are the same IPs as the nodes > we want to add. We aren't sure if this prevent the ceph-mon to join, > because we don't understand when/what "extra_probe_peers" is doing. According to the output above node1 is already out of the cluster, are you referring to node3? > - Remove the node1 from the cluster, set min_mon_release to Quincy > and then try to add the new monitors. > > We are unsure about both approaches because if they don't work we > will have a "unstable" cluster with only 2 monitors and no much > space to operate. > > To "complicate" the matter we notice that on node4 and node5 the > ceph-mon is not logging anymore to "/var/log/ceph/ceph*.log" even > tough a "mon_cluster_log_to_file: true". If I search for open FD > there is nothing in /var/log/ceph for the running ceph-mon... Is there something in journalctl? > Have you some advice or you encounter similar problems in the past? > I didn't found anything, but probably I searched with the wrong > keywords. > > I can provide you with more information if needed. > > > Thanks a lot! > Iztok > > > > -- > Iztok Gregori > ICT Systems and Services > Elettra - Sincrotrone Trieste S.C.p.A. > http://www.elettra.eu > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Upgrade problem from Octopus to Quincy
by Iztok Gregori 14 Apr '26

14 Apr '26

Hi to all! I'm writing this (long) email to seek the wisdom of the community about a problem we are facing in this moment. As I wrote before we would like to upgrade one of our Ceph cluster from Octopus to Quincy (and then to Squid) and, at the same time, upgrade the O.S. from centos7/rocky8 to rocky9. Our plan is (was?) to start from the MON/MGR nodes, bring those nodes to Quincy and rocky9 and then re-install all the OSD servers (one-by-one). Our initial setup for the MON/MGR nodes was: - node1: centos7+ceph15.2.16 - node2: centos7+ceph15.2.16 - node3: centos7+ceph15.2.16 - node4: rocky8+ceph15.2.17 - node5: rocky8+ceph15.2.17 To begin with we removed node1 and node2 from the cluster ("ceph mon remove mon.node1") and reinstall it with rocky9+ceph17.2.9. As next steps we taught to: 1. upgrade Ceph in-place on node4 and node5 to ceph17.2.6 2. add the freshly installed node1 and node2 to the cluster (with ceph17.2.9) 3. remove, reinstall and re-add node3 to the cluster. 4. remove, reinstall and re-add node4 and node5 to the cluster (to have all rocky9 nodes). The first point went smoothly, but unfortunately the second did not (and we are stopped there). We exported the monmap from the running cluster and created on the freshly (re)installed hosts (node1, node2) the mon data directory. Then we started the ceph-mon daemon on host node1. Once the daemon started it got stuck in probing/electing, the commands to the cluster (like "ceph -s") took a long time to return. If the ceph-mon on node1 was stopped the command returned to work as usual and the cluster reported (as it should) a warning about a monitor down (the node1 we just stopped). Removing "mon.node1" from the cluster returned the health to OK. Clearly everything screams about "network" and/or "time-synchronization". But the networks seems fine, no firewall is in place, ping between nodes is fine (checked also jumbo-frames) and the services are able to communicate with each other. The same about time, the clocks are all synchronized and no CLOCK_SKEW is detected. The current situation from the monitor standpoint is the following: in the cluster: - node3: centos7+ceph15.2.16 - node4: rocky8+ceph17.2.6 - node5: rocky8+ceph17.2.6 out-of-the-cluster: - node1: rocky9+ceph17.2.9 - node2: rocky9+ceph17.2.9 We are unable to decide what is our best course of action so we are searching the wisdom from the community :-) We taught, among other things, to: - Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is the only daemon which wasn't restarted and when queried ("ceph daemon mon.node1 mon_status") has in "extra_probe_peers" the IP addresses of the original removed nodes, which are the same IPs as the nodes we want to add. We aren't sure if this prevent the ceph-mon to join, because we don't understand when/what "extra_probe_peers" is doing. - Remove the node1 from the cluster, set min_mon_release to Quincy and then try to add the new monitors. We are unsure about both approaches because if they don't work we will have a "unstable" cluster with only 2 monitors and no much space to operate. To "complicate" the matter we notice that on node4 and node5 the ceph-mon is not logging anymore to "/var/log/ceph/ceph*.log" even tough a "mon_cluster_log_to_file: true". If I search for open FD there is nothing in /var/log/ceph for the running ceph-mon... Have you some advice or you encounter similar problems in the past? I didn't found anything, but probably I searched with the wrong keywords. I can provide you with more information if needed. Thanks a lot! Iztok -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. http://www.elettra.eu

1 0

Re: Status of iSCSI-Gateway
by Alexander Schreiber 14 Apr '26

14 Apr '26

On Mon, Apr 13, 2026 at 02:28:55PM +0000, Eugen Block via ceph-users wrote: > No, my understanding was that you can give clients access to RBD's through > the nvme-gateway, they see it as an nvme drive. And due to Aviv's help on > slack I was able to confirm this understanding, I have a single node ceph > cluster (virtual machine), deployed nvme-of and see the RBD image as an nvme > drive within the VM now: > > test:~ # nvme list > Node Generic SN Model > Namespace Usage Format FW Rev > --------------------- --------------------- -------------------- > ---------------------------------------- ---------- > -------------------------- ---------------- -------- > /dev/nvme1n1 /dev/ng1n1 Ceph42382366687857 Ceph bdev > Controller 0x1 10.74 GB / 10.74 GB 512 B > + 0 B 25.05.1 So it is really just the transport, ok. Thank you. Kind regards, Alex. -- "Opportunity is missed by most people because it is dressed in overalls and looks like work." -- Thomas A. Edison

1 0

Re: Has anyone been able to use NFS in tentacle or it is really broken?
by Eugen Block 14 Apr '26

14 Apr '26

Good point, no it was not. After starting it and redeploying NFS (v3 is then in the config again) the daemon doesn't crash anymore. Thanks for pointing that out! Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Am 13.04.26 um 10:36 AM schrieb Eugen Block via ceph-users: >> >> the host where I try to deploy nfs (openSUSE Leap 16 virtual >> machine) has rpcbind installed and it still fails: >> >> storage:~ # rpm -qa | grep rpcbind >> rpcbind-1.2.7-160000.2.2.x86_64 >> > > Is the rpcbind service running? > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Has anyone been able to use NFS in tentacle or it is really broken?
by Robert Sander 14 Apr '26

14 Apr '26

Am 13.04.26 um 10:36 AM schrieb Eugen Block via ceph-users: > > the host where I try to deploy nfs (openSUSE Leap 16 virtual machine) > has rpcbind installed and it still fails: > > storage:~ # rpm -qa | grep rpcbind > rpcbind-1.2.7-160000.2.2.x86_64 > Is the rpcbind service running? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: squid 19.2.4 QE validation status
by Ilya Dryomov 13 Apr '26

13 Apr '26

On Mon, Apr 13, 2026 at 9:30 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75029#note-5 > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya rbd and krbd approved. Thanks, Ilya

1 0

Re: Questions about dm-cache
by Anthony D'Atri 13 Apr '26

13 Apr '26

> > On that we agree: WAL+DB dead is a dead OSD. I've seen a speculative outline on recovering the OSD but have not seen someone try. > And it's indeed a fair > tradeoff: I'd expect the bad backend drives to die long before Spinners often have a lower TBW and AFR than even low-endurance SSDs. > the enterprise write-intensive drives Do you find that you strictly need high endurance SSDs? 99% of enterprise SSDs never use more than 15% of their rated endurance. These days write-intensive usually means Optane-replacement SLC. More common here is mixed-use, which is typically the very same hardware as a "read intensive" SSD just with more overprovisioning. If you really need more endurance than comes out of the box with an RI device, you can often adjust the OP yourself at much lower cost. > I have for wal+db. (or, in the case of > spinning rust, I'd expect a mechanical failure long before a flash > failure). Most SSD failures are firmware and can be fixed in-situ. I've had to replace 4000 HDDs with a design flaw - scattered around the world. Nightmare > Thus, in the end, it's no different from a dedicated wal+db iff wal+db > stays dedicated: I've seen people suggest to keep wal+db colocated if > there's a fast flash in front of the drive, as wal+db would essentially > always be hot in the cache. Indeed I've read studies that claim a superset of benefits from a transparent cache compared to just WAL+DB. > (note: I know that the OSD thinks it directly talks to the crap drives, Reads are always served from the slow device, and writes over a certain size go there directly. I've seen use-cases where the hassle of WAL+DB offload doesn't actually make an appreciable improvement.

2 3

CSC meeting minutes 2026-04-13
by Casey Bodley 13 Apr '26

13 Apr '26

minutes from today's meeting of the Ceph Steering Committee: Discuss Umbrella release schedule - https://docs.ceph.com/en/latest/dev/release-checklists/#dev-kickoff - Based on release engineering meeting discussion - Feature freeze - end of May 2026 - RC phase June - Aug 2026 - GA September 2026 - where best to track features? trello/redmine/gh projects -- Core Trello - https://trello.com/b/ugTc2QFH/ceph-backlog -- redmine can list features with filter "Tracker" is "Feature" drop.ceph.com is unreachable post lab migration (https://tracker.ceph.com/issues/74517) - Infra team is aware, but cannot fix it for awhile - Is ceph-post-file still being used by people? If not, we may decide to remove the test. - ssh is blocked in new lab, perhaps a web form for uploads CSC Election - Stage two will run from April 15th to April 29th. This voting round will cover the following topics: -- CSC Co-Chairs -- Adding a removal policy to the CSC Governance -- CSC Representative, as stated in the CSC Charter - Candidates' speeches 1-2 minutes each. we heard from Dan van der Ster, Ernesto Puerta, Gregory Farnum, Neha Ojha, and Patrick Donnelly. Joseph Mundackal will email the csc list today - how does voting guarantee that no company has all 3 co-chairs? consensus: pick the first 2, and fall back to the next highest votes from a different company - co-chairs will select board rep amongst themselves should we enable immutable releases in the Ceph repo? Hardware gaps? CapEx planning.

1 0

Re: Status of iSCSI-Gateway
by Alexander Schreiber 13 Apr '26

13 Apr '26

On Mon, Apr 13, 2026 at 09:34:39AM +0000, Ernesto Puerta via ceph-users wrote: > Hey Eugen, > > AFAIK the Ceph NVMe-oF Gateway superseded the iSCSI one: it's actively > maintained, more performant, scalable, secure and better integrated with > the Ceph management & observability framework. > > I'm curious: what's the target platform/use case for iSCSI that NVMe/TCP > cannot fulfill? As someone who is running pure spinning rust Ceph clusters, I assume the NVMe-oF gateway requires actual NVMe devices to be used for backing, given that the docs mention sending NMVe commands to targets, correct? Handing out iSCSI devices sounds useful as backing storage for VMs (and some can even boot off it). Kind regards, Alex. -- "Opportunity is missed by most people because it is dressed in overalls and looks like work." -- Thomas A. Edison

2 1

Re: Status of iSCSI-Gateway
by Anthony D'Atri 13 Apr '26

13 Apr '26

> On Apr 13, 2026, at 8:32 AM, Alexander Schreiber via ceph-users <ceph-users(a)ceph.io> wrote: > > On Mon, Apr 13, 2026 at 09:34:39AM +0000, Ernesto Puerta via ceph-users wrote: >> Hey Eugen, >> >> AFAIK the Ceph NVMe-oF Gateway superseded the iSCSI one: it's actively >> maintained, more performant, scalable, secure and better integrated with >> the Ceph management & observability framework. The iscsi-gw has lacked a maintainer for quite a while. If someone really really needs to run iSCSI, they might check out https://github.com/frednass/ceph_petasan >> I'm curious: what's the target platform/use case for iSCSI that NVMe/TCP cannot fulfill? Clients that can't talk NVMe/TCP? E.g. some BIOS implementations? Certain VMware releases? > > As someone who is running pure spinning rust Ceph clusters, I assume the > NVMe-oF gateway requires actual NVMe devices to be used for backing, given > that the docs mention sending NMVe commands to targets, correct? I don't *think* that's the case, though I'd be curious re the use-case for the protocol backed by slow media. > > Handing out iSCSI devices sounds useful as backing storage for VMs > (and some can even boot off it). RBD through libvirt is an all-around better solution for that. Fewer moving parts, less overhead, lower resource utilization on the Ceph side. > > Kind regards, > Alex. > -- > "Opportunity is missed by most people because it is dressed in overalls and > looks like work." -- Thomas A. Edison > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: RGW: S3 bucket replication with account‑owned buckets on Ceph Squid 19.2.3
by Ashok Kumar M 13 Apr '26

13 Apr '26

Hello, I am testing S3 bucket replication on Ceph Squid 19.2.3 (RGW multisite with one realm, one zonegroup, and two zones: source and destination). Replication works as expected when: Source and destination buckets are owned by a regular RGW user. However, using the same replication configuration with account‑owned buckets (owner is an RGW account ID like RGWxxxxxxxxxxxxxxx) shows different behavior: The replication configuration is accepted. But new objects written to the source bucket are not replicated to the destination bucket. Additional observation: The S3 replication API is working for user buckets, but it is not working for account-owned buckets My questions Is S3 PutBucketReplication officially supported for account‑owned buckets in Ceph Squid 19.2.3? Are there any known limitations 0r recommended configuration or workaround to account‑owned bucket replication? Any guidance, references to documentation, or known issues would be very helpful. Thank you, Ashok MP Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 0

Re: Status of iSCSI-Gateway
by Aviv Caro 13 Apr '26

13 Apr '26

Note that NVMe-oF is GA'd as a part of tentacle. Earlier releases are limited/might not work. Regards, Aviv -----Original Message----- From: Eugen Block via dev <dev(a)ceph.io> Sent: Monday, 13 April 2026 13:26 To: Ernesto Puerta <epuertat(a)redhat.com> Cc: dev(a)ceph.io; ceph-users(a)ceph.io Subject: [EXTERNAL] Re: [ceph-users] Status of iSCSI-Gateway Yeah that's my first suspicion as well (HugePages), I'm gonna look into it. Thanks a lot! Zitat von Ernesto Puerta <epuertat(a)redhat.com>: > Maybe you're hitting issues configuring HugePages inside a VM? In any > case, there's an #nvmeof Slack channel and the team is always quick to > respond over there. > > Kind Regards, > Ernesto > > > On Mon, Apr 13, 2026 at 11:36 AM Eugen Block <eblock(a)nde.ag> wrote: > >> Hi, >> >> thanks for your quick response. >> >> The reason I'm asking about iSCSI is because I have been able to >> deploy and use it in earlier releases (the last one was probably >> Pacific), but I'm struggling with deploying NVMe-oF in a virtual lab >> cluster. I haven't had the time to really dig into it yet, so there's >> definitely a knowledge gap I need to close. Until I made some >> progress with NVMe-oF I thought maybe iSCSI could still be used. The >> use case is to grant access to an air-gapped Ceph cluster. >> >> Thanks! >> Eugen >> >> Zitat von Ernesto Puerta <epuertat(a)redhat.com>: >> >> > Hey Eugen, >> > >> > AFAIK the Ceph NVMe-oF Gateway superseded the iSCSI one: it's >> > actively maintained, more performant, scalable, secure and better >> > integrated with the Ceph management & observability framework. >> > >> > I'm curious: what's the target platform/use case for iSCSI that >> > NVMe/TCP cannot fulfill? >> > >> > Kind Regards, >> > Ernesto >> > >> > >> > On Mon, Apr 13, 2026 at 8:58 AM Eugen Block via ceph-users < >> > ceph-users(a)ceph.io> wrote: >> > >> >> Hi, >> >> >> >> could someone clarify if using iscsi via cephadm is still >> >> supported or not? According to [0] it's "in maintenance" since >> >> November 2022, so no active development. But it's still documented >> >> in all active releases and can be deployed in Squid or Tentacle >> >> (though I haven't checked if it actually works). >> >> Is it still supported to use iSCSI? >> >> >> >> Thanks! >> >> Eugen >> >> >> >> [0] >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.ceph.com >> >> _en_latest_rbd_iscsi-2Doverview_-23ceph-2Discsi&d=DwIFaQ&c=BSDicqB >> >> QBDjDI9RkVyTcHQ&r=Im4omk4NdceLPvgb6sEXw3r6y71Z9JDuTGk4GMFulW0&m=VD >> >> MpTn9kP-I6POYWFAW6K418IPSCrPIGEyC5M22ibV2sgB49EZVv3TE0VYFRXwsN&s=S >> >> pUmSLpyOcc2_lfh9pDoVoVxJCEwgOwDDvLgQah7Sl4&e= >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send >> >> an email to ceph-users-leave(a)ceph.io >> >> >> >> >> >> >> >>

1 0

Re: Status of iSCSI-Gateway
by Ernesto Puerta 13 Apr '26

13 Apr '26

Hey Eugen, AFAIK the Ceph NVMe-oF Gateway superseded the iSCSI one: it's actively maintained, more performant, scalable, secure and better integrated with the Ceph management & observability framework. I'm curious: what's the target platform/use case for iSCSI that NVMe/TCP cannot fulfill? Kind Regards, Ernesto On Mon, Apr 13, 2026 at 8:58 AM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > could someone clarify if using iscsi via cephadm is still supported or > not? According to [0] it's "in maintenance" since November 2022, so no > active development. But it's still documented in all active releases > and can be deployed in Squid or Tentacle (though I haven't checked if > it actually works). > Is it still supported to use iSCSI? > > Thanks! > Eugen > > [0] https://docs.ceph.com/en/latest/rbd/iscsi-overview/#ceph-iscsi > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 3

Re: Has anyone been able to use NFS in tentacle or it is really broken?
by Eugen Block 13 Apr '26

13 Apr '26

Hm, the host where I try to deploy nfs (openSUSE Leap 16 virtual machine) has rpcbind installed and it still fails: storage:~ # rpm -qa | grep rpcbind rpcbind-1.2.7-160000.2.2.x86_64 Regards, Eugen Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Hi, > > On 4/10/26 17:31, Salatiel Filho via ceph-users wrote: >> It is broken. >> https://tracker.ceph.com/issues/74307 >> https://tracker.ceph.com/issues/74191 > > Thanks for the links. > > It looks like installing rpcbind fixes the issue. > > Why does the orchestrator does not install rpcbind on the node where > an NFS daemon instance is to be deployed? > > Regards > -- > Robert Sander > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > http://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 220009 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Status of iSCSI-Gateway
by Eugen Block 13 Apr '26

13 Apr '26

Hi, could someone clarify if using iscsi via cephadm is still supported or not? According to [0] it's "in maintenance" since November 2022, so no active development. But it's still documented in all active releases and can be deployed in Squid or Tentacle (though I haven't checked if it actually works). Is it still supported to use iSCSI? Thanks! Eugen [0] https://docs.ceph.com/en/latest/rbd/iscsi-overview/#ceph-iscsi

1 0

Re: Has anyone been able to use NFS in tentacle or it is really broken?
by Robert Sander 12 Apr '26

12 Apr '26

Hi, On 4/10/26 17:31, Salatiel Filho via ceph-users wrote: > It is broken. > https://tracker.ceph.com/issues/74307 > https://tracker.ceph.com/issues/74191 Thanks for the links. It looks like installing rpcbind fixes the issue. Why does the orchestrator does not install rpcbind on the node where an NFS daemon instance is to be deployed? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

Re: AutoFS Timeouts with CephFS
by Anthony D'Atri 10 Apr '26

10 Apr '26

I’ve seen, of all things, an IDE periodically scanning homedirs. If that happens every five minutes .. > On Apr 10, 2026, at 3:16 PM, Paul Mezzanini via ceph-users <ceph-users(a)ceph.io> wrote: > > Is there something in cron (or some other system monitoring) that tickles mounted drives on a 5 minute interval? We leverage autofs heavily here and I've never seen that except when netdata decided it wanted to monitor EVERY MOUNT to make sure it wasn't out of disk space. > > -- > > Paul Mezzanini > Platform Engineer III > Research Computing > Rochester Institute of Technology > > > > > ________________________________________ > From: Greg Ballantine via ceph-users <ceph-users(a)ceph.io> > Sent: Friday, April 10, 2026 12:00 PM > To: ceph-users(a)ceph.io > Subject: [ceph-users] AutoFS Timeouts with CephFS > > Hi all, > > We are running into a strange issue with autofs and CephFS at UMBC. We are > attempting to manage CephFS volumes via autofs with an idle timeout, > however my volumes do not appear to be unmounting when idle. After some > digging, it appears when the timeout is set to five minutes the volumes > stay mounted indefinitely, but when the autofs timeout is set to 4.5 > minutes, the volumes unmount as expected. > > Our environment is: > > - > > Ceph cluster version: 18.2.7 > - > > Ceph client version: 18.2.1 kernel client > - > > OS: Red Hat 9.7 > - > > auto.master entry: "/umbc/rs /cm/shared/etc/autofs/mounts.rs > --timeout=300" > - > > mounts.rs entry: "archive -fstype=ceph,name=archive,fs=rrstor > ceph-mon.example.com:/archive" > > We have checked for open file descriptors using lsof, but the mount appears > idle, only showing automount processes as holding open file descriptors in > the mounts. Has anyone encountered similar behavior before and can comment > on what they think might be going on? > > Thank you, > Greg > > -- > Gregory Ballantine > HPC Specialist, Division of IT > University of Maryland, Baltimore County > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: AutoFS Timeouts with CephFS
by Paul Mezzanini 10 Apr '26

10 Apr '26

Is there something in cron (or some other system monitoring) that tickles mounted drives on a 5 minute interval? We leverage autofs heavily here and I've never seen that except when netdata decided it wanted to monitor EVERY MOUNT to make sure it wasn't out of disk space. -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology ________________________________________ From: Greg Ballantine via ceph-users <ceph-users(a)ceph.io> Sent: Friday, April 10, 2026 12:00 PM To: ceph-users(a)ceph.io Subject: [ceph-users] AutoFS Timeouts with CephFS Hi all, We are running into a strange issue with autofs and CephFS at UMBC. We are attempting to manage CephFS volumes via autofs with an idle timeout, however my volumes do not appear to be unmounting when idle. After some digging, it appears when the timeout is set to five minutes the volumes stay mounted indefinitely, but when the autofs timeout is set to 4.5 minutes, the volumes unmount as expected. Our environment is: - Ceph cluster version: 18.2.7 - Ceph client version: 18.2.1 kernel client - OS: Red Hat 9.7 - auto.master entry: "/umbc/rs /cm/shared/etc/autofs/mounts.rs --timeout=300" - mounts.rs entry: "archive -fstype=ceph,name=archive,fs=rrstor ceph-mon.example.com:/archive" We have checked for open file descriptors using lsof, but the mount appears idle, only showing automount processes as holding open file descriptors in the mounts. Has anyone encountered similar behavior before and can comment on what they think might be going on? Thank you, Greg -- Gregory Ballantine HPC Specialist, Division of IT University of Maryland, Baltimore County _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: AutoFS Timeouts with CephFS
by Stefan Möding 10 Apr '26

10 Apr '26

Hi! Greg Ballantine via ceph-users <ceph-users(a)ceph.io> writes: > We have checked for open file descriptors using lsof, but the mount appears > idle, only showing automount processes as holding open file descriptors in > the mounts. Has anyone encountered similar behavior before and can comment > on what they think might be going on? You might want to look for a monitoring/cron job that runs “df” or similar tools. I believe I have seen this issue with autofs even when only the filesystem size or free space is checked. -- Stefan

1 0

AutoFS Timeouts with CephFS
by Greg Ballantine 10 Apr '26

10 Apr '26

Hi all, We are running into a strange issue with autofs and CephFS at UMBC. We are attempting to manage CephFS volumes via autofs with an idle timeout, however my volumes do not appear to be unmounting when idle. After some digging, it appears when the timeout is set to five minutes the volumes stay mounted indefinitely, but when the autofs timeout is set to 4.5 minutes, the volumes unmount as expected. Our environment is: - Ceph cluster version: 18.2.7 - Ceph client version: 18.2.1 kernel client - OS: Red Hat 9.7 - auto.master entry: "/umbc/rs /cm/shared/etc/autofs/mounts.rs --timeout=300" - mounts.rs entry: "archive -fstype=ceph,name=archive,fs=rrstor ceph-mon.example.com:/archive" We have checked for open file descriptors using lsof, but the mount appears idle, only showing automount processes as holding open file descriptors in the mounts. Has anyone encountered similar behavior before and can comment on what they think might be going on? Thank you, Greg -- Gregory Ballantine HPC Specialist, Division of IT University of Maryland, Baltimore County

1 0

Re: Has anyone been able to use NFS in tentacle or it is really broken?
by Salatiel Filho 10 Apr '26

10 Apr '26

It is broken. https://tracker.ceph.com/issues/74307 https://tracker.ceph.com/issues/74191 Atenciosamente/Kind regards, Salatiel On Thu, Apr 9, 2026 at 9:47 AM Robert Sander via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I just tried to deploy an NFS service on a freshly installed 20.2.1 cluster and it fails. > > ceph nfs cluster create nfs01 "label:nfs01" > > There is exactly one host with the label nfs01. > > There are some entries in the log showing errors but the last two are > > - Cannot acquire credentials for principal nfs > - Cannot register NFS V3 on TCP > > /var/lib/ceph/2ac9b090-3363-11f1-bd6a-bc2411d682b7/nfs.nfs01.0.0.rgw01.hpfeuz/etc/ganesha/ganesha.conf > contained "Protocols 3, 4;" in the NFS_CORE_PARAM section. > > After removing the 3 the container starts. > > NFSv3 was added in commit 1e69ebc6ceaf in February 2024. It was not contained in Ceph squid. > > > Apr 09 14:23:18 HOST systemd[1]: Starting ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service - Ceph nfs.nfs01.0.0.HOST.hpfeuz for 2ac9b090-3363-11f1-bd6a-bc2411d682b7... > Apr 09 14:23:18 HOST podman[31327]: 2026-04-09 14:23:18.942654365 +0200 CEST m=+0.048118777 container create c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, CEPH_REF=tentacle, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260331, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, io.buildah.version=1.43.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) > Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:19.000318083 +0200 CEST m=+0.105782538 container init c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, io.buildah.version=1.43.0, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_REF=tentacle, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260331) > Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:19.004217805 +0200 CEST m=+0.109682235 container start c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, CEPH_REF=tentacle, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, org.label-schema.vendor=CentOS, ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260331, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, io.buildah.version=1.43.0, org.label-schema.license=GPLv2) > Apr 09 14:23:19 HOST bash[31327]: c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 > Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:18.922220554 +0200 CEST m=+0.027684991 image pull fb63cba66eea5516fbd9a1ec035c265abdbc1a1a8fc2dc0b51057557e49ca28c quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337 > Apr 09 14:23:19 HOST systemd[1]: Started ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service - Ceph nfs.nfs01.0.0.HOST.hpfeuz for 2ac9b090-3363-11f1-bd6a-bc2411d682b7. > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576. > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] rados_kv_traverse :CLIENT ID :EVENT :Failed to lst kv ret=-2 > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] rados_cluster_read_clids :CLIENT ID :EVENT :Failed to traverse recovery db: -2 > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend > Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] rados_cluster_end_grace :CLIENT ID :EVENT :Failed to remove rec-0000000000000001:0: -2 > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:9): Unknown parameter (allow_set_io_flusher_fail) > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs > Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP > Apr 09 14:23:22 HOST podman[31583]: 2026-04-09 14:23:22.269193428 +0200 CEST m=+0.027823817 container died c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, io.buildah.version=1.43.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=tentacle, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, org.label-schema.build-date=20260331, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>) > Apr 09 14:23:22 HOST podman[31583]: 2026-04-09 14:23:22.325861901 +0200 CEST m=+0.084492274 container remove c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, ceph=True, org.label-schema.vendor=CentOS, CEPH_REF=tentacle, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, org.label-schema.build-date=20260331, io.buildah.version=1.43.0, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) > Apr 09 14:23:22 HOST systemd[1]: ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service: Main process exited, code=exited, status=2/INVALIDARGUMENT > Apr 09 14:23:22 HOST systemd[1]: ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service: Failed with result 'exit-code'. > > Regards > -- > Robert Sander > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > http://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 220009 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

cephadm without containers
by Adrian Sevcenco 10 Apr '26

10 Apr '26

Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine) If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way? I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm Thank you! Adrian

11 20

Re: Questions about dm-cache
by Janne Johansson 10 Apr '26

10 Apr '26

> > A second cache mode is "writeback". Writeback delays writing data > > blocks from the cache back to the origin LV. This mode will > > increase performance, but the loss of a cache device can result in > > lost data. > > So, so much for that. Writethrough will not help my situation, and writeback can cause data loss. This leads to two questions: This isn't much different from OSD on one slow drive and WAL+DB on fast(er) drive. If wal+db disk dies, the OSD is for all practical purposes "dead" too. It is one of the things at least we accept when we use 4-6 WAL+DB-partitions on a single nvme for our spindrive hosts. Most of those have 12 drives so 2 or 3 nvmes are in there and yes losing an nvme would be bad, but still less bad than losing the host, which could occur at any time for some kernel crash or PSU fault, so losing an nvme for us, is like losing half a host, and our clusters need to be able to withstand losing a whole host so for us, it's a decent enough tradeoff. Regardless of how you set up "ssd" acceleration in front of slower disks, there will be a window where if the ssd breaks, bad things happen to the slow disk OSD behind it. Yes, write-back makes that window larger, but you probably won't get a zero timespan for catastrophes in any case. -- May the most significant bit of your life be positive.

2 1

Re: Questions about dm-cache
by Alexis Bauvin 10 Apr '26

10 Apr '26

> > A subsequent question is: is it transparent to the point I can setup the cache on a live OSD, or do I need to recreate it and have ceph-volume have a go *with* the cache setup, so it knows about it? > > The way I understand it is "ceph does not care about dm-cache" so I would need, at worst, to restart the OSD. > > > disclaimer: never used dm-cache in production, last experimented with it > several years ago. > > I seem to remember that the transparency of dm-cache comes with a risk: > LVM device name stays the same whether the cache device is attached or > not. Nothing keeps you from writing new data to the device with cache > device detached even if said caching device still contains some > unwritten data. > So you might lose that in-cache data using LVM volume when cache device > is not attached (e.g. after reboot, perhaps ceph grabbing the device > before cache device attach). And when re-attaching a cache device later > it might just kill the fs by replaying outdated cached data. > > I have used bcache in production with ceph. Bcache requires you to > format the block device as a bcache device, and the backing device knows > if cached data is missing and refuses to be mounted until the > corresponding cache device is attached and such cached data is > accessible. This provides better reliabilty, but with some upfront cost > and some additional parameters to set (cache mode, watermarks, ...) > > Still, this might just be outdated knowledge of the past, YMMV. Ah, this somewhat changes things. It appears your knowledge is is not outdated, as it is explicitly documented in the lvm-cache manpage (https://man7.org/linux/man-pages/man7/lvmcache.7.html) > A second cache mode is "writeback". Writeback delays writing data > blocks from the cache back to the origin LV. This mode will > increase performance, but the loss of a cache device can result in > lost data. So, so much for that. Writethrough will not help my situation, and writeback can cause data loss. This leads to two questions: Since Ceph is aware of dm-cache (inferred by the fact the docs mention it), will the OSD stop if the cache device is not present? My setup has WAL+DB on dedicated drives, and I planned to use the free space left on those drives as a cache. Since the OSD cannot work if WAL+DB absent, I'm in a situation where the cache can only be absent if the WAL+DB is absent too. That is, WAL+DB is here iff cache is here too. Is the reasoning correct? Or can there still be a race where OSD starts writing data to the volume before the kernel has caching setup? Thanks for the warning! Alexis > Matthias > > > > > > > Is it that I can just slap an lvm cache on the LVs created by ceph-volume and call it a day? > > > > Is this the canonical replacement to Ceph's native cache tiers? > > > > > > I guess it is one of the default alternatives now that tiering is off > > > the map. We do a lot of "put WAL + DB on nvme partition, data on > > > spin-disks" with something like 4 or 6 nvme partitions per OSD disk, > > > so this mostly improves database access parts and less of the bulk > > > data transfers, but we think that is ok. > > > > Yeah, HDD setups is what I tried to mimic, as I have the WAL+DB on actual write-intensive enterprise drives, but it can only do so much when the backend drive does GC pauses that last for 20 min and increase the IO latency to the 2s-20s range. I'm hoping caching can shield most workloads from it. > > > > > > Is the tuning of the cache settings somehow managed by Ceph or is it fully on the user? And, related, can the caching be declared in the orch OSD spec? > > > > > > If you stack devices in LVM, then I guess that is where any tuning > > > would have to happen, though I don't think there is a large need to > > > tune those settings a lot. The defaults should probably work fine for > > > most people. > > > > Noted. I don't expect to have to tune a lot, but given the (horrible) behavior of my drives, being able to tune flushes and invalidations is important. > > > > > > As for the context: I have a cluster with *very* bad SSDs (consumer DRAM-less BX500) and I'm trying to get it somewhat OK until the market for flash improves. As the problems are due to GC pauses from the drive caused by the sustained IO, I'm hoping that putting a cache tier (with smaller faster enterprise SSDs) in front of the drives will change the IO behavior enough for the drives to perform their GC without impacting the cluster. I already have the good smaller drives that have WAL+DB for the slow drives, but they have enough free space to cache the block. > > > > > > I don't have any idea on which method will be best in the end, but I > > > think dm-cache is stable enough for people to recommend it. > > > > That's what I gathered from poking around the internet. The fact that the ceph documentations mention it, while not an endorsement, is already meaningful for the viability of dm-cache. Thanks for the confirmation! > > > > Thank you for the reply and the pointers! > > > > > -- > > > May the most significant bit of your life be positive. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Has anyone been able to use NFS in tentacle or it is really broken?
by Eugen Block 10 Apr '26

10 Apr '26

Hi, I tend to agree. I'm trying to set up an nfs cluster in a lab and fail with v20. I also was under the impression that NFSv3 isn't supported anymore, so I was just as surprised as you to see it in the config. In a different lab cluster on Squid the nfs daemon starts immediately (without v3). Regards, Eugen Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Hi, > > I just tried to deploy an NFS service on a freshly installed 20.2.1 > cluster and it fails. > > ceph nfs cluster create nfs01 "label:nfs01" > > There is exactly one host with the label nfs01. > > There are some entries in the log showing errors but the last two are > > - Cannot acquire credentials for principal nfs > - Cannot register NFS V3 on TCP > > /var/lib/ceph/2ac9b090-3363-11f1-bd6a-bc2411d682b7/nfs.nfs01.0.0.rgw01.hpfeuz/etc/ganesha/ganesha.conf > contained "Protocols 3, 4;" in the NFS_CORE_PARAM section. > > After removing the 3 the container starts. > > NFSv3 was added in commit 1e69ebc6ceaf in February 2024. It was not > contained in Ceph squid. > > > Apr 09 14:23:18 HOST systemd[1]: Starting > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service > - Ceph nfs.nfs01.0.0.HOST.hpfeuz for > 2ac9b090-3363-11f1-bd6a-bc2411d682b7... > Apr 09 14:23:18 HOST podman[31327]: 2026-04-09 14:23:18.942654365 > +0200 CEST m=+0.048118777 container create > c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 > (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, CEPH_REF=tentacle, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260331, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, io.buildah.version=1.43.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.schema-version=1.0, > org.label-schema.vendor=CentOS) > Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:19.000318083 > +0200 CEST m=+0.105782538 container init > c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 > (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, io.buildah.version=1.43.0, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_REF=tentacle, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, > org.label-schema.build-date=20260331) > Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:19.004217805 > +0200 CEST m=+0.109682235 container start > c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 > (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, CEPH_REF=tentacle, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, org.label-schema.vendor=CentOS, ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260331, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, io.buildah.version=1.43.0, > org.label-schema.license=GPLv2) > Apr 09 14:23:19 HOST bash[31327]: > c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 > Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:18.922220554 > +0200 CEST m=+0.027684991 image pull > fb63cba66eea5516fbd9a1ec035c265abdbc1a1a8fc2dc0b51057557e49ca28c > quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337 > Apr 09 14:23:19 HOST systemd[1]: Started > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service > - Ceph nfs.nfs01.0.0.HOST.hpfeuz for > 2ac9b090-3363-11f1-bd6a-bc2411d682b7. > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > init_logging :LOG :NULL :LOG: Setting log level for all components > to NIV_EVENT > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file > successfully parsed > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed > limit on FDs to 1048576. > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully > initialized. > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > rados_kv_traverse :CLIENT ID :EVENT :Failed to lst kv ret=-2 > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > rados_cluster_read_clids :CLIENT ID :EVENT :Failed to traverse > recovery db: -2 > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :grace reload client info completed > from backend > Apr 09 14:23:19 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) > clid count(0) > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > rados_cluster_end_grace :CLIENT ID :EVENT :Failed to remove > rec-0000000000000001:0: -2 > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > main :NFS STARTUP :WARN :No export entries found in configuration > file !!! > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > config_errs_to_log :CONFIG :WARN :Config File > (/etc/ganesha/ganesha.conf:9): Unknown parameter > (allow_set_io_flusher_fail) > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > config_errs_to_log :CONFIG :WARN :Config File > (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > config_errs_to_log :CONFIG :WARN :Config File > (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully > removed for proper quota management in FSAL > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: > cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect > to socket /run/dbus/system_bus_socket: No such file or directory) > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > gsh_dbus_register_path :DBUS :CRIT > :dbus_connection_register_object_path called with no DBUS connection > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > gsh_dbus_register_path :DBUS :CRIT > :dbus_connection_register_object_path called with no DBUS connection > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > gsh_dbus_register_path :DBUS :CRIT > :dbus_connection_register_object_path called with no DBUS connection > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs > Apr 09 14:23:22 HOST > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: > 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] > __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP > Apr 09 14:23:22 HOST podman[31583]: 2026-04-09 14:23:22.269193428 > +0200 CEST m=+0.027823817 container died > c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 > (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, io.buildah.version=1.43.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=tentacle, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, org.label-schema.build-date=20260331, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team > <ceph-maintainers(a)ceph.io>) > Apr 09 14:23:22 HOST podman[31583]: 2026-04-09 14:23:22.325861901 > +0200 CEST m=+0.084492274 container remove > c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 > (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, ceph=True, org.label-schema.vendor=CentOS, CEPH_REF=tentacle, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, org.label-schema.build-date=20260331, io.buildah.version=1.43.0, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.name=CentOS Stream 9 Base Image, > FROM_IMAGE=quay.io/centos/centos:stream9) > Apr 09 14:23:22 HOST systemd[1]: > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service: > Main process exited, code=exited, status=2/INVALIDARGUMENT > Apr 09 14:23:22 HOST systemd[1]: > ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service: > Failed with result 'exit-code'. > > Regards > -- > Robert Sander > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > http://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 220009 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: v20.2.2 merge window is now open!
by Eugen Block 09 Apr '26

09 Apr '26

Hi, I am the author of that blog post referenced in the tracker (CephPGImbalance). I'm not a developer though, one of my co-workers who is a dev (not for Ceph though) helped creating that workaround. But I created that tracker and hoped one of the Ceph devs would pick it up and implement a real fix. We are not really familiar with the Ceph development processes, and my co-worker isn't familiar with Ceph internals, so it's difficult for us to create a real fix for the imbalance calculation. If anyone considers this a as a valid workaround that could land in the MGR module and also change the Prometheus alerts, we could create a PR containing exactly the workaround we implemented. But I have serious doubts about it, tbh. Thanks, Eugen Zitat von Matthew H via ceph-users <ceph-users(a)ceph.io>: > Hi, > > I don't see a PR for this, nor do I see the tracker having a > backport label to tentacle. > > It would be good to have you (or someone else) propose a PR, and it > can be added to the tracker for review. > > ________________________________ > From: Alfonso Escribano Merino via ceph-users <ceph-users(a)ceph.io> > Sent: Thursday, April 9, 2026 1:07 AM > To: Patrick Donnelly <pdonnell(a)redhat.com>; dev <dev(a)ceph.io> > Cc: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: v20.2.2 merge window is now open! > > Hello, > It woulb be nice to include this: > https://tracker.ceph.com/issues/71310<https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> > I have two classes of nvme disks, ones (TLC) with 7TB and others > (QLC) with 30TB for different pools and it would be great to have > the patch that is in #71310 merged into master to have it on 20.2.2 > and also backport to older releases. > > Thanks, > Alfonso Escribano > > Patrick Donnelly via ceph-users wrote: > > Hello everyone, > > v20.2.1 has finally been released! > > The tentacle merge window is open again. Please follow this tracker > ticket for updates and processes for getting backporrts merged: > > https://tracker.ceph.com/issues/75878<https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> > > For discussion, please visit #ceph-upstream-releases-tentacle on > Slack, respond to this mail, or come to the "Ceph Release Engineering" > on Thursdays (see Ceph community calendar). > > Thanks, > > > > -- > > > Alfonso Escribano > alfonso.escribano(a)meteologica.com<mailto:alfonso.escribano@meteologica.com> > > [http://www.meteologica.com/assets/images/logo.png]<https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fwww.meteologica.com…> > c/ Costa Brava 10 - 28034 Madrid > Office: +34 91 456 10 01 > > This email and any files transmitted with it are confidential and > may be privileged and are intended solely for the use of the > individual or entity to whom they are addressed. If you are not an > addressee, any unauthorized direct or indirect dissemination, > distribution, publication, reading, or copying of this message and > any attachments is strictly prohibited. If you have received this > email in error please notify the sender immediately, and delete this > email from your system. Email communication does not guarantee the > confidentiality, integrity, or proper receipt of the messages sent. > Meteológica, S.A. accepts no liability for such circumstances. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Questions about dm-cache
by Matthias Ferdinand 09 Apr '26

09 Apr '26

On Thu, Apr 09, 2026 at 03:22:00PM +0000, Alexis Bauvin via ceph-users wrote: > > Le 09/04/2026 16:44 CEST, Janne Johansson <icepic.dz(a)gmail.com> a écrit : > > > > > > Den tors 9 apr. 2026 kl 15:58 skrev Alexis Bauvin via ceph-users > > <ceph-users(a)ceph.io>: > > > Since a few versions, Ceph has deprecated caching tiers with no alternative mentioned. Thus I've been looking and somewhat settled on using dm-cache for two reasons: it's LVM and ceph-volume mentions it's supported and "transparent": > > > > > > > For ceph-volume, the use of dm-cache is transparent, there is no difference for the tool, and it treats dm-cache like a plain logical volume. > > > > > > However, what does this really mean? > > > > It means you do it disk-for-disk instead of a fast pool in front of a > > slow pool. In the end, the same results should be there, faster > > accesses for writes and recently accessed data. > > Oh, that makes sense. Thanks for the clarification, and it explains a lot. > > A subsequent question is: is it transparent to the point I can setup the cache on a live OSD, or do I need to recreate it and have ceph-volume have a go *with* the cache setup, so it knows about it? > The way I understand it is "ceph does not care about dm-cache" so I would need, at worst, to restart the OSD. disclaimer: never used dm-cache in production, last experimented with it several years ago. I seem to remember that the transparency of dm-cache comes with a risk: LVM device name stays the same whether the cache device is attached or not. Nothing keeps you from writing new data to the device with cache device detached even if said caching device still contains some unwritten data. So you might lose that in-cache data using LVM volume when cache device is not attached (e.g. after reboot, perhaps ceph grabbing the device before cache device attach). And when re-attaching a cache device later it might just kill the fs by replaying outdated cached data. I have used bcache in production with ceph. Bcache requires you to format the block device as a bcache device, and the backing device knows if cached data is missing and refuses to be mounted until the corresponding cache device is attached and such cached data is accessible. This provides better reliabilty, but with some upfront cost and some additional parameters to set (cache mode, watermarks, ...) Still, this might just be outdated knowledge of the past, YMMV. Matthias > > > > Is it that I can just slap an lvm cache on the LVs created by ceph-volume and call it a day? > > > Is this the canonical replacement to Ceph's native cache tiers? > > > > I guess it is one of the default alternatives now that tiering is off > > the map. We do a lot of "put WAL + DB on nvme partition, data on > > spin-disks" with something like 4 or 6 nvme partitions per OSD disk, > > so this mostly improves database access parts and less of the bulk > > data transfers, but we think that is ok. > > Yeah, HDD setups is what I tried to mimic, as I have the WAL+DB on actual write-intensive enterprise drives, but it can only do so much when the backend drive does GC pauses that last for 20 min and increase the IO latency to the 2s-20s range. I'm hoping caching can shield most workloads from it. > > > > Is the tuning of the cache settings somehow managed by Ceph or is it fully on the user? And, related, can the caching be declared in the orch OSD spec? > > > > If you stack devices in LVM, then I guess that is where any tuning > > would have to happen, though I don't think there is a large need to > > tune those settings a lot. The defaults should probably work fine for > > most people. > > Noted. I don't expect to have to tune a lot, but given the (horrible) behavior of my drives, being able to tune flushes and invalidations is important. > > > > As for the context: I have a cluster with *very* bad SSDs (consumer DRAM-less BX500) and I'm trying to get it somewhat OK until the market for flash improves. As the problems are due to GC pauses from the drive caused by the sustained IO, I'm hoping that putting a cache tier (with smaller faster enterprise SSDs) in front of the drives will change the IO behavior enough for the drives to perform their GC without impacting the cluster. I already have the good smaller drives that have WAL+DB for the slow drives, but they have enough free space to cache the block. > > > > I don't have any idea on which method will be best in the end, but I > > think dm-cache is stable enough for people to recommend it. > > That's what I gathered from poking around the internet. The fact that the ceph documentations mention it, while not an endorsement, is already meaningful for the viability of dm-cache. Thanks for the confirmation! > > Thank you for the reply and the pointers! > > > -- > > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Questions about dm-cache
by Janne Johansson 09 Apr '26

09 Apr '26

Den tors 9 apr. 2026 kl 15:58 skrev Alexis Bauvin via ceph-users <ceph-users(a)ceph.io>: > Since a few versions, Ceph has deprecated caching tiers with no alternative mentioned. Thus I've been looking and somewhat settled on using dm-cache for two reasons: it's LVM and ceph-volume mentions it's supported and "transparent": > > > For ceph-volume, the use of dm-cache is transparent, there is no difference for the tool, and it treats dm-cache like a plain logical volume. > > However, what does this really mean? It means you do it disk-for-disk instead of a fast pool in front of a slow pool. In the end, the same results should be there, faster accesses for writes and recently accessed data. > Is it that I can just slap an lvm cache on the LVs created by ceph-volume and call it a day? > Is this the canonical replacement to Ceph's native cache tiers? I guess it is one of the default alternatives now that tiering is off the map. We do a lot of "put WAL + DB on nvme partition, data on spin-disks" with something like 4 or 6 nvme partitions per OSD disk, so this mostly improves database access parts and less of the bulk data transfers, but we think that is ok. > Is the tuning of the cache settings somehow managed by Ceph or is it fully on the user? And, related, can the caching be declared in the orch OSD spec? If you stack devices in LVM, then I guess that is where any tuning would have to happen, though I don't think there is a large need to tune those settings a lot. The defaults should probably work fine for most people. > As for the context: I have a cluster with *very* bad SSDs (consumer DRAM-less BX500) and I'm trying to get it somewhat OK until the market for flash improves. As the problems are due to GC pauses from the drive caused by the sustained IO, I'm hoping that putting a cache tier (with smaller faster enterprise SSDs) in front of the drives will change the IO behavior enough for the drives to perform their GC without impacting the cluster. I already have the good smaller drives that have WAL+DB for the slow drives, but they have enough free space to cache the block. I don't have any idea on which method will be best in the end, but I think dm-cache is stable enough for people to recommend it. -- May the most significant bit of your life be positive.

2 1

Questions about dm-cache
by Alexis Bauvin 09 Apr '26

09 Apr '26

Hi, Since a few versions, Ceph has deprecated caching tiers with no alternative mentioned. Thus I've been looking and somewhat settled on using dm-cache for two reasons: it's LVM and ceph-volume mentions it's supported and "transparent": > For ceph-volume, the use of dm-cache is transparent, there is no difference for the tool, and it treats dm-cache like a plain logical volume. However, what does this really mean? Is it that I can just slap an lvm cache on the LVs created by ceph-volume and call it a day? Is this the canonical replacement to Ceph's native cache tiers? Is the tuning of the cache settings somehow managed by Ceph or is it fully on the user? And, related, can the caching be declared in the orch OSD spec? As for the context: I have a cluster with *very* bad SSDs (consumer DRAM-less BX500) and I'm trying to get it somewhat OK until the market for flash improves. As the problems are due to GC pauses from the drive caused by the sustained IO, I'm hoping that putting a cache tier (with smaller faster enterprise SSDs) in front of the drives will change the IO behavior enough for the drives to perform their GC without impacting the cluster. I already have the good smaller drives that have WAL+DB for the slow drives, but they have enough free space to cache the block. Thanks!

1 0

Re: ACME for the mgmt-gateway
by Redouane Kachach 09 Apr '26

09 Apr '26

I see, the filter-by only exists on main at the moment. Another alternative is to use "ceph orch certmgr cert check" which should report expiring certificataes as well. Just FYI mgmt-gateway is actually an Nginx-based reverse proxy; that's why I didn't recommend putting an additional reverse proxy in front of it. Anyway good to know the Caddy-based workaround is working in your case. Best, Redouane. On Thu, Apr 9, 2026 at 10:33 AM Matthias Riße via ceph-users < ceph-users(a)ceph.io> wrote: > Hey Redouane, > > thank you for the detailed response! > > Am 06.04.26 um 18:10 schrieb Redouane Kachach via ceph-users: > > Hi Matthias, > > > > Currently, the mgmt-gateway (and the rest of cephadm services) don't > > support ACME integration. Users must manually provide/update the > > external certificates. So as I see it it's not a "lack of documentation", > > in this case ACME is not a supported option. > > Even if there is nothing explicitly built-in there could be a documented > "best-practice" approach outlining how to integrate with external tools > like certbot, lego, or caddy. But I don't know if this is even common > enough of a use-case, maybe most people are contend with using the > self-signed certificates since the dashboard is internal-/admin-only > anyway, and I am the outlier :) > > > > > That said, I think this can be achieved using the current implementation > > without any extra code changes in cephadm. The key insight is that > > cephadm's certmgr already > > has all the necessary components: it monitors cert expiry, and when it > > detects a certificate has changed in the store, it automatically > triggers a > > reconfig of the mgmt-gateway. > > So the only missing piece for externally-managed certs is the renewal > > trigger itself. For that, "ceph orch certmgr cert ls --filter-by > > status=expiring" can be used to detect expiring > > certificates and kick off the renewal process. > > This command doesn't seem to work, the --filter-by flag is unknown: > > # ceph orch certmgr cert ls --filter-by status=expiring > Invalid command: Unexpected argument '--filter-by' > orch certmgr cert ls [--show-details] [--format > {plain|json|json-pretty|yaml|xml-pretty|xml}] : > Error EINVAL: invalid command > # ceph --version > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle > (stable) > > But certbot, lego, et al. on a cron timer should check themselves if the > certs are close to expiry and only update them when they are about to > expire, so this shouldn't be needed anyway. > > > > > The procedure can be: > > > > 1. Set certificate_source: reference in the mgmt-gateway spec. This > > tells certmgr the cert is externally owned and makes it accept > > pushes via "ceph > > orch certmgr cert set." > > 2. Run certbot (or your tool of choice) with DNS-01 (or HTTP) > challenge > > which avoids any port-80 conflict with mgmt-gateway already owning > the > > HTTPS endpoint. > > 3. A simple daily systemd timer runs a script that: > > - Queries ceph orch certmgr cert ls --filter-by "status=expiring" > to > > check if renewal is needed > > - If yes, calls cerbot renew procedure > > - Pushes the new cert+key into certmgr via ceph "orch certmgr cert > > set / key set" > > - cephadm's serve loop detects the cert change on its next cycle > and > > reconfigures mgmt-gateway automatically (no manual restart > needed). > > 4. For security, the script should use a dedicated client keyring > with > > caps restricted to only the two certmgr write commands. > > > > For now I've gone with the different approach of putting a caddy reverse > proxy in front of the mgmt-gateway, but only because I had a config for > that using our EAB setup lying around anyway. To do so I've reconfigured > mgmt-gateway to only listen locally on a different port: > > placement: > label: mgmt > networks: > - 127.0.0.1 > spec: > port: 5000 > > and created a custom container service running caddy with the same > placement rule: > > placement: > label: mgmt > spec: > args: > - --net=host > bind_mounts: > - - type=bind > - source=conf > - destination=/etc/caddy > - ro=true > dirs: > - conf > files: > conf/Caddyfile: > - '{' > - ' acme_ca <redacted>' > - ' acme_eab {' > - ' key_id <redacted>' > - ' mac_key <redacted>' > - ' }' > - '}' > - <redacted> { > - ' reverse_proxy 127.0.0.1:5000' > - '}' > image: docker.io/library/caddy:2 > ports: > - 80 > - 443 > > Seems to be working fine for now, but I still might replace this with > setting the mgmt-gateway certs directly. > > Best > Matthias > > > > > Best, > > Redouane. > > > > > > On Mon, Mar 30, 2026 at 2:27 PM Matthias Riße via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Hey all! > >> > >> I've just managed to get the mgmt-gateway working on our Ceph Cluster > >> with cephadm, which is quite cool to have as a single HA entrypoint. I > >> am surprised about the lack of documentation to get ACME-signed > >> certificates into / in front of the mgmt-gateway. > >> > >> Is anyone doing this already? If yes, what's your setup? I am wondering > >> if it is easier to put certbot, lego, or similar into a "custom > >> container service" and update the mgmt-gateway service with renewed > >> certs from there, or run an additional caddy instance (also as a custom > >> container service?) as a reverse proxy in front of the mgmt-gateway and > >> let it handle ACME. > >> > >> Kind regards > >> Matthias > >> > >> -- > >> > >> > --------------------------------------------------------------------------------------------- > >> > >> > --------------------------------------------------------------------------------------------- > >> Forschungszentrum Jülich GmbH > >> 52425 Jülich > >> Sitz der Gesellschaft: Jülich > >> Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > >> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller > >> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > >> Dr. Stephanie Bauer (stellvertretende Vorsitzende), > >> Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers > >> > >> > --------------------------------------------------------------------------------------------- > >> > >> > --------------------------------------------------------------------------------------------- > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > > --------------------------------------------------------------------------------------------- > > --------------------------------------------------------------------------------------------- > Forschungszentrum Jülich GmbH > 52425 Jülich > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Stefan Müller > Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > Dr. Stephanie Bauer (stellvertretende Vorsitzende), > Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers > > --------------------------------------------------------------------------------------------- > > --------------------------------------------------------------------------------------------- > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Has anyone been able to use NFS in tentacle or it is really broken?
by Robert Sander 09 Apr '26

09 Apr '26

Hi, I just tried to deploy an NFS service on a freshly installed 20.2.1 cluster and it fails. ceph nfs cluster create nfs01 "label:nfs01" There is exactly one host with the label nfs01. There are some entries in the log showing errors but the last two are - Cannot acquire credentials for principal nfs - Cannot register NFS V3 on TCP /var/lib/ceph/2ac9b090-3363-11f1-bd6a-bc2411d682b7/nfs.nfs01.0.0.rgw01.hpfeuz/etc/ganesha/ganesha.conf contained "Protocols 3, 4;" in the NFS_CORE_PARAM section. After removing the 3 the container starts. NFSv3 was added in commit 1e69ebc6ceaf in February 2024. It was not contained in Ceph squid. Apr 09 14:23:18 HOST systemd[1]: Starting ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service - Ceph nfs.nfs01.0.0.HOST.hpfeuz for 2ac9b090-3363-11f1-bd6a-bc2411d682b7... Apr 09 14:23:18 HOST podman[31327]: 2026-04-09 14:23:18.942654365 +0200 CEST m=+0.048118777 container create c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, CEPH_REF=tentacle, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260331, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, io.buildah.version=1.43.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:19.000318083 +0200 CEST m=+0.105782538 container init c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, io.buildah.version=1.43.0, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_REF=tentacle, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260331) Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:19.004217805 +0200 CEST m=+0.109682235 container start c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, CEPH_REF=tentacle, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, org.label-schema.vendor=CentOS, ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260331, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, io.buildah.version=1.43.0, org.label-schema.license=GPLv2) Apr 09 14:23:19 HOST bash[31327]: c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:18.922220554 +0200 CEST m=+0.027684991 image pull fb63cba66eea5516fbd9a1ec035c265abdbc1a1a8fc2dc0b51057557e49ca28c quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337 Apr 09 14:23:19 HOST systemd[1]: Started ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service - Ceph nfs.nfs01.0.0.HOST.hpfeuz for 2ac9b090-3363-11f1-bd6a-bc2411d682b7. Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576. Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] rados_kv_traverse :CLIENT ID :EVENT :Failed to lst kv ret=-2 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] rados_cluster_read_clids :CLIENT ID :EVENT :Failed to traverse recovery db: -2 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] rados_cluster_end_grace :CLIENT ID :EVENT :Failed to remove rec-0000000000000001:0: -2 Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:9): Unknown parameter (allow_set_io_flusher_fail) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP Apr 09 14:23:22 HOST podman[31583]: 2026-04-09 14:23:22.269193428 +0200 CEST m=+0.027823817 container died c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, io.buildah.version=1.43.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=tentacle, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, org.label-schema.build-date=20260331, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>) Apr 09 14:23:22 HOST podman[31583]: 2026-04-09 14:23:22.325861901 +0200 CEST m=+0.084492274 container remove c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, ceph=True, org.label-schema.vendor=CentOS, CEPH_REF=tentacle, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, org.label-schema.build-date=20260331, io.buildah.version=1.43.0, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 09 14:23:22 HOST systemd[1]: ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Apr 09 14:23:22 HOST systemd[1]: ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7(a)nfs.nfs01.0.0.HOST.hpfeuz.service: Failed with result 'exit-code'. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

Re: v20.2.2 merge window is now open!
by Janne Johansson 09 Apr '26

09 Apr '26

> > > It woulb be nice to include this: > https://tracker.ceph.com/issues/71310 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> > Sorry for hijacking this thread, but could someone close to the list admins rework the mail list manager settings so that the above link does NOT point to https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F… and instead actually takes you to tracker.ceph.com directly? It's kind of a dark pattern to say "this link goes here" and then actually send people somewhere else. In my case, the ad-blockers and whatnot are preventing me from doing the double-jump, but it becomes tedious to have to cut-n-paste the text-of-the-link instead of just clicking the link directly. You have our emails already, don't have to grab our browser cookies or place trackers on us for these kinds of technical links. If anyone is REALLY interested in if people will go to the issue 71310, I'm sure the tracker site has access logs you can look at instead. -- May the most significant bit of your life be positive.

1 0

Re: v20.2.2 merge window is now open!
by Matthew H 09 Apr '26

09 Apr '26

Hi, I don't see a PR for this, nor do I see the tracker having a backport label to tentacle. It would be good to have you (or someone else) propose a PR, and it can be added to the tracker for review. ________________________________ From: Alfonso Escribano Merino via ceph-users <ceph-users(a)ceph.io> Sent: Thursday, April 9, 2026 1:07 AM To: Patrick Donnelly <pdonnell(a)redhat.com>; dev <dev(a)ceph.io> Cc: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: v20.2.2 merge window is now open! Hello, It woulb be nice to include this: https://tracker.ceph.com/issues/71310<https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> I have two classes of nvme disks, ones (TLC) with 7TB and others (QLC) with 30TB for different pools and it would be great to have the patch that is in #71310 merged into master to have it on 20.2.2 and also backport to older releases. Thanks, Alfonso Escribano Patrick Donnelly via ceph-users wrote: Hello everyone, v20.2.1 has finally been released! The tentacle merge window is open again. Please follow this tracker ticket for updates and processes for getting backporrts merged: https://tracker.ceph.com/issues/75878<https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> For discussion, please visit #ceph-upstream-releases-tentacle on Slack, respond to this mail, or come to the "Ceph Release Engineering" on Thursdays (see Ceph community calendar). Thanks, -- Alfonso Escribano alfonso.escribano(a)meteologica.com<mailto:alfonso.escribano@meteologica.com> [http://www.meteologica.com/assets/images/logo.png]<https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fwww.meteologica.com…> c/ Costa Brava 10 - 28034 Madrid Office: +34 91 456 10 01 This email and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not an addressee, any unauthorized direct or indirect dissemination, distribution, publication, reading, or copying of this message and any attachments is strictly prohibited. If you have received this email in error please notify the sender immediately, and delete this email from your system. Email communication does not guarantee the confidentiality, integrity, or proper receipt of the messages sent. Meteológica, S.A. accepts no liability for such circumstances.

1 0

Re: ACME for the mgmt-gateway
by Matthias Riße 09 Apr '26

09 Apr '26

Hey Redouane, thank you for the detailed response! Am 06.04.26 um 18:10 schrieb Redouane Kachach via ceph-users: > Hi Matthias, > > Currently, the mgmt-gateway (and the rest of cephadm services) don't > support ACME integration. Users must manually provide/update the > external certificates. So as I see it it's not a "lack of documentation", > in this case ACME is not a supported option. Even if there is nothing explicitly built-in there could be a documented "best-practice" approach outlining how to integrate with external tools like certbot, lego, or caddy. But I don't know if this is even common enough of a use-case, maybe most people are contend with using the self-signed certificates since the dashboard is internal-/admin-only anyway, and I am the outlier :) > > That said, I think this can be achieved using the current implementation > without any extra code changes in cephadm. The key insight is that > cephadm's certmgr already > has all the necessary components: it monitors cert expiry, and when it > detects a certificate has changed in the store, it automatically triggers a > reconfig of the mgmt-gateway. > So the only missing piece for externally-managed certs is the renewal > trigger itself. For that, "ceph orch certmgr cert ls --filter-by > status=expiring" can be used to detect expiring > certificates and kick off the renewal process. This command doesn't seem to work, the --filter-by flag is unknown: # ceph orch certmgr cert ls --filter-by status=expiring Invalid command: Unexpected argument '--filter-by' orch certmgr cert ls [--show-details] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] : Error EINVAL: invalid command # ceph --version ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable) But certbot, lego, et al. on a cron timer should check themselves if the certs are close to expiry and only update them when they are about to expire, so this shouldn't be needed anyway. > > The procedure can be: > > 1. Set certificate_source: reference in the mgmt-gateway spec. This > tells certmgr the cert is externally owned and makes it accept > pushes via "ceph > orch certmgr cert set." > 2. Run certbot (or your tool of choice) with DNS-01 (or HTTP) challenge > which avoids any port-80 conflict with mgmt-gateway already owning the > HTTPS endpoint. > 3. A simple daily systemd timer runs a script that: > - Queries ceph orch certmgr cert ls --filter-by "status=expiring" to > check if renewal is needed > - If yes, calls cerbot renew procedure > - Pushes the new cert+key into certmgr via ceph "orch certmgr cert > set / key set" > - cephadm's serve loop detects the cert change on its next cycle and > reconfigures mgmt-gateway automatically (no manual restart needed). > 4. For security, the script should use a dedicated client keyring with > caps restricted to only the two certmgr write commands. > For now I've gone with the different approach of putting a caddy reverse proxy in front of the mgmt-gateway, but only because I had a config for that using our EAB setup lying around anyway. To do so I've reconfigured mgmt-gateway to only listen locally on a different port: placement: label: mgmt networks: - 127.0.0.1 spec: port: 5000 and created a custom container service running caddy with the same placement rule: placement: label: mgmt spec: args: - --net=host bind_mounts: - - type=bind - source=conf - destination=/etc/caddy - ro=true dirs: - conf files: conf/Caddyfile: - '{' - ' acme_ca <redacted>' - ' acme_eab {' - ' key_id <redacted>' - ' mac_key <redacted>' - ' }' - '}' - <redacted> { - ' reverse_proxy 127.0.0.1:5000' - '}' image: docker.io/library/caddy:2 ports: - 80 - 443 Seems to be working fine for now, but I still might replace this with setting the mgmt-gateway certs directly. Best Matthias > > Best, > Redouane. > > > On Mon, Mar 30, 2026 at 2:27 PM Matthias Riße via ceph-users < > ceph-users(a)ceph.io> wrote: > >> Hey all! >> >> I've just managed to get the mgmt-gateway working on our Ceph Cluster >> with cephadm, which is quite cool to have as a single HA entrypoint. I >> am surprised about the lack of documentation to get ACME-signed >> certificates into / in front of the mgmt-gateway. >> >> Is anyone doing this already? If yes, what's your setup? I am wondering >> if it is easier to put certbot, lego, or similar into a "custom >> container service" and update the mgmt-gateway service with renewed >> certs from there, or run an additional caddy instance (also as a custom >> container service?) as a reverse proxy in front of the mgmt-gateway and >> let it handle ACME. >> >> Kind regards >> Matthias >> >> -- >> >> --------------------------------------------------------------------------------------------- >> >> --------------------------------------------------------------------------------------------- >> Forschungszentrum Jülich GmbH >> 52425 Jülich >> Sitz der Gesellschaft: Jülich >> Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 >> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >> Dr. Stephanie Bauer (stellvertretende Vorsitzende), >> Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers >> >> --------------------------------------------------------------------------------------------- >> >> --------------------------------------------------------------------------------------------- >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

1 0

Re: MGR stuck in starting state
by Iztok Gregori 09 Apr '26

09 Apr '26

Hi! A little update. We were able to make the ceph-mgr back working, but we are still don't know what was wrong. To made it work again we restarted (one at the time) the MONs, after restarting the 2 MON (out of 3) the MGR started working again. We are still investigation why we are unable to add new MONs to the cluster (they are stucked in probing/electing phase), but it is a topic for another thread. Cheers Iztok On 08/04/26 18:14, Iztok Gregori via ceph-users wrote: > Hi to all! > > First a little of background. We are in the process of a massive upgrade > from Octopus to Quincy (during which there is also the upgrade > (reinstall) from CentOS 7/Rocky 8 to Rocky 9). > > During the first phase we removed from the cluster 2 monitor/managers > and reinstalled them with Rocky 9 and Ceph 17.2.9). The 3 remaining > monitors/manager were: > - 2 x Rocky 8 with 15.2.17 > - 1 x CentOS 7 with 15.2.16 > > Then we upgraded Ceph from 15.2.17 to 17.9.6 on the 2 nodes with Rocky 8 > and restarted the services. So to summarize we have: > - 2 x MON Rocky 8 with 17.2.6 > - 1 x MON CentOS 7 with 15.2.16 > > and > - 1 x MGR CentOS 7 with 15.2.16 > > The next step was add the new (reinstalled) nodes as MON in the cluster > we encounter the first problem. Once started the new ceph-mon daemon > entered the cluster but all the commands ("ceph -s" for example) > remained stuck. If we stopped the new ceph-mon daemon the commands > restart to work (the status was in warning because the new ceph-mon was > down). To regain the quorum we deleted the new ceph-mon. > > And here I think we made a mistake, we started the ceph-mgr daemons on > the 2x Rocky 8 nodes. So we had: > - 2 x MGR Rocky 8 with 17.2.6 (standby) > - 1 x MGR CentOS 7 with 15.2.16 (active) > > We wrongly assume that the MGR was the culprit and we fail the active > MGR. From then on the status of the active MGR is in starting and all > the PGS are unknown: > > cluster: > id: 5bed1bf6-123f-4597-b8b1-d77931228548 > health: HEALTH_WARN > Reduced data availability: 13825 pgs inactive > > services: > mon: 3 daemons, quorum n-3-c7,n-2-r8,n-1-r8 (age 91m) > mgr: n-1-r8 (active, starting, since 13m), standbys: n-2-r8, n-3-c7 > osd: 852 osds: 849 up (since 8d), 849 in (since 7w) > > data: > pools: 11 pools, 13825 pgs > objects: 0 objects, 0 B > usage: 0 B used, 0 B / 0 B avail > pgs: 100.000% pgs unknown > 13825 unknown > > From the logs we see > >> 2026-04-08T16:59:26.244+0200 7fe0596a1700 0 ms_deliver_dispatch: >> unhandled message 0x5583db0d0c60 mon_map magic: 0 v1 from mon.2 >> v2:172.19.240.35:3300/0 >> 2026-04-08T16:59:26.807+0200 7fe0596a1700 1 mgr handle_mgr_map >> Activating! >> 2026-04-08T16:59:26.808+0200 7fe0596a1700 1 mgr handle_mgr_map I am >> now activating >> 2026-04-08T16:59:34.923+0200 7fdfd7e70700 1 mgr.server send_report >> Not sending PG status to monitor yet, waiting for OSDs >> 2026-04-08T16:59:36.924+0200 7fdfd7e70700 1 mgr.server send_report >> Not sending PG status to monitor yet, waiting for OSDs >> 2026-04-08T16:59:38.924+0200 7fdfd7e70700 1 mgr.server send_report >> Not sending PG status to monitor yet, waiting for OSDs >> 2026-04-08T16:59:40.924+0200 7fdfd7e70700 1 mgr.server send_report >> Not sending PG status to monitor yet, waiting for OSDs >> 2026-04-08T16:59:42.925+0200 7fdfd7e70700 1 mgr.server send_report >> Not sending PG status to monitor yet, waiting for OSDs >> 2026-04-08T16:59:44.925+0200 7fdfd7e70700 1 mgr.server send_report >> Not sending PG status to monitor yet, waiting for OSDs >> 2026-04-08T16:59:46.925+0200 7fdfd7e70700 1 mgr.server send_report >> Not sending PG status to monitor yet, waiting for OSDs >> 2026-04-08T16:59:48.926+0200 7fdfd7e70700 1 mgr.server send_report >> Not sending PG status to monitor yet, waiting for OSDs >> 2026-04-08T16:59:50.925+0200 7fdfd7e70700 1 mgr.server send_report >> Not sending PG status to monitor yet, waiting for OSDs >> 2026-04-08T16:59:52.926+0200 7fdfd7e70700 1 mgr.server send_report >> Giving up on OSDs that haven't reported yet, sending potentially >> incomplete PG state to mon > > The mgr dump looks like this: > >> { >> "epoch": 373, >> "active_gid": 16955846717, >> "active_name": "n-1-r8", >> "active_addrs": { >> "addrvec": [] >> }, >> "active_addr": "(unrecognized address family 0)/0", >> "active_change": "2026-04-08T16:59:26.282185+0200", >> "active_mgr_features": 4540138320759226367, >> "available": false, >> "standbys": [ >> { >> "gid": 16955784229, >> "name": "n-2-r8", >> "mgr_features": 4540138320759226367, >> "available_modules": [ > > We think that "unrecognized address family" could be the problem, > because we have a cluster/public network which is on a different > interface and could be that is not recognized/binded on mgr startup. But > we might be wrong. > > Somebody can give us some help? > > Thanks in advance > Iztok Gregori > > >

1 0

Re: v20.2.2 merge window is now open!
by Alfonso Escribano Merino 09 Apr '26

09 Apr '26

1 0

Re: Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Gregory Farnum 08 Apr '26

08 Apr '26

Can you please talk more about how you’re running the cluster and what data is already on it? There’s nothing about growing a stretch cluster that naively would lead to this behavior, but it does have a lot more constraints than a regular Ceph cluster does about which OSDs it can use to go active with (because it needs a current copy in both sites, and it’s trying to keep 4 copies up to date rather than 3). -Greg On Mon, Apr 6, 2026 at 7:26 AM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > Are you using upmap-remapped to mitigate the thundering herd? And, if > you're using the PG autoscaler, to mitigate the thundering herd of backfill > that it adds? > > > On Apr 6, 2026, at 7:33 AM, Michal Strnad via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > Hi. > > > > We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 > OSDs and another 1000 OSDs to be added. We are concerned with issues > related to a large number of peering connections and remappings when a node > with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of > roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we > see reflected in the ceph status output before the cluster finally > stabilizes. > > > > In a classic (non‑stretch) Ceph cluster the convergence is noticeably > quicker (inactive PGs appear only for a very short interval). In our > stretch deployment, however, the window when PGs are inactive is > considerably longer, consistent with our observations. The network latency > between the two sites is consistently below 10 ms with no packet loss. > > > > We are currently trying to determine whether this impacts client‑side > read/write simulations via S3, but this has not yet been verified. Our next > step is to test with RBD, which should be more resilient and demonstrable. > Naturally, we would like to control the process in a way that minimizes the > impact on clients. Could you please confirm that the periods of PG > inactivity are short enough to prevent client visible glitches, or even > better, could you recommend an optimal protocol for adding new OSDs to a > cluster of this size to ensure a smooth convergence? Do you have any tested > or recommended timeout adjustments for the stretch configuration? > > > > Thank you for any feedback > > -- > > Michal > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Question Regarding Ceph Monitor Election Strategy Default Configuration
by Gregory Farnum 08 Apr '26

08 Apr '26

Mode 1 (classic) works well most of the time and is the default simply because it was first and has the most history. In mode 3 (connectivity) each monitor rates every other based on response rates to a periodic ping, and they use those scores to elect the monitor with the highest total score. Since this one is checking networking consistency and has a slight bias towards not changing the current leader, it makes sense that it would do better under a network loss scenario. Maybe someday we will make it the default, but there’s been no real reason to make the change so far. They ought to have no other differences, nor impact the operations of the rest of the Ceph cluster at all. -Greg On Mon, Apr 6, 2026 at 12:35 AM Christian Mauricio Narvaez Acaro via ceph-users <ceph-users(a)ceph.io> wrote: > Hi, > We are currently setting up and executing a testbench to analyze and > validate the resilience and robustness of the SEPATH hypervisor running > across the three nodes of a cluster (SEAPATH - LF Energy Wiki< > https://lf-energy.atlassian.net/wiki/spaces/SEAP/overview>). > As part of these tests, we evaluated the behavior of the Ceph service in a > simulated failure scenario. Using tc and netem, we introduced approximately > 40% random packet loss on the cluster network of one node, which resulted > in quorum instability. > > The main observations were as follows: > > * Continuous increase in election_epoch, indicating frequent monitor > re-elections. > * Intermittent quorum formation, with transitions between (1,2), > (1,3), (2,3), and (1,2,3). > * Repeated changes in quorum_leader_name, alternating mainly between > node 1 and node 2. > * A tendency for hypervisor1 (rank 0) to regain leadership whenever it > restored connectivity with a majority of nodes. > > > At the time of the test, we found that election_strategy was configured > with value 1. Based on our understanding, this behavior is consistent with > the classic election mechanism, where all monitors are treated equally and > the monitor with the lowest rank is typically selected as leader. > > To further analyze this behavior, we compared the two monitor election > strategies under the same induced failure condition. In both cases, we > captured TCP traffic on port 3300 over a 10-minute period using tshark. > With the classic strategy, repeated tests showed between 68 and 74 leader > changes, continuous quorum leave-and-rejoin events for the affected node, > and approximately double the traffic compared to the lossless scenario > (636Kbits/s versus 340Kbits/s). By contrast, with the connectivity-based > strategy, repeated tests showed only 1 to 4 leader changes, leadership > limited to node 2 and node 3, no rejoin of the affected node to the quorum, > and only an approximately 10% increase in TCP traffic compared to the > lossless case (437 Kbits/s versus 400 Kbits/s) > From a traffic perspective, this approach consumes less bandwidth and > significantly improves system stability during cluster network failures, > since node 1 is no longer repeatedly re-elected as leader while > experiencing network issues. > Based on the measurements we have collected and our comparison of the > election algorithms, strategy 3 appears to be the most suitable option for > our use case. We would therefore appreciate understanding whether there is > any additional consideration or technical reason for preferring strategy 1 > over strategy 3, which may explain why it is the default configuration. > Thanks in advance. > > Best regards, > > Christian Narváez A. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: v20.2.2 merge window is now open!
by Stefan Kooman 08 Apr '26

08 Apr '26

On 4/6/26 20:02, Patrick Donnelly via dev wrote: > Hello everyone, > > v20.2.1 has finally been released! > > The tentacle merge window is open again. Please follow this tracker > ticket for updates and processes for getting backporrts merged: > > https://tracker.ceph.com/issues/75878 I would like to have this in: https://tracker.ceph.com/issues/75267 I've successfully tested this on a patched 20.2.0. Thanks, Gr. Stefan

1 0

MGR stuck in starting state
by Iztok Gregori 08 Apr '26

08 Apr '26

Hi to all! First a little of background. We are in the process of a massive upgrade from Octopus to Quincy (during which there is also the upgrade (reinstall) from CentOS 7/Rocky 8 to Rocky 9). During the first phase we removed from the cluster 2 monitor/managers and reinstalled them with Rocky 9 and Ceph 17.2.9). The 3 remaining monitors/manager were: - 2 x Rocky 8 with 15.2.17 - 1 x CentOS 7 with 15.2.16 Then we upgraded Ceph from 15.2.17 to 17.9.6 on the 2 nodes with Rocky 8 and restarted the services. So to summarize we have: - 2 x MON Rocky 8 with 17.2.6 - 1 x MON CentOS 7 with 15.2.16 and - 1 x MGR CentOS 7 with 15.2.16 The next step was add the new (reinstalled) nodes as MON in the cluster we encounter the first problem. Once started the new ceph-mon daemon entered the cluster but all the commands ("ceph -s" for example) remained stuck. If we stopped the new ceph-mon daemon the commands restart to work (the status was in warning because the new ceph-mon was down). To regain the quorum we deleted the new ceph-mon. And here I think we made a mistake, we started the ceph-mgr daemons on the 2x Rocky 8 nodes. So we had: - 2 x MGR Rocky 8 with 17.2.6 (standby) - 1 x MGR CentOS 7 with 15.2.16 (active) We wrongly assume that the MGR was the culprit and we fail the active MGR. From then on the status of the active MGR is in starting and all the PGS are unknown: cluster: id: 5bed1bf6-123f-4597-b8b1-d77931228548 health: HEALTH_WARN Reduced data availability: 13825 pgs inactive services: mon: 3 daemons, quorum n-3-c7,n-2-r8,n-1-r8 (age 91m) mgr: n-1-r8 (active, starting, since 13m), standbys: n-2-r8, n-3-c7 osd: 852 osds: 849 up (since 8d), 849 in (since 7w) data: pools: 11 pools, 13825 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 13825 unknown From the logs we see > 2026-04-08T16:59:26.244+0200 7fe0596a1700 0 ms_deliver_dispatch: unhandled message 0x5583db0d0c60 mon_map magic: 0 v1 from mon.2 v2:172.19.240.35:3300/0 > 2026-04-08T16:59:26.807+0200 7fe0596a1700 1 mgr handle_mgr_map Activating! > 2026-04-08T16:59:26.808+0200 7fe0596a1700 1 mgr handle_mgr_map I am now activating > 2026-04-08T16:59:34.923+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:36.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:38.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:40.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:42.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:44.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:46.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:48.926+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:50.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:52.926+0200 7fdfd7e70700 1 mgr.server send_report Giving up on OSDs that haven't reported yet, sending potentially incomplete PG state to mon The mgr dump looks like this: > { > "epoch": 373, > "active_gid": 16955846717, > "active_name": "n-1-r8", > "active_addrs": { > "addrvec": [] > }, > "active_addr": "(unrecognized address family 0)/0", > "active_change": "2026-04-08T16:59:26.282185+0200", > "active_mgr_features": 4540138320759226367, > "available": false, > "standbys": [ > { > "gid": 16955784229, > "name": "n-2-r8", > "mgr_features": 4540138320759226367, > "available_modules": [ We think that "unrecognized address family" could be the problem, because we have a cluster/public network which is on a different interface and could be that is not recognized/binded on mgr startup. But we might be wrong. Somebody can give us some help? Thanks in advance Iztok Gregori -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A.

1 0

Re: v20.2.2 merge window is now open!
by Sascha Lucas 07 Apr '26

07 Apr '26

Hi Robert, On Tue, 7 Apr 2026, Robert Sander via ceph-users wrote: > It would be nice if https://tracker.ceph.com/issues/72696 could make it into > 20.2.2. Also having older clusters with DB devices on NVME. I thought this setups are not so uncommon? But good to know, not to upgrade to v19/v20 until fixed. This BUG seems not yet listed on the clyso's Known Bugs page[1]. There should be more transparency for normal ceph users, so that not everyone ends up with broken setups on upgrade. BUGs are uncomfortable, but not knowing them is sad. Thanks, Sascha. [1] https://docs.clyso.com/docs/kb/category/known-bugs/

1 0

Re: Changing EC profile
by Rok Jaklič 07 Apr '26

07 Apr '26

On Fri, Apr 3, 2026 at 4:06 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > >> * Are you looking to change the profile to get more usable capacity? > >> * Are you looking to change the profile to get more durability / > sustain more failures? > > Not necessarily, we would rather have a responsive rgw for example > during failures, so when there is backfill/recovery going on that rgw does > not stop responding. > > The wider profile may actually make that worse. > > Since you mention RGW and only mention HDDs, does that mean that your > index pool is on HDDs? There's a good chance that was a factor. HDDs have > very limited IOPS and all of the RGW pools that aren't bucket data really > should be on SSDs. I speculate that client ops, backfill, and RGW ops were > DoSing each other. > Forgot to add, index pools are on NVMEs. Some old OSD nodes have 2 NVMEe inside, newer ones still do not have, so we will have to consider adding NVMEs also. > An expedient way to add those would be something like > > > https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcTRl12SIsjlOJTId1V6Z… > > > A PCI-e AIC that accepts 2x NVMe M.2 drives, assuming that your > motherboard can do PCIe bifurcation. You could populate each with like 2x > 1.9T Micron 7450 and place the RGW pools on those. For this purpose having > more OSDs is more beneficial than having larger OSDs. You could easily see > the size of your existing pools to help gauge what you need. > Thx. > > > It happened a few times. e.g. slow/faulty disk from on which reweight > was just set > > You aren't using "ceph osd reweight" or "ceph osd reweight-by-utilization" > are you? Those are legacies and should be avoided. You want the REWEIGHT > value on each OSD to be 1.0000 and to use the manager balancer. > > ceph config set mgr mgr/balancer/upmap_max_deviation 1 > Actually I am doing `ceph osd crush reweight X ...` when using it. But not for faulty OSDs. On faulty OSDs, I set the OSD to out, replace it, activate it. I am using following procedure: 1. systemctl stop ceph-osd@{id} 2. ceph osd down osd.{id} 3. ceph osd out osd.{id} 4. pocakas rebalance 5. ceph osd destroy {id} --yes-i-really-mean-it --- 6. replace disk... 7. ceph-volume lvm prepare --osd-id {id} --data /dev/sdX 8. ceph-volume lvm list 9. ceph-volume lvm activate {id} {osd fsid} Does that mean you're only using RGW? No RBD, no CephFS? This informs the > process by which you would migrate data between pools. If you don't have > capacity concerns, maybe migration isn't worth the trouble. Only RGW for now. We did test CephFS a little bit, but RGW suffices our needs currently. Rok

1 0

Re: v20.2.2 merge window is now open!
by Robert Sander 07 Apr '26

07 Apr '26

Hi Patrick, Am 06.04.26 um 8:03 PM schrieb Patrick Donnelly via ceph-users: > For discussion, please visit #ceph-upstream-releases-tentacle on > Slack, respond to this mail, or come to the "Ceph Release Engineering" > on Thursdays (see Ceph community calendar). It would be nice if https://tracker.ceph.com/issues/72696 could make it into 20.2.2. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Janne Johansson 07 Apr '26

07 Apr '26

> We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 > OSDs and another 1000 OSDs to be added. We are concerned with issues > related to a large number of peering connections and remappings when a > node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of > roughly 2000 PGs and can leave tens of minutes of PGs inactive > Could you please confirm that the > periods of PG inactivity are short enough to prevent client visible > glitches, or even better, could you recommend an optimal protocol for > adding new OSDs to a cluster of this size to ensure a smooth > convergence? Do you have any tested or recommended timeout adjustments > for the stretch configuration? You could ask the new OSDs to join with initial size of 0.01 (or 0.0001 but that has certain issues too) so that few/no PGs decides to move over, then increase size later on slowly. We used to set initial OSD crush weight to 0.0001 which works, but clients that check "max free space" think the cluster is almost full, so now we do 0.01 on existing clusters, which for some reason makes free-testing-clients not think the cluster is about to be full any second. Then stop the auto-balancer, have norebalance set, bump sizes one OSD at a time, sleep some 20-30 secs (in our case), bump next OSD and so on, and lastly run pgremapper to cancel backfills using upmap, then unset norebalance and start the balancer so it can un-upmap each such PG in an orderly fashion. -- May the most significant bit of your life be positive.

1 0

v20.2.2 merge window is now open!
by Patrick Donnelly 06 Apr '26

06 Apr '26

Hello everyone, v20.2.1 has finally been released! The tentacle merge window is open again. Please follow this tracker ticket for updates and processes for getting backporrts merged: https://tracker.ceph.com/issues/75878 For discussion, please visit #ceph-upstream-releases-tentacle on Slack, respond to this mail, or come to the "Ceph Release Engineering" on Thursdays (see Ceph community calendar). Thanks, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

v20.2.1 Tentacle released
by Yuri Weinstein 06 Apr '26

06 Apr '26

This is the first minor release in the Tentacle series. We recommend that all users update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-1-tentacle-released/ Notable Changes --------------- OSD / BlueStore --------------- * EC Recovery: Fixed a length calculation bug in erase_after_ro_offset() that caused empty shards to retain data, leading to shard_size >= tobj_size assertion failures when recovering small objects in EC pools. * BlueFS Volume Selector: Updated the BlueFS volume selector to properly account for file size changes when recovering the WAL in envelope mode. * BlueFS: Fixed a bug where stat() missed the actual file size update after indexing WAL envelope files. Monitor (mon) ------------- * Fast EC Restrictions: Denied the ability to enable EC optimizations ("fast EC") for non-4K-aligned chunk sizes. Unaligned chunk sizes handled by fast EC perform poorly and suffer from bugs, so attempts to force this configuration are now rejected. * Peering: Ensured ceph pg repeer proposes a correctly sized pg temp, as optimized EC cannot cope with mismatched sizes. * NVMeoF Gateway: Added a new nvme-gw listeners command to display all existing listeners (including auto-listeners) inside a pool/group. * NVMeoF Failover: Overhauled the NVMeoF Gateway fast-failover logic. Beacon timeouts are now evaluated within prepare_beacon to support shorter intervals, and the mechanism for detecting monitor slowness was improved. librbd & rbd-mirror ------------------- * RBD: Introduced a new ``RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT`` policy for ``rbd_lock_acquire()``. This is a low-level interface intended to allow a peer to grab exclusive lock manually for short periods of time with other peers pausing their activity and waiting for the lock to be released rather than instantly aborting I/O and returning an error. It's possible to switch from ``RBD_LOCK_MODE_EXCLUSIVE`` to ``RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT`` policy and vice versa even if the lock is already held. Ceph Object Gateway (RGW) ------------------------- * Multi-Part Operations: Fixed conditional validation handling in MultiWrite, Delete, and MultiDelete workflows. mgr/dashboard ------------- * UI Navigation: Redesigned the main landing page; the "Dashboard" navigation item was renamed to "Overview" and uses a new carbonized productive card layout. * NVMeoF Management: Added the nvmeof get_subsystems CLI command, fixed JSON output indentation for NVMeoF CLI commands, and reverted the server_addr API parameter back to traddr for consistency. * Hosts View: Fixed a bug causing the IP addresses of hosts to be hidden on the Hosts page due to an issue with fact merging. * Forms & Modals: Standardized forms onto the Carbon Design System, including the pools form, service form, multi-site realm token export modal, delete zone modal, and password change forms. * Form Validation: Generalized form error handling and validations using a new cdValidate directive. mgr/cephadm ----------- * Monitoring Stack: Bumped the default container image versions for the monitoring stack: Prometheus to v3.6.0, Node-exporter to v1.9.1, Alertmanager to v0.28.1, and Grafana to v12.2.0. Security Changes ---------------- * Monitoring Stack Images: Updated Prometheus, Alertmanager, and Grafana container image versions, picking up upstream security and stability fixes. Configuration Changes --------------------- * ``bluefs_check_volume_selector_on_mount``: The previous bluefs_check_volume_selector_on_umount debug setting was renamed and repurposed. It now checks for volume selector inconsistencies on both mount and unmount phases. * ``mon_nvmeofgw_beacon_grace``: The default grace period before marking a gateway as failed has been reduced from 10 seconds to 7 seconds for faster failover. * ``nvmeof_mon_client_tick_period``: The default beacon tick interval has been lowered from 2 seconds to 1 second. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-1-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.2.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 6a49aff47758778a5f5951e731d437c317f72fb2

1 0

Re: ACME for the mgmt-gateway
by Redouane Kachach 06 Apr '26

06 Apr '26

Hi Matthias, Currently, the mgmt-gateway (and the rest of cephadm services) don't support ACME integration. Users must manually provide/update the external certificates. So as I see it it's not a "lack of documentation", in this case ACME is not a supported option. That said, I think this can be achieved using the current implementation without any extra code changes in cephadm. The key insight is that cephadm's certmgr already has all the necessary components: it monitors cert expiry, and when it detects a certificate has changed in the store, it automatically triggers a reconfig of the mgmt-gateway. So the only missing piece for externally-managed certs is the renewal trigger itself. For that, "ceph orch certmgr cert ls --filter-by status=expiring" can be used to detect expiring certificates and kick off the renewal process. The procedure can be: 1. Set certificate_source: reference in the mgmt-gateway spec. This tells certmgr the cert is externally owned and makes it accept pushes via "ceph orch certmgr cert set." 2. Run certbot (or your tool of choice) with DNS-01 (or HTTP) challenge which avoids any port-80 conflict with mgmt-gateway already owning the HTTPS endpoint. 3. A simple daily systemd timer runs a script that: - Queries ceph orch certmgr cert ls --filter-by "status=expiring" to check if renewal is needed - If yes, calls cerbot renew procedure - Pushes the new cert+key into certmgr via ceph "orch certmgr cert set / key set" - cephadm's serve loop detects the cert change on its next cycle and reconfigures mgmt-gateway automatically (no manual restart needed). 4. For security, the script should use a dedicated client keyring with caps restricted to only the two certmgr write commands. Best, Redouane. On Mon, Mar 30, 2026 at 2:27 PM Matthias Riße via ceph-users < ceph-users(a)ceph.io> wrote: > Hey all! > > I've just managed to get the mgmt-gateway working on our Ceph Cluster > with cephadm, which is quite cool to have as a single HA entrypoint. I > am surprised about the lack of documentation to get ACME-signed > certificates into / in front of the mgmt-gateway. > > Is anyone doing this already? If yes, what's your setup? I am wondering > if it is easier to put certbot, lego, or similar into a "custom > container service" and update the mgmt-gateway service with renewed > certs from there, or run an additional caddy instance (also as a custom > container service?) as a reverse proxy in front of the mgmt-gateway and > let it handle ACME. > > Kind regards > Matthias > > -- > > --------------------------------------------------------------------------------------------- > > --------------------------------------------------------------------------------------------- > Forschungszentrum Jülich GmbH > 52425 Jülich > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Stefan Müller > Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > Dr. Stephanie Bauer (stellvertretende Vorsitzende), > Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers > > --------------------------------------------------------------------------------------------- > > --------------------------------------------------------------------------------------------- > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Anthony D'Atri 06 Apr '26

06 Apr '26

Are you using upmap-remapped to mitigate the thundering herd? And, if you're using the PG autoscaler, to mitigate the thundering herd of backfill that it adds? > On Apr 6, 2026, at 7:33 AM, Michal Strnad via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi. > > We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 OSDs and another 1000 OSDs to be added. We are concerned with issues related to a large number of peering connections and remappings when a node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we see reflected in the ceph status output before the cluster finally stabilizes. > > In a classic (non‑stretch) Ceph cluster the convergence is noticeably quicker (inactive PGs appear only for a very short interval). In our stretch deployment, however, the window when PGs are inactive is considerably longer, consistent with our observations. The network latency between the two sites is consistently below 10 ms with no packet loss. > > We are currently trying to determine whether this impacts client‑side read/write simulations via S3, but this has not yet been verified. Our next step is to test with RBD, which should be more resilient and demonstrable. Naturally, we would like to control the process in a way that minimizes the impact on clients. Could you please confirm that the periods of PG inactivity are short enough to prevent client visible glitches, or even better, could you recommend an optimal protocol for adding new OSDs to a cluster of this size to ensure a smooth convergence? Do you have any tested or recommended timeout adjustments for the stretch configuration? > > Thank you for any feedback > -- > Michal > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Michal Strnad 06 Apr '26

06 Apr '26

Hi. We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 OSDs and another 1000 OSDs to be added. We are concerned with issues related to a large number of peering connections and remappings when a node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we see reflected in the ceph status output before the cluster finally stabilizes. In a classic (non‑stretch) Ceph cluster the convergence is noticeably quicker (inactive PGs appear only for a very short interval). In our stretch deployment, however, the window when PGs are inactive is considerably longer, consistent with our observations. The network latency between the two sites is consistently below 10 ms with no packet loss. We are currently trying to determine whether this impacts client‑side read/write simulations via S3, but this has not yet been verified. Our next step is to test with RBD, which should be more resilient and demonstrable. Naturally, we would like to control the process in a way that minimizes the impact on clients. Could you please confirm that the periods of PG inactivity are short enough to prevent client visible glitches, or even better, could you recommend an optimal protocol for adding new OSDs to a cluster of this size to ensure a smooth convergence? Do you have any tested or recommended timeout adjustments for the stretch configuration? Thank you for any feedback -- Michal

1 0

Question Regarding Ceph Monitor Election Strategy Default Configuration
by Christian Mauricio Narvaez Acaro 06 Apr '26

06 Apr '26

Hi, We are currently setting up and executing a testbench to analyze and validate the resilience and robustness of the SEPATH hypervisor running across the three nodes of a cluster (SEAPATH - LF Energy Wiki<https://lf-energy.atlassian.net/wiki/spaces/SEAP/overview>). As part of these tests, we evaluated the behavior of the Ceph service in a simulated failure scenario. Using tc and netem, we introduced approximately 40% random packet loss on the cluster network of one node, which resulted in quorum instability. The main observations were as follows: * Continuous increase in election_epoch, indicating frequent monitor re-elections. * Intermittent quorum formation, with transitions between (1,2), (1,3), (2,3), and (1,2,3). * Repeated changes in quorum_leader_name, alternating mainly between node 1 and node 2. * A tendency for hypervisor1 (rank 0) to regain leadership whenever it restored connectivity with a majority of nodes. At the time of the test, we found that election_strategy was configured with value 1. Based on our understanding, this behavior is consistent with the classic election mechanism, where all monitors are treated equally and the monitor with the lowest rank is typically selected as leader. To further analyze this behavior, we compared the two monitor election strategies under the same induced failure condition. In both cases, we captured TCP traffic on port 3300 over a 10-minute period using tshark. With the classic strategy, repeated tests showed between 68 and 74 leader changes, continuous quorum leave-and-rejoin events for the affected node, and approximately double the traffic compared to the lossless scenario (636Kbits/s versus 340Kbits/s). By contrast, with the connectivity-based strategy, repeated tests showed only 1 to 4 leader changes, leadership limited to node 2 and node 3, no rejoin of the affected node to the quorum, and only an approximately 10% increase in TCP traffic compared to the lossless case (437 Kbits/s versus 400 Kbits/s) From a traffic perspective, this approach consumes less bandwidth and significantly improves system stability during cluster network failures, since node 1 is no longer repeatedly re-elected as leader while experiencing network issues. Based on the measurements we have collected and our comparison of the election algorithms, strategy 3 appears to be the most suitable option for our use case. We would therefore appreciate understanding whether there is any additional consideration or technical reason for preferring strategy 1 over strategy 3, which may explain why it is the default configuration. Thanks in advance. Best regards, Christian Narváez A.

1 0

Re: Cephadm woes on EVPN loopback network interfaces, detects wrong IPs
by Gregor Radtke 05 Apr '26

05 Apr '26

Martin, in case this is still an issue (apparently it is april already...): I am assuming here you have your /32 on lo and run IS-IS/openfabric/eBGP unnumbered on your eth[0-9]. In this case, i faced a similar issue a while back and, while i was not able to identify the actual root cause, my gut feeling told me that ceph would simply look which interface the default route would point to, and then take that interface's IP address. Which might be entirely wrong in setups like these... You might want to try putting the same /32 (same for /128 v6 if applicable) on both lo and your eth[0-9]. This probably feels entirely wrong (it does to me!), but i was successful in resolving this exact issue in this way. I came to think about this solution, as i vaguely remember the proxmox folks documenting a full-mesh / routed setup where the same IP + /24 (in their example) mask would be put on multiple interfaces at the same time. This might have had the same idea in mind, but i cannot find it anymore (and, just in case, shame on me, not them..). Be careful however, if you do eBGP unnumbered, your next-hops might get signalled w/ LLA + GLA as a result. This is what happened to me with PVE 9.1.5 (FRR 10.4.1) talking towards JunOS. JunOS would then erronuously pick the GLA as next-hop, resulting in the entire route becoming invalid. My interim fix is to set the next-hop to the peer address in the import policy, and carefully avoiding 24.4R1-S3 on MX (this release lets you set the next-hop to the LLA, but then still deems the next-hop to be unusable. Have not found a matching PR yet, but it works in e.g. 24.4R2-S2 and -S3). Cheers & happy Easter! Gregor P.S. (Configuring the same IP/32 on lo and eth[0-9] also helps with systemd-networksomething to behave more nicely, because it doesn't feel obligated to wait for connectivity on the eth interfaces _based on its weird (default) understanding of connectivity_. Can be fine-tuned with systemd-networkd options...) On 13.03.26 19:43, Martin Gerhard Loschwitz via ceph-users wrote: > Folks, > > I have a setup here where a server has a single /32 IP from 10.1.0.0/16. It has a perfectly normal IP address and is reachable from within the local network thanks to EVPN. cephadm however only detects nonsense: > > { > "fe80::/64": { > "data1": [ > "fe80::425b:7fff:fec8:c956" > ], > "loopback0": [ > "fe80::1c54:fbff:fe6a:51e7" > ] > } > } > > How can I set the networks the machine has manually so that it becomes usable for MON placement? At this time, ceph-mgr does not deploy additional MONs, for instance, because it claims that the host network does not match the public network, while I assume the host network is directly read from the „cephadm list-networks“ output. > > Thank you very much in advance > Martin >

1 0

Re: tentacle 20.2.1 QE validation status
by Yuri Weinstein 03 Apr '26

03 Apr '26

We have not done the Sepia LRC upgrade to 20.2.1 yet. Do we want to do it for this release? Neha, Patrick? On Wed, Apr 1, 2026 at 3:37 PM Laura Flores via dev <dev(a)ceph.io> wrote: > The only outstanding issue (https://tracker.ceph.com/issues/75570 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…>) > is not a blocker to the release. We found that it is due to an > overaggressive thrasher in the stress-split upgrade tests. > > More details of the analysis are on the ticket. > > Thanks, > Laura > > *Laura Flores* > Software Engineer; Ceph Storage > 1 708 738 8804 Mobile > She/Her/Hers > > IBM > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Wednesday, April 1, 2026 3:41 PM > *To:* Laura Flores <lflores(a)ibm.com> > *Cc:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io>; Adam King < > adking(a)redhat.com>; Sridhar Seshasayee <Sridhar.Seshasayee(a)ibm.com> > *Subject:* [EXTERNAL] Re: tentacle 20.2.1 QE validation status > > Laura, ack Thx for the update! On Wed, Apr 1, 2026 at 9: 45 AM Laura > Flores via dev <dev@ ceph. io> wrote: Hi Yuri, The orch team is still > investigating the issue that was found in the upgrade suite (https: > //tracker. ceph. com/issues/75570). > Laura, ack > Thx for the update! > > On Wed, Apr 1, 2026 at 9:45 AM Laura Flores via dev <dev(a)ceph.io> wrote: > > Hi Yuri, > > The orch team is still investigating the issue that was found in the > upgrade suite (https://tracker.ceph.com/issues/75570 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…>). > We are leaning toward it being a testing/environmental failure, but we > would like a bit more time to make a concrete consensus. Someone will try > to update for sure by the end of today. > > Thanks, > Laura > > *Laura Flores* > Software Engineer; Ceph Storage > 1 708 738 8804 Mobile > She/Her/Hers > > IBM > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Wednesday, April 1, 2026 10:12 AM > *To:* Yuri Weinstein <yweinste(a)redhat.com> > *Cc:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io>; Adam King < > adking(a)redhat.com>; Sridhar Seshasayee <Sridhar.Seshasayee(a)ibm.com> > *Subject:* [EXTERNAL] Re: tentacle 20.2.1 QE validation status > > We got all approvals and are ready to start building. Please speak up now > if anything else is needed. On Wed, Apr 1, 2026 at 7: 54 AM Yuri Weinstein > via dev <dev@ ceph. io> wrote: In a separate email Venky replied that > failures in orch is > We got all approvals and are ready to start building. > Please speak up now if anything else is needed. > > On Wed, Apr 1, 2026 at 7:54 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > In a separate email Venky replied that failures in orch is a known issue > > "This is a known issue and is tracked here > > https://tracker.ceph.com/issues/64502 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…> > > Redo ^^" > > Adam King, I'd like to assume orch is good to go. > Please confirm. > > On Tue, Mar 31, 2026 at 4:43 PM Yuri Weinstein via dev <dev(a)ceph.io> > wrote: > > We are still waiting for orch approval. > > On Tue, Mar 31, 2026 at 12:26 PM Adam Emerson <aemerson(a)redhat.com> wrote: > > On 22/03/2026, Yuri Weinstein via dev wrote: > > We have merged several important PRs and reran all suites. > > > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/74838#note-4 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…> > > > > Seeking reviews and approval for all suites, please see runs on build 2 > > RGW Approved. > >

1 0

Re: Changing EC profile
by Rok Jaklič 03 Apr '26

03 Apr '26

On Fri, Apr 3, 2026 at 2:31 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > > > we expanded the cluster to 12 nodes with 28 osds > > 2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged > (Proxmox?) > 28 osds per node. > > > but we are still running on EC 3+2, since we started cluster with 6 > nodes. > > There are ways to fake failure domains (hosts, racks, etc) when you *know* > that a cluster will be expanding, albeit with increased risk until it does. > We use host failure domain for now. > > We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable > or will 8+3 take too much > > CPU/network capacities? > > That depends on several things. > > * Are you looking to change the profile to get more usable capacity? > * Are you looking to change the profile to get more durability / sustain > more failures? > Not necessarily, we would rather have a responsive rgw for example during failures, so when there is backfill/recovery going on that rgw does not stop responding. It happened a few times. e.g. slow/faulty disk from on which reweight was just set, was the reason that rgw stopped responding, since some queue filled up ... because ceph was unable to read from faulty pgs or smth... > * What is your hardware like? CPU, drive size/model, network links > There is a mix of: - Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 20Gb network bonds - AMD EPYC 9254 24-Core Processor with 50Gb network bonds with mostly old 6T Seagate ST6000NM0095 drives, but we are just in the process rotating them. > * What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny > files? > RGW, S3 users, backup data, nextcloud storage or just storage for placing files. Mostly tiny files, however backups with >100GB or even TB files can be found also. usage: 775 TiB used, 1.0 PiB / 1.8 PiB avail * Which Ceph release are you running? > > Reef 18.2.8 Rok

2 1

Re: Changing EC profile
by Anthony D'Atri 03 Apr '26

03 Apr '26

> we expanded the cluster to 12 nodes with 28 osds 2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged (Proxmox?) > but we are still running on EC 3+2, since we started cluster with 6 nodes. There are ways to fake failure domains (hosts, racks, etc) when you *know* that a cluster will be expanding, albeit with increased risk until it does. > We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much > CPU/network capacities? That depends on several things. * Are you looking to change the profile to get more usable capacity? * Are you looking to change the profile to get more durability / sustain more failures? * What is your hardware like? CPU, drive size/model, network links * What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny files? * Which Ceph release are you running? > Should I use isa/isa-l instead of jerasure? For new pools I would suggest that. The two are very similar, but ISA is maintained while jerasure is abandonware. > Can anyone share experience with 8+3? It goes to 11 ;) That said, props for having k+m < the number of failure domains. This gives you better RAS, and lets the size of those failure domains vary (somewhat) without stranding capacity. Having k+m=failure domains is a very common ... well I won't go so far as to call it a mistake, but let's say suboptimal strategy. I helped someone with 10 dense HDD nodes running 8+2. Adding one more (and later more) node made a big difference. If you are storing mostly very small files/ RGW objects, 8+3 will "waste" a high percentage of underlying storage. https://markhpc.github.io/2019/11/21/BlueFS-Space-Amplification.html If there are a few small files mixed in with much bigger ones, that becomes less of a concern. This space amplification drove the reduction of the default min_alloc_size from 64KB to 4KB in the O/P era. If you are running Tentacle and the pool has the ec optimization flag turned on, this space amp will be much reduced since RADOS will no longer add null padding. A wide EC profile will make scrubs and recovery slower due to the gridlock effect -- scrubs and recovery rely on reservations, and there will be more collision and thus queueing. A wide EC profile will result in slower writes. Your media and workload are factors in whether or not that's a problem for you. Whether your data is on CephFS, RGW, and/or RBD informs the actual process for moving. Please share more detail. There has been talk of being able to do this transparently with Umbrella, but it's hearsay until it's merged. > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Changing EC profile
by Rok Jaklič 03 Apr '26

03 Apr '26

Hi, we expanded the cluster to 12 nodes with 28 osds, but we are still running on EC 3+2, since we started cluster with 6 nodes. We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much CPU/network capacities? Should I use isa/isa-l instead of jerasure? Can anyone share experience with 8+3? Kind regards, Rok

1 0

Re: Q1 2026 Ceph Foundation Newsletter
by Matthias Ferdinand 03 Apr '26

03 Apr '26

On Thu, Apr 02, 2026 at 09:01:04PM +0000, Anthony Middleton via ceph-users wrote: > Read the full newsletter here: https://t.ly/Q1-2026-Newsletter for users of w3m, curl, Tor-Browser: https://ceph.io/en/news/blog/2026/Q1-community-newsletter/ (avoids js-based redirects etc.)

1 0

Q1 2026 Ceph Foundation Newsletter
by Anthony Middleton 02 Apr '26

02 Apr '26

Hey Ceph friends, Hope you're doing well! The Q1 2026 Ceph Foundation Newsletter is out, and it’s packed with updates from the first quarter. We've made great strides, from new governance charters to cost-saving initiatives, all of which shape how Ceph is built and supported. Highlights include: - Outcomes from recent CSC and Foundation Board meetings - How OVHcloud savings are benefiting our community - Our new event plans beyond Cephalocon - Opportunities to get involved If you're interested in Ceph’s future and want to contribute, this is worth a read. Read the full newsletter here: https://t.ly/Q1-2026-Newsletter Feel free to reach out with any questions or concerns. Your feedback about the community is always appreciated. Best, Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: tentacle 20.2.1 QE validation status
by Yuri Weinstein 01 Apr '26

01 Apr '26

Laura, ack Thx for the update! On Wed, Apr 1, 2026 at 9:45 AM Laura Flores via dev <dev(a)ceph.io> wrote: > Hi Yuri, > > The orch team is still investigating the issue that was found in the > upgrade suite (https://tracker.ceph.com/issues/75570 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…>). > We are leaning toward it being a testing/environmental failure, but we > would like a bit more time to make a concrete consensus. Someone will try > to update for sure by the end of today. > > Thanks, > Laura > > *Laura Flores* > Software Engineer; Ceph Storage > 1 708 738 8804 Mobile > She/Her/Hers > > IBM > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Wednesday, April 1, 2026 10:12 AM > *To:* Yuri Weinstein <yweinste(a)redhat.com> > *Cc:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io>; Adam King < > adking(a)redhat.com>; Sridhar Seshasayee <Sridhar.Seshasayee(a)ibm.com> > *Subject:* [EXTERNAL] Re: tentacle 20.2.1 QE validation status > > We got all approvals and are ready to start building. Please speak up now > if anything else is needed. On Wed, Apr 1, 2026 at 7: 54 AM Yuri Weinstein > via dev <dev@ ceph. io> wrote: In a separate email Venky replied that > failures in orch is > We got all approvals and are ready to start building. > Please speak up now if anything else is needed. > > On Wed, Apr 1, 2026 at 7:54 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > In a separate email Venky replied that failures in orch is a known issue > > "This is a known issue and is tracked here > > https://tracker.ceph.com/issues/64502 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…> > > Redo ^^" > > Adam King, I'd like to assume orch is good to go. > Please confirm. > > On Tue, Mar 31, 2026 at 4:43 PM Yuri Weinstein via dev <dev(a)ceph.io> > wrote: > > We are still waiting for orch approval. > > On Tue, Mar 31, 2026 at 12:26 PM Adam Emerson <aemerson(a)redhat.com> wrote: > > On 22/03/2026, Yuri Weinstein via dev wrote: > > We have merged several important PRs and reran all suites. > > > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/74838#note-4 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…> > > > > Seeking reviews and approval for all suites, please see runs on build 2 > > RGW Approved. > >

1 0

Re: tentacle 20.2.1 QE validation status
by Yuri Weinstein 01 Apr '26

01 Apr '26

We got all approvals and are ready to start building. Please speak up now if anything else is needed. On Wed, Apr 1, 2026 at 7:54 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > In a separate email Venky replied that failures in orch is a known issue > > "This is a known issue and is tracked here > > https://tracker.ceph.com/issues/64502 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> > > Redo ^^" > > Adam King, I'd like to assume orch is good to go. > Please confirm. > > On Tue, Mar 31, 2026 at 4:43 PM Yuri Weinstein via dev <dev(a)ceph.io> > wrote: > >> We are still waiting for orch approval. >> >> On Tue, Mar 31, 2026 at 12:26 PM Adam Emerson <aemerson(a)redhat.com> >> wrote: >> >>> On 22/03/2026, Yuri Weinstein via dev wrote: >>> > We have merged several important PRs and reran all suites. >>> > >>> > Details of this release are summarized here: >>> > https://tracker.ceph.com/issues/74838#note-4 >>> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fqcflzmq6.r.us-east-…> >>> > >>> > Seeking reviews and approval for all suites, please see runs on build 2 >>> >>> RGW Approved. >>> >>>

1 0

Re: tentacle 20.2.1 QE validation status
by Yuri Weinstein 01 Apr '26

01 Apr '26

In a separate email Venky replied that failures in orch is a known issue "This is a known issue and is tracked here https://tracker.ceph.com/issues/64502 Redo ^^" Adam King, I'd like to assume orch is good to go. Please confirm. On Tue, Mar 31, 2026 at 4:43 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > We are still waiting for orch approval. > > On Tue, Mar 31, 2026 at 12:26 PM Adam Emerson <aemerson(a)redhat.com> wrote: > >> On 22/03/2026, Yuri Weinstein via dev wrote: >> > We have merged several important PRs and reran all suites. >> > >> > Details of this release are summarized here: >> > https://tracker.ceph.com/issues/74838#note-4 >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> >> > >> > Seeking reviews and approval for all suites, please see runs on build 2 >> >> RGW Approved. >> >>

2 1