- ceph-users - lists.ceph.io

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Eugen Block 14 Apr '26

14 Apr '26

First thing coming to mind is: ceph mgr fail If you already did that something else is going on. Zitat von Marek Szuba via ceph-users <ceph-users(a)ceph.io>: > Hello, > > After the latest run of system updates and reboots on our prototype Ceph > cluster (Ubuntu Jammy + cephadm-managed Ceph 20.2.0 on Podman), we have > ended up with something seriously weird. Specifically, at a glance the > cluster appears to be in service: > > * Ceph reports the health of the cluster as OK > * according to both 'ceph -s' and 'ceph orch ps', all the daemons which > should be up and running ARE up and running. In particular, all the > expected OSDs appear to be operational > * 'ceph -s' shows no recovery I/O > > However: > > * despite all of the above, 'ceph -s' reports 1448 remapped PGs and > about 24% objects classified as misplaced > * 'ceph pg ls remapped' does report a long list of PGs with misplaced > objects > * on said list, most if not all PGs seem (I haven't checked all 1448 > lines) to have 3 OSDs each listed under ACTING but only either 1 or 2 > listed under UP - but not in a consistent way, in that an OSD missing > from the UP list for one PG does appear on the UP lists of other PGs > > Any idea what might be going on here, and how to make it stop? We were > about to upgrade that cluster to 20.2.1 following the system updates, > however we are very much hesitant to do so while things look so odd in > case the upgrade makes things worse. > > -- > MS > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Marek Szuba 14 Apr '26

14 Apr '26

Hello, After the latest run of system updates and reboots on our prototype Ceph cluster (Ubuntu Jammy + cephadm-managed Ceph 20.2.0 on Podman), we have ended up with something seriously weird. Specifically, at a glance the cluster appears to be in service: * Ceph reports the health of the cluster as OK * according to both 'ceph -s' and 'ceph orch ps', all the daemons which should be up and running ARE up and running. In particular, all the expected OSDs appear to be operational * 'ceph -s' shows no recovery I/O However: * despite all of the above, 'ceph -s' reports 1448 remapped PGs and about 24% objects classified as misplaced * 'ceph pg ls remapped' does report a long list of PGs with misplaced objects * on said list, most if not all PGs seem (I haven't checked all 1448 lines) to have 3 OSDs each listed under ACTING but only either 1 or 2 listed under UP - but not in a consistent way, in that an OSD missing from the UP list for one PG does appear on the UP lists of other PGs Any idea what might be going on here, and how to make it stop? We were about to upgrade that cluster to 20.2.1 following the system updates, however we are very much hesitant to do so while things look so odd in case the upgrade makes things worse. -- MS

1 0

Bluestore deferred writes (Squid)
by Magnus Grönlund 14 Apr '26

14 Apr '26

Hi All, We are running a 5 year old 15 node Ceph cluster with spinners for data and NVMe for DB/Wal. After the recent upgrade to Squid (from Reef) some users reported issues with fluctuating (often bad) disk performance especially affecting k8s etcd. They point to the change to bluestore_prefer_deferred_size_hdd claiming that it's detrimental to the "synchronous write patter" of escd. They also point to submitted_deferred_writes and issued_deferred_writes claiming that there is a massive and growing backlog of deferred writes. I'm not 100% correct that this is the issue but I'm having a hard time finding any relevant information (bad google-fu?) especially about what submitted and issued deferred writes actually represents and why the difference between issued - submitted keeps growing. Anyone got some insights to share? Thanks in advance Magnus

1 0

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 14 Apr '26

14 Apr '26

Hi Eugen! Thanks for the support, my response is inline On 14/04/26 13:03, Eugen Block via ceph-users wrote: > Hi, > > I don't have a solution (yet?), but a couple of points to clarify. > > - I assume your cluster isn't managed by cephadm, correct? Correct! It's an old cluster initialy deployed with ceph-deploy in the Hammer days and upgraded since then. Right now is manually managed (but one of the targets at the end of the upgrade is to switch to cephadm). > - Can you share the output of 'ceph mon dump'? Sure! # ceph mon dump epoch 18 fsid 5bed1bf6-123f-4597-aaaa-d77931228548 last_changed 2026-04-08T20:23:59.781295+0200 created 2018-01-09T10:20:36.351967+0100 min_mon_release 15 (octopus) election_strategy: 1 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 dumped monmap epoch 18 > - There were a couple of reports back at O/P versions about MONs not > joining the quorum. Sometimes it worked after hours, sometimes it > didn't, and I'm still not aware of a real solution. One of our customers > has been facing severe MON sync delay, sometimes taking around 40 > minutes until a MON joined quorum. It turned out to be caused by some > paxos default settings, requring the MONs to do a full sync instead of a > diff sync. This has to do with millions of snapshots they have in their > cluster. What made it worse was the MON's filesystems were on HDDs, > increasing the sync time as well. We didn't wait for a long time before stopping the new MON a regain "control" of the cluster, maybe 10 minutes max... > - Inspecting debug logs on the syncing MON could help finding the root > cause. With "syncing MON" you mean the MON which is just added to the cluster, right? Collecting this kind of logs is on the to-do list, but blocked because the "log problem" mentioned bellow. > > Some more comments inline. > [cut] >> >> We taught, among other things, to: >> >> - Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is the >> only daemon which wasn't restarted and when queried ("ceph daemon >> mon.node1 mon_status") has in "extra_probe_peers" the IP addresses of >> the original removed nodes, which are the same IPs as the nodes we >> want to add. We aren't sure if this prevent the ceph-mon to join, >> because we don't understand when/what "extra_probe_peers" is doing. > > According to the output above node1 is already out of the cluster, are > you referring to node3? > >> - Remove the node1 from the cluster, set min_mon_release to Quincy and >> then try to add the new monitors. Yes, sorry for the misunderstanding, I was referring to node3 in both instances. >> >> We are unsure about both approaches because if they don't work we will >> have a "unstable" cluster with only 2 monitors and no much space to >> operate. >> >> To "complicate" the matter we notice that on node4 and node5 the ceph- >> mon is not logging anymore to "/var/log/ceph/ceph*.log" even tough a >> "mon_cluster_log_to_file: true". If I search for open FD there is >> nothing in /var/log/ceph for the running ceph-mon... > > Is there something in journalctl? We increased the 'debug_mon' to 20 ('ceph daemon mon.node4 config set debug_mon 20') and we check it with 'ceph daemon mon.node4 config show' but the output of 'journalctl -u ceph-mon(a)mon.node4` is not more verbose then before, the last line is from 2:00 this morning and it's related to sudo: apr 14 02:35:40 node4 sudo[881890]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/md127 Iztok

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 14 Apr '26

14 Apr '26

Hi, I don't have a solution (yet?), but a couple of points to clarify. - I assume your cluster isn't managed by cephadm, correct? - Can you share the output of 'ceph mon dump'? - There were a couple of reports back at O/P versions about MONs not joining the quorum. Sometimes it worked after hours, sometimes it didn't, and I'm still not aware of a real solution. One of our customers has been facing severe MON sync delay, sometimes taking around 40 minutes until a MON joined quorum. It turned out to be caused by some paxos default settings, requring the MONs to do a full sync instead of a diff sync. This has to do with millions of snapshots they have in their cluster. What made it worse was the MON's filesystems were on HDDs, increasing the sync time as well. - Inspecting debug logs on the syncing MON could help finding the root cause. Some more comments inline. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi to all! > > I'm writing this (long) email to seek the wisdom of the community > about a problem we are facing in this moment. > > As I wrote before we would like to upgrade one of our Ceph cluster > from Octopus to Quincy (and then to Squid) and, at the same time, > upgrade the O.S. from centos7/rocky8 to rocky9. > > Our plan is (was?) to start from the MON/MGR nodes, bring those > nodes to Quincy and rocky9 and then re-install all the OSD servers > (one-by-one). Our initial setup for the MON/MGR nodes was: > > - node1: centos7+ceph15.2.16 > - node2: centos7+ceph15.2.16 > - node3: centos7+ceph15.2.16 > - node4: rocky8+ceph15.2.17 > - node5: rocky8+ceph15.2.17 > > To begin with we removed node1 and node2 from the cluster ("ceph mon > remove mon.node1") and reinstall it with rocky9+ceph17.2.9. > > As next steps we taught to: > 1. upgrade Ceph in-place on node4 and node5 to ceph17.2.6 > 2. add the freshly installed node1 and node2 to the cluster (with ceph17.2.9) > 3. remove, reinstall and re-add node3 to the cluster. > 4. remove, reinstall and re-add node4 and node5 to the cluster (to > have all rocky9 nodes). > > The first point went smoothly, but unfortunately the second did not > (and we are stopped there). > > We exported the monmap from the running cluster and created on the > freshly (re)installed hosts (node1, node2) the mon data directory. > Then we started the ceph-mon daemon on host node1. Once the daemon > started it got stuck in probing/electing, the commands to the > cluster (like "ceph -s") took a long time to return. If the ceph-mon > on node1 was stopped the command returned to work as usual and the > cluster reported (as it should) a warning about a monitor down (the > node1 we just stopped). Removing "mon.node1" from the cluster > returned the health to OK. > > Clearly everything screams about "network" and/or > "time-synchronization". But the networks seems fine, no firewall is > in place, ping between nodes is fine (checked also jumbo-frames) and > the services are able to communicate with each other. The same about > time, the clocks are all synchronized and no CLOCK_SKEW is detected. > > The current situation from the monitor standpoint is the following: > > in the cluster: > - node3: centos7+ceph15.2.16 > - node4: rocky8+ceph17.2.6 > - node5: rocky8+ceph17.2.6 > out-of-the-cluster: > - node1: rocky9+ceph17.2.9 > - node2: rocky9+ceph17.2.9 > > We are unable to decide what is our best course of action so we are > searching the wisdom from the community :-) > > We taught, among other things, to: > > - Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is the > only daemon which wasn't restarted and when queried ("ceph daemon > mon.node1 mon_status") has in "extra_probe_peers" the IP addresses > of the original removed nodes, which are the same IPs as the nodes > we want to add. We aren't sure if this prevent the ceph-mon to join, > because we don't understand when/what "extra_probe_peers" is doing. According to the output above node1 is already out of the cluster, are you referring to node3? > - Remove the node1 from the cluster, set min_mon_release to Quincy > and then try to add the new monitors. > > We are unsure about both approaches because if they don't work we > will have a "unstable" cluster with only 2 monitors and no much > space to operate. > > To "complicate" the matter we notice that on node4 and node5 the > ceph-mon is not logging anymore to "/var/log/ceph/ceph*.log" even > tough a "mon_cluster_log_to_file: true". If I search for open FD > there is nothing in /var/log/ceph for the running ceph-mon... Is there something in journalctl? > Have you some advice or you encounter similar problems in the past? > I didn't found anything, but probably I searched with the wrong > keywords. > > I can provide you with more information if needed. > > > Thanks a lot! > Iztok > > > > -- > Iztok Gregori > ICT Systems and Services > Elettra - Sincrotrone Trieste S.C.p.A. > http://www.elettra.eu > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Upgrade problem from Octopus to Quincy
by Iztok Gregori 14 Apr '26

14 Apr '26

Hi to all! I'm writing this (long) email to seek the wisdom of the community about a problem we are facing in this moment. As I wrote before we would like to upgrade one of our Ceph cluster from Octopus to Quincy (and then to Squid) and, at the same time, upgrade the O.S. from centos7/rocky8 to rocky9. Our plan is (was?) to start from the MON/MGR nodes, bring those nodes to Quincy and rocky9 and then re-install all the OSD servers (one-by-one). Our initial setup for the MON/MGR nodes was: - node1: centos7+ceph15.2.16 - node2: centos7+ceph15.2.16 - node3: centos7+ceph15.2.16 - node4: rocky8+ceph15.2.17 - node5: rocky8+ceph15.2.17 To begin with we removed node1 and node2 from the cluster ("ceph mon remove mon.node1") and reinstall it with rocky9+ceph17.2.9. As next steps we taught to: 1. upgrade Ceph in-place on node4 and node5 to ceph17.2.6 2. add the freshly installed node1 and node2 to the cluster (with ceph17.2.9) 3. remove, reinstall and re-add node3 to the cluster. 4. remove, reinstall and re-add node4 and node5 to the cluster (to have all rocky9 nodes). The first point went smoothly, but unfortunately the second did not (and we are stopped there). We exported the monmap from the running cluster and created on the freshly (re)installed hosts (node1, node2) the mon data directory. Then we started the ceph-mon daemon on host node1. Once the daemon started it got stuck in probing/electing, the commands to the cluster (like "ceph -s") took a long time to return. If the ceph-mon on node1 was stopped the command returned to work as usual and the cluster reported (as it should) a warning about a monitor down (the node1 we just stopped). Removing "mon.node1" from the cluster returned the health to OK. Clearly everything screams about "network" and/or "time-synchronization". But the networks seems fine, no firewall is in place, ping between nodes is fine (checked also jumbo-frames) and the services are able to communicate with each other. The same about time, the clocks are all synchronized and no CLOCK_SKEW is detected. The current situation from the monitor standpoint is the following: in the cluster: - node3: centos7+ceph15.2.16 - node4: rocky8+ceph17.2.6 - node5: rocky8+ceph17.2.6 out-of-the-cluster: - node1: rocky9+ceph17.2.9 - node2: rocky9+ceph17.2.9 We are unable to decide what is our best course of action so we are searching the wisdom from the community :-) We taught, among other things, to: - Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is the only daemon which wasn't restarted and when queried ("ceph daemon mon.node1 mon_status") has in "extra_probe_peers" the IP addresses of the original removed nodes, which are the same IPs as the nodes we want to add. We aren't sure if this prevent the ceph-mon to join, because we don't understand when/what "extra_probe_peers" is doing. - Remove the node1 from the cluster, set min_mon_release to Quincy and then try to add the new monitors. We are unsure about both approaches because if they don't work we will have a "unstable" cluster with only 2 monitors and no much space to operate. To "complicate" the matter we notice that on node4 and node5 the ceph-mon is not logging anymore to "/var/log/ceph/ceph*.log" even tough a "mon_cluster_log_to_file: true". If I search for open FD there is nothing in /var/log/ceph for the running ceph-mon... Have you some advice or you encounter similar problems in the past? I didn't found anything, but probably I searched with the wrong keywords. I can provide you with more information if needed. Thanks a lot! Iztok -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. http://www.elettra.eu

1 0

Re: Status of iSCSI-Gateway
by Alexander Schreiber 14 Apr '26

14 Apr '26

On Mon, Apr 13, 2026 at 02:28:55PM +0000, Eugen Block via ceph-users wrote: > No, my understanding was that you can give clients access to RBD's through > the nvme-gateway, they see it as an nvme drive. And due to Aviv's help on > slack I was able to confirm this understanding, I have a single node ceph > cluster (virtual machine), deployed nvme-of and see the RBD image as an nvme > drive within the VM now: > > test:~ # nvme list > Node Generic SN Model > Namespace Usage Format FW Rev > --------------------- --------------------- -------------------- > ---------------------------------------- ---------- > -------------------------- ---------------- -------- > /dev/nvme1n1 /dev/ng1n1 Ceph42382366687857 Ceph bdev > Controller 0x1 10.74 GB / 10.74 GB 512 B > + 0 B 25.05.1 So it is really just the transport, ok. Thank you. Kind regards, Alex. -- "Opportunity is missed by most people because it is dressed in overalls and looks like work." -- Thomas A. Edison

1 0

Re: Has anyone been able to use NFS in tentacle or it is really broken?
by Eugen Block 14 Apr '26

14 Apr '26

Good point, no it was not. After starting it and redeploying NFS (v3 is then in the config again) the daemon doesn't crash anymore. Thanks for pointing that out! Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Am 13.04.26 um 10:36 AM schrieb Eugen Block via ceph-users: >> >> the host where I try to deploy nfs (openSUSE Leap 16 virtual >> machine) has rpcbind installed and it still fails: >> >> storage:~ # rpm -qa | grep rpcbind >> rpcbind-1.2.7-160000.2.2.x86_64 >> > > Is the rpcbind service running? > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Has anyone been able to use NFS in tentacle or it is really broken?
by Robert Sander 14 Apr '26

14 Apr '26

Am 13.04.26 um 10:36 AM schrieb Eugen Block via ceph-users: > > the host where I try to deploy nfs (openSUSE Leap 16 virtual machine) > has rpcbind installed and it still fails: > > storage:~ # rpm -qa | grep rpcbind > rpcbind-1.2.7-160000.2.2.x86_64 > Is the rpcbind service running? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: squid 19.2.4 QE validation status
by Ilya Dryomov 13 Apr '26

13 Apr '26

On Mon, Apr 13, 2026 at 9:30 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75029#note-5 > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya rbd and krbd approved. Thanks, Ilya

1 0