May 2026 - ceph-users - lists.ceph.io

tentacle 20.2.2 QE validation status
by Yuri Weinstein 08 Jun '26

08 Jun '26

Details of this release are summarized here: https://tracker.ceph.com/issues/75878#note-6 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv, Vallari upgrade: Patrick upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 4

Re: Best "modern" approach to drain/fill a OSD server
by Janne Johansson 05 Jun '26

05 Jun '26

> If it can be useful, I'm thinking of organizing an OSD balancing happy > hour call in the next month or so. > Maybe mid-May or so on a Friday afternoon (EU time) / morning (US time). Also interested. -- May the most significant bit of your life be positive.

3 2

Re: RBD: group snap rollback invalid argument
by Ilya Dryomov 03 Jun '26

03 Jun '26

On Fri, May 29, 2026 at 2:33 PM Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I wanted to rollback a group snapshot on Ubuntu 24.04 (rbd client > version 19.2.1), the Ceph cluster version is 19.2.3. The client fails > with "invalid argument": > > controller02:~# rbd --id <user> group snap rollback --pool images > --group test-servers --snap 20260430_start > Rolling back to group snapshot: 0% complete...failed. > rbd: rollback group to snapshot failed: (22) Invalid argument > > controller02:~# ceph -v > ceph version 19.2.1 (9efac4a81335940925dd17dbf407bfd6d3860d28) squid (stable) > > But running the same command (just as admin not as <user>) on a Ceph > node works: > > ceph03:~ # rbd group snap rollback --pool images --group test-servers > --snap 20260430_start > Rolling back to group snapshot: 100% complete...done. > > ceph03:~ # ceph -v > ceph version 16.2.13-66-g54799ee0666 > (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) > > > What seems to be the issue here is that I removed an image from the > group snapshot. I wonder if it could be this bug [0] which is supposed > to be fixed in 19.2.0 according to the "Released In" field of the > Squid backport tracker [1]. > > This seems a little inconsistent to me, could someone please clarify? Hi Eugen, Did you see "group snapshot membership does not match group membership" error message when the rollback command failed with 19.2.1 client? Thanks, Ilya

2 10

Re: Integrating OSDs from a previous Ceph installation
by Jacek Rużyczka 31 May '26

31 May '26

> > No, the data is not inaccessible until you wipe the OSDs. Ah, that's good to know that at least the data are still there. How many OSDs are we talking about here? I've got 4 OSDs, which hold 1.3 TB of data in total. I do have a backup of them on an external HDD, which spit out a handful of warnings during the last restore, so destroying the previous SSD, creating new ones, and trying another restore are things I'd like to avoid. And regarding the fsid, you can specify it in the bootstrap command, > you won’t be able to change it afterwards. This is exactly what I tried to do: sudo cephadm bootstrap --mon-ip $MYIP --config /etc/ceph/ceph.conf~~ --allow-overwrite And /etc/ceph/ceph.conf~~ contains the old FSID: # minimal ceph.conf for 8aad3073-39a1-11f1-bf6e-f2704a1efa9b [global] fsid = 8aad3073-39a1-11f1-bf6e-f2704a1efa9b mon_host = [v2:10.20.0.11:3300/0,v1:10.20.0.11:6789/0] [v2: 10.20.0.12:3300/0,v1:10.20.0.12:6789/0] [v2: 10.20.0.13:3300/0,v1:10.20.0.13:6789/0] [v2: 10.20.0.14:3300/0,v1:10.20.0.14:6789/0] Nevertheless, cephadm ignored the old settings and gave the new cluster a generated ID. So I've gotta tear down the cluster and start again? What about the key pairs stored in /etc/ceph? You can simply create the OSD directories and create a symbolic link to > the block device. Cephadm simply maps it then. I've now recreated the dir /var/lib/ceph/osd/ceph-2 and have to symlink /dev/ceph-81bc1761-e8f6-446c-96f3-eb1b8f92628b/o sd-block-9f7fd40d-0698-40b9-8718-62942b03e263 there? Or should ceph-2 be the name of the symlink? I recommend to hire a professional to help you out, just to be on the safe > side. I've got a Ceph expert in our hackspace, but I'm not sure whether he can help me.

2 1

Re: Integrating OSDs from a previous Ceph installation
by Jacek Rużyczka 31 May '26

31 May '26

Hi, and here, everything ends for me: ceph-objectstore-tool --data-path */var/lib/ceph/osd/ceph-**0* --no-mon-config --op update-mon-db --mon-store-path /tmp/mon-store As all hosts have died, I no longer have the necessary /var/lib/ceph/osd folder. And: I've tried to bring up a new cluster with cephadm bootstrap --mon-ip $MYIP --initial-config /etc/ceph/ ceph.conf~~, which itself was the config file containing the original ID, namely, 8aad3073-39a1-11f1-bf6e-f2704a1efa9b, but this was apparently ignored by cephadm. My new cluster now has this ID: add010c1-5cfa-11f1-8af8-ceccf52b4a0f So: Have my data become inaccessible?

2 1

Re: The mail threading on this list is broken
by Adam Kraitman 31 May '26

31 May '26

Thank you for the detailed analysis, the threading issue you've described is a known problem with the current mailing list infrastructure. We're planning to migrate all mailing lists to Groups.io, and this issue will be resolved as part of that migration. We'll share more details and the migration timeline once everything is finalized On Sun, May 31, 2026 at 3:08 PM Kai Stian Olstad via ceph-users < ceph-users(a)ceph.io> wrote: > Hi > > For several month the mail threading on this list broken and making the > list hard to read. > It can easily be seen by looking at the archive, in this link for about > 9 mount(as of writing) > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… > > Subjects all have nice subject without "Re:" for replay, and on the > right you see how may people(person icon) in the thread and number of > post(talking bubble) in the thread. > > But if we look at more recent posts like this one > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… > you see a lot of "Re:" in the subject and most of them have one person > and only one post because the threading is broken. > > Take this recent mail "Weird behavior for 2 OSDs in our cluster" > The fist mail by Wannes Smet > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/62B63BH33F3… > when i check my mail client this mail has following ID. > "Message-ID: > < > 0100019e7c77595f-ffb4d04b-1771-46cf-8b60-10795bc6a1c5-000000(a)email.amazonses.com > >" > > The reply from Eugen Block > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/E63Z3JR7DMS… > the header says this is a reply to different ID > "In-Reply-To: > < > 0100019e7c7979e5-8d4d96e6-1c93-4a76-a5cd-efdf25ada853-000000(a)email.amazonses.com > >" > > When I search the header of the mail from Wannes Smet I find no such ID > that is in the In-Reply-To. > > It looks to me that the mailing-list server is sending different > "Messages-ID" to the members on the list. > This should be the same or the threading breaks as we can se. > > Can this be fixed please? > > -- > Kai Stian Olstad > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

The mail threading on this list is broken
by Kai Stian Olstad 31 May '26

31 May '26

Hi For several month the mail threading on this list broken and making the list hard to read. It can easily be seen by looking at the archive, in this link for about 9 mount(as of writing) https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… Subjects all have nice subject without "Re:" for replay, and on the right you see how may people(person icon) in the thread and number of post(talking bubble) in the thread. But if we look at more recent posts like this one https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… you see a lot of "Re:" in the subject and most of them have one person and only one post because the threading is broken. Take this recent mail "Weird behavior for 2 OSDs in our cluster" The fist mail by Wannes Smet https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/62B63BH33F3… when i check my mail client this mail has following ID. "Message-ID: <0100019e7c77595f-ffb4d04b-1771-46cf-8b60-10795bc6a1c5-000000(a)email.amazonses.com>" The reply from Eugen Block https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/E63Z3JR7DMS… the header says this is a reply to different ID "In-Reply-To: <0100019e7c7979e5-8d4d96e6-1c93-4a76-a5cd-efdf25ada853-000000(a)email.amazonses.com>" When I search the header of the mail from Wannes Smet I find no such ID that is in the In-Reply-To. It looks to me that the mailing-list server is sending different "Messages-ID" to the members on the list. This should be the same or the threading breaks as we can se. Can this be fixed please? -- Kai Stian Olstad

1 0

Re: Weird behavior for 2 OSDs in our cluster
by Eugen Block 31 May '26

31 May '26

Hi, the MGR doesn't always report the correct PG status, so don't rely on that too much. Sometimes it's necessary to restart primary OSDs for stuck PGs, although a repeer could have been sufficient. Your Ceph clients had to refresh their osdmap, that's when they notice that there had been down OSDs. It's not a real-time log in this case, no need to worry. It's a common question though, I think we also asked it 8 to 10 years ago. ;-) Regards, Eugen Zitat von Wannes Smet via ceph-users <ceph-users(a)ceph.io>: > Hi, > > I'm running a Ceph cluster 19.2.2, 23 nodes, 152 OSDs, cephadm > deployed. Most SAS SSDs, 12 NVMe SSDs. > > Yesterday we experienced a total power failure and everything went > down hard. Also our Ceph cluster. There were a couple of things, but > this stood out after it got back up: > > [ERR] OSD_UNREACHABLE: 2 osds(s) are not reachable > osd.53's public address is not in '192.168.11.0/24' subnet > osd.86's public address is not in '192.168.11.0/24' subnet > > ceph -s did not say reduced data {availability,redundancy} which is > a bit "off", given that both OSDs are in separate hosts, failure > domain=host. There must have been PGs with less than 3 replicas and > also PGs with just one replica left? > > So I manually restarted those OSDs with systemctl , a recovery > process started and all our VMs, "magically" started booting now. > I'm also surprised that the recovery process only started when those > OSDs got back up. > > I didn't make too much of the above, but now this morning, I'm > looking at the kernel ring buffer of our PVE nodes and I notice the > logs below. Just a single "blip". All at the same time on all of our > PVE nodes (ceph clients): > > [Sat May 30 22:03:46 2026] libceph > (e8020818-2100-11f0-8a12-9cdc71772100 e179035): osd53 down > [Sat May 30 22:03:46 2026] libceph > (e8020818-2100-11f0-8a12-9cdc71772100 e179050): osd53 up > [Sat May 30 22:03:46 2026] libceph > (e8020818-2100-11f0-8a12-9cdc71772100 e179057): osd86 down > [Sat May 30 22:03:46 2026] libceph > (e8020818-2100-11f0-8a12-9cdc71772100 e179074): osd86 up > > I don't see anything weird in the Ceph cluster itself, neither in > the log files of the ODS. > > I'm not sure what to make from this. Why would this happen and what > would you do? > > Thanks for your insights, > > Wannes Smet > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Integrating OSDs from a previous Ceph installation
by Eugen Block 31 May '26

31 May '26

Again, please don't drop the list from your responses. I played around a bit with a single-host cluster, created three OSDs, added a pool and copied a few bytes of data into it. Then I attached the drives to a different host where I bootstrapped a fresh cluster with the previous FSID. But the OSDs can't be activated, unfortunately. I think the main problem is this: orchestrator._interface.OrchestratorError: osd.0 not in osdmap I'm not sure if the procedure to recover from OSDs will help here, if I have the time, I'm gonna try. Or another approach could be to create new OSDs in the new cluster you built (if you have the resources) and then use ceph-objectstore-tool export/import to get the data onto the new OSDs. But that's also an advanced procedure, you require experience to accomplish that. I still think the easiest approach would have been to reduce the monmap to 1 and start from there, but it's too late for that now. Zitat von Jacek Rużyczka <jacek.ruzyczka(a)googlemail.com>: > But then I'd have to destroy the old Ceph cluster, right? > > Am Sa., 30. Mai 2026 um 11:15 Uhr schrieb Eugen Block via ceph-users < > ceph-users(a)ceph.io>: > >> I haven't tried it myself yet, but you could bootstrap a fresh cluster >> with the flag "--fsid", excerpt from the 'cephadm bootstrap -h' command: >> >> --fsid FSID cluster FSID >> >> and set the FSID your OSDs show. But again, I have no idea if that's >> gonna work. >> >> Zitat von Eugen Block <eblock(a)nde.ag>: >> >> > You're not gonna be able to reactivate those OSDs because they were >> > built with a different cluster FSID: >> > >> > Your OSDs: >> > >> >> cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b >> > >> > Your new cluster: >> > >> >> Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f >> > >> > And if you have a cephadm-managed cluster, you shouldn't use >> > ceph-volume directly but via cephadm [0]: >> > >> > ceph cephadm osd activate <node> >> > >> > But this won't help you if the cluster's fsid differs as mentioned >> > above. I think you might be able to recover if you recreate the >> > monmap by using the OSDs [1]. But this would mean you'd need to tear >> > down your MONs, it's a difficult situation, I can't tell if it's >> > gonna work that way. The procedure itself has been verified several >> > times on this list, but again, it's hard to tell. >> > >> > [0] >> > >> https://docs.ceph.com/en/latest/cephadm/services/osd/#activate-existing-osds >> > [1] >> > >> https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… >> > >> > >> > Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: >> > >> >> Hi, >> >> >> >> After being forced to reinstall the O/S on all my Ceph nodes, I'm >> trying to >> >> reactivate the OSDs, whose data lie on separate drives. The OSDs are >> >> recognised: >> >> >> >> mixtile@blade3n3:~$ sudo sudo ceph-volume lvm list >> >> [sudo] password for mixtile: >> >> >> >> >> >> ====== osd.1 ======= >> >> >> >> [block] >> >> /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd >> >> 2-0503-440f-b7c2-47e8d8e85253 >> >> >> >> block device >> >> /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/o >> >> sd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 >> >> block uuid 21ftde-FdBB-s4DM-Qeq7-TU4R-vjY7-6VXLoW >> >> cephx lockbox secret >> >> cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b >> >> cluster name ceph >> >> crush device class >> >> encrypted 0 >> >> osd fsid ccba3dd2-0503-440f-b7c2-47e8d8e85253 >> >> osd id 1 >> >> osdspec affinity all-available-devices >> >> type block >> >> vdo 0 >> >> with tpm 0 >> >> devices /dev/nvme0n1 >> >> >> >> *But: *When trying to link the OSD to the all-new Ceph installation >> (I've >> >> made several Google searches with contradictory results. This is the >> last >> >> one.), I get an error: >> >> >> >> mixtile@blade3n3:~$ sudo cephadm shell >> >> Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f >> >> Inferring config >> >> /var/lib/ceph/98e04296-5b5a-11f1-84cf-ceccf52b4a0f/mon.blade3n3 >> >> /config >> >> Not using image >> >> 'sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16 >> >> c26b6cee' as it's not in list of non-dangling images with ceph=True >> label >> >> Unable to find image 'quay.io/ceph/ceph:v19' locally >> >> v19: Pulling from ceph/ceph >> >> Digest: >> >> sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee >> >> Status: Downloaded newer image for quay.io/ceph/ceph:v19 >> >> >> >> root@blade3n3:/usr/bin# cd >> >> >> >> root@blade3n3:~# ceph-volume lvm activate --all >> >> Running command: /usr/bin/ceph-authtool --gen-print-key >> >> Running command: /usr/bin/ceph-authtool --gen-print-key >> >> --> Activating OSD ID 1 FSID ccba3dd2-0503-440f-b7c2-47e8d8e85253 >> >> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 >> >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 >> >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph >> prime-osd-dir >> >> --dev >> >> >> /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 >> >> --path /var/lib/ceph/osd/ceph-1 --no-mon-config >> >> stderr: 2026-05-29T17:28:36.854+0000 ffffbd2de040 -1 bdev(0xaaab21623800 >> >> /dev/c >> >> >> eph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d >> >> 8e85253) open stat got: (1) Operation not permitted >> >> stderr: failed to read label for >> >> /dev/ceph-5b8338a0-9246-48f5-9cfd-1bbfa1cc199 >> >> /osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253: (1) Operation not >> >> permitted >> >> --> RuntimeError: command returned non-zero exit status: 1 >> >> >> >> This puzzles me because I'm already root, but the Bluestore tool won't >> let >> >> me reactivate the OSD? The old OSD is now full of valuable production >> data. >> >> And: Isn't it possible to use ceph orch to perform this operation? >> >> >> >> Regards >> >> Jacek Rużyczka >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 1

Weird behavior for 2 OSDs in our cluster
by Wannes Smet 31 May '26

31 May '26

Hi, I'm running a Ceph cluster 19.2.2, 23 nodes, 152 OSDs, cephadm deployed. Most SAS SSDs, 12 NVMe SSDs. Yesterday we experienced a total power failure and everything went down hard. Also our Ceph cluster. There were a couple of things, but this stood out after it got back up: [ERR] OSD_UNREACHABLE: 2 osds(s) are not reachable osd.53's public address is not in '192.168.11.0/24' subnet osd.86's public address is not in '192.168.11.0/24' subnet ceph -s did not say reduced data {availability,redundancy} which is a bit "off", given that both OSDs are in separate hosts, failure domain=host. There must have been PGs with less than 3 replicas and also PGs with just one replica left? So I manually restarted those OSDs with systemctl , a recovery process started and all our VMs, "magically" started booting now. I'm also surprised that the recovery process only started when those OSDs got back up. I didn't make too much of the above, but now this morning, I'm looking at the kernel ring buffer of our PVE nodes and I notice the logs below. Just a single "blip". All at the same time on all of our PVE nodes (ceph clients): [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179035): osd53 down [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179050): osd53 up [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179057): osd86 down [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179074): osd86 up I don't see anything weird in the Ceph cluster itself, neither in the log files of the ODS. I'm not sure what to make from this. Why would this happen and what would you do? Thanks for your insights, Wannes Smet

1 0

Re: Integrating OSDs from a previous Ceph installation
by Eugen Block 30 May '26

30 May '26

You're not gonna be able to reactivate those OSDs because they were built with a different cluster FSID: Your OSDs: > cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b Your new cluster: > Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f And if you have a cephadm-managed cluster, you shouldn't use ceph-volume directly but via cephadm [0]: ceph cephadm osd activate <node> But this won't help you if the cluster's fsid differs as mentioned above. I think you might be able to recover if you recreate the monmap by using the OSDs [1]. But this would mean you'd need to tear down your MONs, it's a difficult situation, I can't tell if it's gonna work that way. The procedure itself has been verified several times on this list, but again, it's hard to tell. [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#activate-existing-osds [1] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > Hi, > > After being forced to reinstall the O/S on all my Ceph nodes, I'm trying to > reactivate the OSDs, whose data lie on separate drives. The OSDs are > recognised: > > mixtile@blade3n3:~$ sudo sudo ceph-volume lvm list > [sudo] password for mixtile: > > > ====== osd.1 ======= > > [block] > /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd > 2-0503-440f-b7c2-47e8d8e85253 > > block device > /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/o > sd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 > block uuid 21ftde-FdBB-s4DM-Qeq7-TU4R-vjY7-6VXLoW > cephx lockbox secret > cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > cluster name ceph > crush device class > encrypted 0 > osd fsid ccba3dd2-0503-440f-b7c2-47e8d8e85253 > osd id 1 > osdspec affinity all-available-devices > type block > vdo 0 > with tpm 0 > devices /dev/nvme0n1 > > *But: *When trying to link the OSD to the all-new Ceph installation (I've > made several Google searches with contradictory results. This is the last > one.), I get an error: > > mixtile@blade3n3:~$ sudo cephadm shell > Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f > Inferring config > /var/lib/ceph/98e04296-5b5a-11f1-84cf-ceccf52b4a0f/mon.blade3n3 > /config > Not using image > 'sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16 > c26b6cee' as it's not in list of non-dangling images with ceph=True label > Unable to find image 'quay.io/ceph/ceph:v19' locally > v19: Pulling from ceph/ceph > Digest: > sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee > Status: Downloaded newer image for quay.io/ceph/ceph:v19 > > root@blade3n3:/usr/bin# cd > > root@blade3n3:~# ceph-volume lvm activate --all > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph-authtool --gen-print-key > --> Activating OSD ID 1 FSID ccba3dd2-0503-440f-b7c2-47e8d8e85253 > Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir > --dev > /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 > --path /var/lib/ceph/osd/ceph-1 --no-mon-config > stderr: 2026-05-29T17:28:36.854+0000 ffffbd2de040 -1 bdev(0xaaab21623800 > /dev/c > eph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d > 8e85253) open stat got: (1) Operation not permitted > stderr: failed to read label for > /dev/ceph-5b8338a0-9246-48f5-9cfd-1bbfa1cc199 > /osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253: (1) Operation not > permitted > --> RuntimeError: command returned non-zero exit status: 1 > > This puzzles me because I'm already root, but the Bluestore tool won't let > me reactivate the OSD? The old OSD is now full of valuable production data. > And: Isn't it possible to use ceph orch to perform this operation? > > Regards > Jacek Rużyczka > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

Short survey for anyone running 15-30TB SSD OSDs
by Anthony D'Atri 29 May '26

29 May '26

* Or even 60 TB SSD OSDs * Which Ceph release? * Are you using the bitmap or hybrid bluefs / BlueStore allocators? * Are you serving CephFS from them? * Autotuning or static BlueStore cache / osd_memory_target values? * What’s the RSS like on your OSDs? I’m seeing some unexpected behavior and would like some context from others’ experience. I suggest replying privately so as to not flood the list, and I’ll summarize. This will help me directly as well as evolve the hardware recommendations in the docs. Thanks. Anthony D’Atri Ceph Ambassador, Doc lead, etc.

1 0

Re: tentacle 20.2.2 QE validation status
by Adam Emerson 29 May '26

29 May '26

On 29/05/2026, Yuri Weinstein via dev wrote: > We are still missing several approvals (rgw, rbd, orch, ceph-volume) > Please review ASAP > TIA RGW approved. > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/75878#note-6 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya > > nvmeof: Aviv, Vallari > > > > upgrade: Patrick > > > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > > > perf-basic: PASSED > > > > crimson-rados: N/A > > > > powercycle: PASSED > > > > TIA > > -- Adam Emerson https://people.redhat.com/~aemerson/ Principal Software Engineer, IBM Storage they/them/their aemerson(a)redhat.com IRC: Aemerson@{RedHat,OFTC} 0x80F7544B90EDBFB9 E707 86BA 0C1B 62CC 152C 7C12 80F7 544B 90ED BFB9

1 0

Integrating OSDs from a previous Ceph installation
by Jacek Rużyczka 29 May '26

29 May '26

Hi, After being forced to reinstall the O/S on all my Ceph nodes, I'm trying to reactivate the OSDs, whose data lie on separate drives. The OSDs are recognised: mixtile@blade3n3:~$ sudo sudo ceph-volume lvm list [sudo] password for mixtile: ====== osd.1 ======= [block] /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd 2-0503-440f-b7c2-47e8d8e85253 block device /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/o sd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 block uuid 21ftde-FdBB-s4DM-Qeq7-TU4R-vjY7-6VXLoW cephx lockbox secret cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b cluster name ceph crush device class encrypted 0 osd fsid ccba3dd2-0503-440f-b7c2-47e8d8e85253 osd id 1 osdspec affinity all-available-devices type block vdo 0 with tpm 0 devices /dev/nvme0n1 *But: *When trying to link the OSD to the all-new Ceph installation (I've made several Google searches with contradictory results. This is the last one.), I get an error: mixtile@blade3n3:~$ sudo cephadm shell Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f Inferring config /var/lib/ceph/98e04296-5b5a-11f1-84cf-ceccf52b4a0f/mon.blade3n3 /config Not using image 'sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16 c26b6cee' as it's not in list of non-dangling images with ceph=True label Unable to find image 'quay.io/ceph/ceph:v19' locally v19: Pulling from ceph/ceph Digest: sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee Status: Downloaded newer image for quay.io/ceph/ceph:v19 root@blade3n3:/usr/bin# cd root@blade3n3:~# ceph-volume lvm activate --all Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key --> Activating OSD ID 1 FSID ccba3dd2-0503-440f-b7c2-47e8d8e85253 Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 --path /var/lib/ceph/osd/ceph-1 --no-mon-config stderr: 2026-05-29T17:28:36.854+0000 ffffbd2de040 -1 bdev(0xaaab21623800 /dev/c eph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d 8e85253) open stat got: (1) Operation not permitted stderr: failed to read label for /dev/ceph-5b8338a0-9246-48f5-9cfd-1bbfa1cc199 /osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253: (1) Operation not permitted --> RuntimeError: command returned non-zero exit status: 1 This puzzles me because I'm already root, but the Bluestore tool won't let me reactivate the OSD? The old OSD is now full of valuable production data. And: Isn't it possible to use ceph orch to perform this operation? Regards Jacek Rużyczka

1 0

Re: tentacle 20.2.2 QE validation status
by Ilya Dryomov 29 May '26

29 May '26

On Fri, May 29, 2026 at 4:43 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > We are still missing several approvals (rgw, rbd, orch, ceph-volume) rbd and krbd approved. Thanks, Ilya > Please review ASAP > TIA > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/75878#note-6 >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya >> nvmeof: Aviv, Vallari >> >> upgrade: Patrick >> >> upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD >> upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD >> >> perf-basic: PASSED >> >> crimson-rados: N/A >> >> powercycle: PASSED >> >> TIA

1 0

RBD: group snap rollback invalid argument
by Eugen Block 29 May '26

29 May '26

Hi, I wanted to rollback a group snapshot on Ubuntu 24.04 (rbd client version 19.2.1), the Ceph cluster version is 19.2.3. The client fails with "invalid argument": controller02:~# rbd --id <user> group snap rollback --pool images --group test-servers --snap 20260430_start Rolling back to group snapshot: 0% complete...failed. rbd: rollback group to snapshot failed: (22) Invalid argument controller02:~# ceph -v ceph version 19.2.1 (9efac4a81335940925dd17dbf407bfd6d3860d28) squid (stable) But running the same command (just as admin not as <user>) on a Ceph node works: ceph03:~ # rbd group snap rollback --pool images --group test-servers --snap 20260430_start Rolling back to group snapshot: 100% complete...done. ceph03:~ # ceph -v ceph version 16.2.13-66-g54799ee0666 (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) What seems to be the issue here is that I removed an image from the group snapshot. I wonder if it could be this bug [0] which is supposed to be fixed in 19.2.0 according to the "Released In" field of the Squid backport tracker [1]. This seems a little inconsistent to me, could someone please clarify? Thanks! Eugen [0] https://tracker.ceph.com/issues/66300 [1] https://tracker.ceph.com/issues/66589 -- Eugen Block NDE Netzdesign und -entwicklung AG voice: +49 40 5595175 Postfach 61 03 15 e-mail: eblock(a)nde.ag D-22423 Hamburg Vorstand: Jens-U. Mozdzen Aufsichtsratsvorsitzende: Angelika Torlée-Mozdzen Sitz und Registergericht: Hamburg, HRB 90934 USt-IdNr: DE 814 013 983

1 0

Re: tentacle 20.2.2 QE validation status
by Travis Nielsen 28 May '26

28 May '26

Rook approved. As discussed separately, we will follow up after 20.2.2 on this rbd mirror issue. https://tracker.ceph.com/issues/76967 Thanks, Travis On Thu, May 28, 2026 at 2:02 AM Nitzan Mordechai via dev <dev(a)ceph.io> wrote: > Rados approved > Upgrade approved > > > Summary of analysis: > > https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2022-httpstrackercep… > > ------------------------------ > *From:* Vallari Agrawal via dev <dev(a)ceph.io> > *Sent:* Wednesday, May 27, 2026 6:12 PM > *To:* Afreen <afreen23.git(a)gmail.com>; Aviv Caro <acaro(a)redhat.com> > *Cc:* Yuri Weinstein <yweinste(a)redhat.com>; dev <dev(a)ceph.io>; ceph-users > <ceph-users(a)ceph.io> > *Subject:* [EXTERNAL] Re: [ceph-users] Re: tentacle 20.2.2 QE validation > status > > nvmeof approved cc: @Aviv Caro On Tue, May 26, 2026 at 10: 44 PM Afreen > via dev <dev@ ceph. io> wrote: Dashboard approved On Tue, May 26, 2026 at > 10: 00 PM Yuri Weinstein via ceph-users <ceph-users@ ceph. io> wrote: > Resending because of > nvmeof approved > cc: @Aviv Caro <acaro(a)redhat.com> > > On Tue, May 26, 2026 at 10:44 PM Afreen via dev <dev(a)ceph.io> wrote: > > Dashboard approved > > On Tue, May 26, 2026 at 10:00 PM Yuri Weinstein via ceph-users < > ceph-users(a)ceph.io> wrote: > > Resending because of emails problems > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/75878#note-6 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya > > nvmeof: Aviv, Vallari > > > > upgrade: Patrick > > > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > > > perf-basic: PASSED > > > > crimson-rados: N/A > > > > powercycle: PASSED > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: tentacle 20.2.2 QE validation status
by Nitzan Mordechai 28 May '26

28 May '26

Rados approved Upgrade approved Summary of analysis: https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2022-httpstrackercep… ________________________________ From: Vallari Agrawal via dev <dev(a)ceph.io> Sent: Wednesday, May 27, 2026 6:12 PM To: Afreen <afreen23.git(a)gmail.com>; Aviv Caro <acaro(a)redhat.com> Cc: Yuri Weinstein <yweinste(a)redhat.com>; dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] Re: [ceph-users] Re: tentacle 20.2.2 QE validation status nvmeof approved cc: @Aviv Caro On Tue, May 26, 2026 at 10: 44 PM Afreen via dev <dev@ ceph. io> wrote: Dashboard approved On Tue, May 26, 2026 at 10: 00 PM Yuri Weinstein via ceph-users <ceph-users@ ceph. io> wrote: Resending because of nvmeof approved cc: @Aviv Caro<mailto:acaro@redhat.com> On Tue, May 26, 2026 at 10:44 PM Afreen via dev <dev(a)ceph.io<mailto:dev@ceph.io>> wrote: Dashboard approved On Tue, May 26, 2026 at 10:00 PM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> wrote: Resending because of emails problems On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com<mailto:yweinste@redhat.com>> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75878#note-6<https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > TIA > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

Re: Cluster Dead After Network Failure. Connection Timeout.
by Eugen Block 27 May '26

27 May '26

Please don't drop the list from your responses, you'd benefit from more people reading it. The cephadm ls output isn't really helpful, you need to figure out why docker doesn't start. Either syslog, journald or dmesg or whatever should give some clue. And to me it sounds like there has been more going on beside the network outage, maybe some leftovers from previous deployments or tests or something else that "confuses" docker? Maybe you have both podman and docker installed and since cephadm prefers podman, the containers fail to start? Zitat von Jacek Rużyczka <jacek.ruzyczka(a)googlemail.com>: > I can't run the Ceph container on the master node (blade3n1) anymore. It's > not executed anymore without an error message. Here is what cephadm ls says: > > mixtile@blade3n1:~$ sudo cephadm ls > [ > { > "style": "cephadm:v1", > "name": "mon.blade3n1", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)mon.blade3n1" > , > "enabled": true, > "state": "error", > "service_name": "mon", > "memory_request": null, > "memory_limit": null, > "ports": [], > "container_id": null, > "container_image_name": "quay.io/ceph/ceph:v19", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T14:35:47.634066Z", > "deployed": "2026-04-16T14:35:45.414037Z", > "configured": "2026-04-20T17:16:32.722329Z" > }, > { > "style": "cephadm:v1", > "name": "node-exporter.blade3n1", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b@node-exporter > .blade3n1", > "enabled": true, > "state": "error", > "service_name": "node-exporter", > "ports": [ > 9100 > ], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": "quay.io/prometheus/node-exporter:v1.7.0", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-21T12:55:39.731035Z", > "deployed": "2026-04-21T12:55:38.217675Z", > "configured": "2026-04-21T12:55:39.734369Z" > }, > { > "style": "cephadm:v1", > "name": "ceph-exporter.blade3n1", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b@ceph-exporter > .blade3n1", > "enabled": true, > "state": "error", > "service_name": "ceph-exporter", > "ports": [], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe2 > 19dfc8d0c3efc1f05102a753902f33ee16c26b6cee", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T14:37:32.218782Z", > "deployed": "2026-04-16T14:37:30.612094Z", > "configured": "2026-04-20T17:16:36.139048Z" > }, > { > "style": "cephadm:v1", > "name": "mgr.blade3n1.rrlwwv", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)mgr.blade3n1. > rrlwwv", > "enabled": true, > "state": "error", > "service_name": "mgr", > "memory_request": null, > "memory_limit": null, > "ports": [ > 9283, > 8765, > 8443 > ], > "container_id": null, > "container_image_name": "quay.io/ceph/ceph:v19", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T14:35:54.054151Z", > "deployed": "2026-04-16T14:35:52.430796Z", > "configured": "2026-04-20T17:16:37.612403Z" > }, > { > "style": "cephadm:v1", > "name": "crash.blade3n1", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)crash.blade3n > 1", > "enabled": true, > "state": "error", > "service_name": "crash", > "ports": [], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe2 > 19dfc8d0c3efc1f05102a753902f33ee16c26b6cee", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T14:37:36.855510Z", > "deployed": "2026-04-16T14:37:35.268822Z", > "configured": "2026-04-20T17:16:39.025758Z" > }, > { > "style": "cephadm:v1", > "name": "osd.3", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)osd.3", > "enabled": true, > "state": "error", > "service_name": "osd", > "ports": [], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe2 > 19dfc8d0c3efc1f05102a753902f33ee16c26b6cee", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-23T15:05:00.686688Z", > "deployed": "2026-04-23T15:04:59.176667Z", > "configured": "2026-04-23T15:05:00.693355Z" > }, > { > "style": "cephadm:v1", > "name": "mds.data.blade3n1.eczeqc", > "fsid": "8aad3073-39a1-11f1-bf6e-f2704a1efa9b", > "systemd_unit": > "ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b(a)mds.data.blad > e3n1.eczeqc", > "enabled": true, > "state": "error", > "service_name": "mds.data", > "ports": [], > "ip": null, > "deployed_by": [ > " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102 > a753902f33ee16c26b6cee" > ], > "rank": null, > "rank_generation": null, > "extra_container_args": null, > "extra_entrypoint_args": null, > "memory_request": null, > "memory_limit": null, > "container_id": null, > "container_image_name": " > quay.io/ceph/ceph@sha256:af0c5903e901e329adabe2 > 19dfc8d0c3efc1f05102a753902f33ee16c26b6cee", > "container_image_id": null, > "container_image_digests": null, > "version": null, > "started": null, > "created": "2026-04-16T15:54:13.264224Z", > "deployed": "2026-04-16T15:54:10.870858Z", > "configured": "2026-04-20T17:16:40.499113Z" > } > ] > > > Am Mi., 27. Mai 2026 um 15:07 Uhr schrieb Jacek Rużyczka < > jacek.ruzyczka(a)googlemail.com>: > >> Hi Eugen, >> >> You might need to run 'systemctl reset-failed...' to let systemd start the >>> containers. >> >> >> I've already done that. No use. Even worse: On node #1, Docker no longer >> starts. When trying to restart the daemon, I get errors like this: >> >> docker.service: Failed with result 'core-dump'. >> >> But before you do that, do you have MON logs with an explanation why they >>> refuse to start? >> >> >> Unfortunately no, not even in the syslog. In the meantime, I was able to >> start another MON via Cephadm (because the Docker instance had even deleted >> the image), but now I've got the problem with the one node, where Docker >> refuses to start. >> >> Regarding Ceph images, your cluster uses af0c5903e901 for the Ceph >>> services, what does 'docker images | grep af0c5903e901' show? >> >> >> On the affected node, nothing 'cause the Docker daemon wouldn't even start. >> >> I have the impression that this is a "regular" cephadm cluster >> >> >> True >> >> BTW, when running the test script supplied by the Docker guys >> https://docs.docker.com/engine/daemon/troubleshoot/, I get some warnings: >> >> - Network Drivers: >> - "bridge": >> - sysctl net.ipv4.ip_forward: disabled >> - sysctl net.ipv6.conf.all.forwarding: disabled >> - sysctl net.ipv6.conf.default.forwarding: disabled >> >> On nodes #2 thru #4, net.ipv4.ip_forward is enabled. >> >> Regards >> Jacek Rużyczka >>

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by C U 27 May '26

27 May '26

then my problem was different. v20 did introduce python subprocesses, py03. On 5/27/26 07:13, Iztok Gregori via ceph-users wrote: > Hi! > > On 26/05/26 18:16, C U via ceph-users wrote: >> fqdn on the dashboard needs working certificate validation. >> >> "Squid did cert work inline → bad cert raised crypto.Error → caught >> and rewrapped as ServerConfigException cleanly. Tentacle does cert >> work via a >> subprocess; failures can now come from (a) the fork itself, (b) >> PyO3 init crashing in the child (the documented subinterpreter >> collision), or (c) >> genuine cert errors marshalled back as JSON. When (b) happens, the >> dashboard module's cert provisioning never completes, the module ends up >> half-initialized, and downstream dashboard→RGW calls fail with >> SignatureDoesNotMatch because the dashboard is operating without >> properly loaded >> credentials/sigv4 plumbing." > > In this context you mean the validation of the certificate build > around the FQDN of the RGW VIP, right? > > The certificate in question is a public one, the CA is correctly > installed in the system paths, an "openssl s_client -showcerts > -connect vip.domain.com:443" shows no error (verify: return 1) and has > the correct FQDN. I executed the openssl command within the active MGR > container. > > Is there an other location where I the CA should be (if the problem is > indeed the certificate validation)? > > Cheers > Iztok > > > > > > > > >> >> On 5/23/26 17:54, Eugen Block via ceph-users wrote: >>> Hi, >>> >>> I might be wrong, but I don't think it's a cert issue. If you look >>> in the mgr log, do you see more information that just the >>> SignatureDoesNotMatch error? >>> >>> Could it be mismatching dashboard-rgw-api settings? Have you checked >>> these settings? >>> >>> ceph dashboard get-rgw-api-admin-resource >>> >>> ceph dashboard get-rgw-api-access-key >>> >>> ceph dashboard get-rgw-api-secret-key >>> >>> They should match with: >>> >>> radosgw-admin user info --uid dashboard | jq -r '.keys' >>> >>> But as I wrote, it might something else, I would expect the mgr log >>> to contain more details. >>> >>> Regards >>> Eugen >>> >>> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >>> >>>> Hi to all! >>>> >>>> After upgrading my cluster from squid to tentacle (now on 20.2.1) >>>> I'm getting the following error when I try to access any 'page' in >>>> the Object section of the Ceph Dashboard: >>>> >>>> Error connecting to Object Gateway: RGW REST API failed request >>>> with status code 403 >>>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx0000000b5bf42356' >>>> b'85174-006a0d95b0-308299-eros","HostId":"308299-zone-default"}') >>>> >>>> I'm pretty sure that in Squid it was working (I don't usually >>>> access the Dashboard, but I did after the upgrade to check if >>>> everything is ok). >>>> >>>> The error 'SignatureDoesNotMatch' leads me to believe that there is >>>> a problem somewhere with the SSL certificates. But I put >>>> RGW_API_SSL_VERIFY to false... >>>> >>>> I have a SSL certificate (issued by harica.gr) which has different >>>> SANs including the hostname of all the nodes where RGW daemon is >>>> running (plus a cluster hostname and a wildcard hostname). I used >>>> this certificate for the ingress service and for the rgw daemons. >>>> Everything is done with a spec file using ceph orchestrator. I'm >>>> using the fullchain for the certificates and the the rgw/ingress >>>> services are running without any problem >>>> >>>> Or I'm completely wrong and the problem itself is with the access >>>> permissions (error code 403)? >>>> >>>> Any ideas? >>>> >>>> Thanks >>>> Iztok >>>> >>>> -- >>>> Iztok Gregori >>>> ICT Systems and Services >>>> Elettra - Sincrotrone Trieste S.C.p.A. >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: tentacle 20.2.2 QE validation status
by Vallari Agrawal 27 May '26

27 May '26

nvmeof approved cc: @Aviv Caro <acaro(a)redhat.com> On Tue, May 26, 2026 at 10:44 PM Afreen via dev <dev(a)ceph.io> wrote: > Dashboard approved > > On Tue, May 26, 2026 at 10:00 PM Yuri Weinstein via ceph-users < > ceph-users(a)ceph.io> wrote: > >> Resending because of emails problems >> >> On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> >> wrote: >> >> > Details of this release are summarized here: >> > >> > https://tracker.ceph.com/issues/75878#note-6 >> > >> > Release Notes - TBD >> > LRC upgrade - TBD >> > >> > Seeking approvals/reviews for: >> > >> > smoke: PASSED >> > >> > rados: Radek, Laura, Afreen, Travis >> > rgw: Adam Emerson >> > fs: Venky >> > orch/ceph-volume: Adam King & Guillaume >> > rbd: Ilya >> > krbd: Ilya >> > nvmeof: Aviv, Vallari >> > >> > upgrade: Patrick >> > >> > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD >> > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD >> > >> > perf-basic: PASSED >> > >> > crimson-rados: N/A >> > >> > powercycle: PASSED >> > >> > TIA >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Casey Bodley 27 May '26

27 May '26

On Wed, May 27, 2026 at 6:36 AM Iztok Gregori via ceph-users <ceph-users(a)ceph.io> wrote: > > On 27/05/26 12:17, Eugen Block via ceph-users wrote: > > Hi, > > > > > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > > > >> On 27/05/26 09:36, Eugen Block via ceph-users wrote: > >>> The description in this tracker [0] looks identical, but it is > >>> supposed to be fixed. Do you have rgw_dns_name set? > >> > >> Yes, at certain point I configured client.rgw rgw_dns_name to the FQDN > >> belonging to the virtual IP of the ingress service. > > > > does it mean that it works now? > > No, still doesn't work. To try I've also remove it and then again set > it, but without luck. > > I also tried to call the API endpoint directly with curl using the > dashboard credentials and found that from 'older' versions of curl > (provided for example by Rocky 9) the command gives me the same error > ('SignatureDoesNotMatch'), but when I was using newer one (8.5.0 on > Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer my query. regarding curl versions, i had fixed its sigv4 bugs for curl 8.0.0 with https://github.com/curl/curl/pull/9995 but i believe the dashboard uses its own custom sigv2 implementation in src/pybind/mgr/dashboard/awsauth.py > > Could it be the problem? > > > > >> BTW, how you 'unset' a configuration variable, you just put a empty > >> string or you use the rm option? > >> > >> ceph config set client.rgw rgw_dns_name '' > >> or > >> ceph config rm client.rgw rgw_dns_name > >> ? > > > > That's the correct one (config rm). > > Thank you! I sometimes get lost in the documentation :-) > > Cheers > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Eugen Block 27 May '26

27 May '26

I see what you mean, in our Squid cluster the curl request from within the mgr container fails with the same error. So I should probably expect that we're gonna hit the same dashboard issue once we upgrade to Tentacle, I guess. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > On 27/05/26 12:47, Eugen Block via ceph-users wrote: >> I'm not sure about the curl versions. > > Curl was only a tool and it could be unrelated to the problem I'm > facing, but it was interesting that when I try to query > > # curl "https://rgw.domain.com/admin/metadata/user" --aws-sigv4 > "aws:amz:ZONEGROUP:s3" --user "DASHBOARDKEY:DASHBOARDSECRET" > > it worked fine with newer versions. But the older ones (like the one > shipped with the MGR container) failed with the > 'SignatureDoesNotMatch' error. When I tried with more verbosity I > see that the oldest version is "signing" only 2 headers > (SignedHeaders=host;x-amz-date) and the newer ones 3 > (SignedHeaders=host;x-amz-content-sha256;x-amz-date), but this could > be due to '--aws-sigv4' option in curl. > > >> Did you notice the response from > "C U" the day before yesterday? >> He or she commented: >> >>> fqdn on the dashboard needs working certificate validation. >> >>> "Squid did cert work inline → bad cert raised crypto.Error → >>> caught and rewrapped as ServerConfigException cleanly. Tentacle >>> does cert work via a >> subprocess; failures can now come from (a) the fork itself, (b) >> PyO3 init crashing in the child (the documented subinterpreter >> collision), or (c) >> genuine cert errors marshalled back as JSON. When (b) happens, >> the dashboard module's cert provisioning never completes, the >> module ends up >> half-initialized, and downstream dashboard→RGW calls fail with >> SignatureDoesNotMatch because the dashboard is operating without >> properly loaded >> credentials/sigv4 plumbing." > > I did some tests using openssl s_client from the mgr container to > verify the certificates and I had no problems. Clearly I wasn't > using python, but unless is not using the system paths for > certificates, it shouldn't matter. > > Iztok > >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> >>> On 27/05/26 12:17, Eugen Block via ceph-users wrote: >>>> Hi, >>>> >>>> >>>> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >>>> >>>>> On 27/05/26 09:36, Eugen Block via ceph-users wrote: >>>>>> The description in this tracker [0] looks identical, but it is >>>>>> supposed to be fixed. Do you have rgw_dns_name set? >>>>> >>>>> Yes, at certain point I configured client.rgw rgw_dns_name to >>>>> the FQDN belonging to the virtual IP of the ingress service. >>>> >>>> does it mean that it works now? >>> >>> No, still doesn't work. To try I've also remove it and then again >>> set it, but without luck. >>> >>> I also tried to call the API endpoint directly with curl using the >>> dashboard credentials and found that from 'older' versions of curl >>> (provided for example by Rocky 9) the command gives me the same >>> error ('SignatureDoesNotMatch'), but when I was using newer one >>> (8.5.0 on Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer >>> my query. >>> >>> Could it be the problem? >>> >>>> >>>>> BTW, how you 'unset' a configuration variable, you just put a >>>>> empty string or you use the rm option? >>>>> >>>>> ceph config set client.rgw rgw_dns_name '' >>>>> or >>>>> ceph config rm client.rgw rgw_dns_name >>>>> ? >>>> >>>> That's the correct one (config rm). >>> >>> Thank you! I sometimes get lost in the documentation :-) >>> >>> Cheers >>> Iztok >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 27 May '26

27 May '26

On 27/05/26 12:47, Eugen Block via ceph-users wrote: > I'm not sure about the curl versions. Curl was only a tool and it could be unrelated to the problem I'm facing, but it was interesting that when I try to query # curl "https://rgw.domain.com/admin/metadata/user" --aws-sigv4 "aws:amz:ZONEGROUP:s3" --user "DASHBOARDKEY:DASHBOARDSECRET" it worked fine with newer versions. But the older ones (like the one shipped with the MGR container) failed with the 'SignatureDoesNotMatch' error. When I tried with more verbosity I see that the oldest version is "signing" only 2 headers (SignedHeaders=host;x-amz-date) and the newer ones 3 (SignedHeaders=host;x-amz-content-sha256;x-amz-date), but this could be due to '--aws-sigv4' option in curl. > Did you notice the response from > "C U" the day before yesterday? He or she commented: > >> fqdn on the dashboard needs working certificate validation. > >> "Squid did cert work inline → bad cert raised crypto.Error → caught >> and rewrapped as ServerConfigException cleanly. Tentacle does cert >> work via a > subprocess; failures can now come from (a) the fork itself, (b) PyO3 > init crashing in the child (the documented subinterpreter collision), or > (c) > genuine cert errors marshalled back as JSON. When (b) happens, the > dashboard module's cert provisioning never completes, the module ends up > half-initialized, and downstream dashboard→RGW calls fail with > SignatureDoesNotMatch because the dashboard is operating without > properly loaded > credentials/sigv4 plumbing." I did some tests using openssl s_client from the mgr container to verify the certificates and I had no problems. Clearly I wasn't using python, but unless is not using the system paths for certificates, it shouldn't matter. Iztok > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > >> On 27/05/26 12:17, Eugen Block via ceph-users wrote: >>> Hi, >>> >>> >>> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >>> >>>> On 27/05/26 09:36, Eugen Block via ceph-users wrote: >>>>> The description in this tracker [0] looks identical, but it is >>>>> supposed to be fixed. Do you have rgw_dns_name set? >>>> >>>> Yes, at certain point I configured client.rgw rgw_dns_name to the >>>> FQDN belonging to the virtual IP of the ingress service. >>> >>> does it mean that it works now? >> >> No, still doesn't work. To try I've also remove it and then again set >> it, but without luck. >> >> I also tried to call the API endpoint directly with curl using the >> dashboard credentials and found that from 'older' versions of curl >> (provided for example by Rocky 9) the command gives me the same error >> ('SignatureDoesNotMatch'), but when I was using newer one (8.5.0 on >> Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer my query. >> >> Could it be the problem? >> >>> >>>> BTW, how you 'unset' a configuration variable, you just put a empty >>>> string or you use the rm option? >>>> >>>> ceph config set client.rgw rgw_dns_name '' >>>> or >>>> ceph config rm client.rgw rgw_dns_name >>>> ? >>> >>> That's the correct one (config rm). >> >> Thank you! I sometimes get lost in the documentation :-) >> >> Cheers >> Iztok >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 27 May '26

27 May '26

Hi! On 26/05/26 18:16, C U via ceph-users wrote: > fqdn on the dashboard needs working certificate validation. > > "Squid did cert work inline → bad cert raised crypto.Error → caught and > rewrapped as ServerConfigException cleanly. Tentacle does cert work via a > subprocess; failures can now come from (a) the fork itself, (b) PyO3 > init crashing in the child (the documented subinterpreter collision), or > (c) > genuine cert errors marshalled back as JSON. When (b) happens, the > dashboard module's cert provisioning never completes, the module ends up > half-initialized, and downstream dashboard→RGW calls fail with > SignatureDoesNotMatch because the dashboard is operating without > properly loaded > credentials/sigv4 plumbing." In this context you mean the validation of the certificate build around the FQDN of the RGW VIP, right? The certificate in question is a public one, the CA is correctly installed in the system paths, an "openssl s_client -showcerts -connect vip.domain.com:443" shows no error (verify: return 1) and has the correct FQDN. I executed the openssl command within the active MGR container. Is there an other location where I the CA should be (if the problem is indeed the certificate validation)? Cheers Iztok > > On 5/23/26 17:54, Eugen Block via ceph-users wrote: >> Hi, >> >> I might be wrong, but I don't think it's a cert issue. If you look in >> the mgr log, do you see more information that just the >> SignatureDoesNotMatch error? >> >> Could it be mismatching dashboard-rgw-api settings? Have you checked >> these settings? >> >> ceph dashboard get-rgw-api-admin-resource >> >> ceph dashboard get-rgw-api-access-key >> >> ceph dashboard get-rgw-api-secret-key >> >> They should match with: >> >> radosgw-admin user info --uid dashboard | jq -r '.keys' >> >> But as I wrote, it might something else, I would expect the mgr log to >> contain more details. >> >> Regards >> Eugen >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> >>> Hi to all! >>> >>> After upgrading my cluster from squid to tentacle (now on 20.2.1) I'm >>> getting the following error when I try to access any 'page' in the >>> Object section of the Ceph Dashboard: >>> >>> Error connecting to Object Gateway: RGW REST API failed request with >>> status code 403 >>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx0000000b5bf42356' b'85174-006a0d95b0-308299-eros","HostId":"308299-zone-default"}') >>> >>> I'm pretty sure that in Squid it was working (I don't usually access >>> the Dashboard, but I did after the upgrade to check if everything is >>> ok). >>> >>> The error 'SignatureDoesNotMatch' leads me to believe that there is a >>> problem somewhere with the SSL certificates. But I put >>> RGW_API_SSL_VERIFY to false... >>> >>> I have a SSL certificate (issued by harica.gr) which has different >>> SANs including the hostname of all the nodes where RGW daemon is >>> running (plus a cluster hostname and a wildcard hostname). I used >>> this certificate for the ingress service and for the rgw daemons. >>> Everything is done with a spec file using ceph orchestrator. I'm >>> using the fullchain for the certificates and the the rgw/ingress >>> services are running without any problem >>> >>> Or I'm completely wrong and the problem itself is with the access >>> permissions (error code 403)? >>> >>> Any ideas? >>> >>> Thanks >>> Iztok >>> >>> -- >>> Iztok Gregori >>> ICT Systems and Services >>> Elettra - Sincrotrone Trieste S.C.p.A. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Eugen Block 27 May '26

27 May '26

I'm not sure about the curl versions. Did you notice the response from "C U" the day before yesterday? He or she commented: > fqdn on the dashboard needs working certificate validation. > "Squid did cert work inline → bad cert raised crypto.Error → caught > and rewrapped as ServerConfigException cleanly. Tentacle does cert > work via a subprocess; failures can now come from (a) the fork itself, (b) PyO3 init crashing in the child (the documented subinterpreter collision), or (c) genuine cert errors marshalled back as JSON. When (b) happens, the dashboard module's cert provisioning never completes, the module ends up half-initialized, and downstream dashboard→RGW calls fail with SignatureDoesNotMatch because the dashboard is operating without properly loaded credentials/sigv4 plumbing." Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > On 27/05/26 12:17, Eugen Block via ceph-users wrote: >> Hi, >> >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> >>> On 27/05/26 09:36, Eugen Block via ceph-users wrote: >>>> The description in this tracker [0] looks identical, but it is >>>> supposed to be fixed. Do you have rgw_dns_name set? >>> >>> Yes, at certain point I configured client.rgw rgw_dns_name to the >>> FQDN belonging to the virtual IP of the ingress service. >> >> does it mean that it works now? > > No, still doesn't work. To try I've also remove it and then again > set it, but without luck. > > I also tried to call the API endpoint directly with curl using the > dashboard credentials and found that from 'older' versions of curl > (provided for example by Rocky 9) the command gives me the same > error ('SignatureDoesNotMatch'), but when I was using newer one > (8.5.0 on Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer my > query. > > Could it be the problem? > >> >>> BTW, how you 'unset' a configuration variable, you just put a >>> empty string or you use the rm option? >>> >>> ceph config set client.rgw rgw_dns_name '' >>> or >>> ceph config rm client.rgw rgw_dns_name >>> ? >> >> That's the correct one (config rm). > > Thank you! I sometimes get lost in the documentation :-) > > Cheers > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 27 May '26

27 May '26

On 27/05/26 12:17, Eugen Block via ceph-users wrote: > Hi, > > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > >> On 27/05/26 09:36, Eugen Block via ceph-users wrote: >>> The description in this tracker [0] looks identical, but it is >>> supposed to be fixed. Do you have rgw_dns_name set? >> >> Yes, at certain point I configured client.rgw rgw_dns_name to the FQDN >> belonging to the virtual IP of the ingress service. > > does it mean that it works now? No, still doesn't work. To try I've also remove it and then again set it, but without luck. I also tried to call the API endpoint directly with curl using the dashboard credentials and found that from 'older' versions of curl (provided for example by Rocky 9) the command gives me the same error ('SignatureDoesNotMatch'), but when I was using newer one (8.5.0 on Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer my query. Could it be the problem? > >> BTW, how you 'unset' a configuration variable, you just put a empty >> string or you use the rm option? >> >> ceph config set client.rgw rgw_dns_name '' >> or >> ceph config rm client.rgw rgw_dns_name >> ? > > That's the correct one (config rm). Thank you! I sometimes get lost in the documentation :-) Cheers Iztok

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Eugen Block 27 May '26

27 May '26

Hi, Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > On 27/05/26 09:36, Eugen Block via ceph-users wrote: >> The description in this tracker [0] looks identical, but it is >> supposed to be fixed. Do you have rgw_dns_name set? > > Yes, at certain point I configured client.rgw rgw_dns_name to the > FQDN belonging to the virtual IP of the ingress service. does it mean that it works now? > BTW, how you 'unset' a configuration variable, you just put a empty > string or you use the rm option? > > ceph config set client.rgw rgw_dns_name '' > or > ceph config rm client.rgw rgw_dns_name > ? That's the correct one (config rm). > > >> >> [0] https://tracker.ceph.com/issues/62396 >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> >>> Hi Eugen! >>> >>> >>> On 23/05/26 23:54, Eugen Block via ceph-users wrote: >>>> Hi, >>>> >>>> I might be wrong, but I don't think it's a cert issue. If you >>>> look in the mgr log, do you see more information that just the >>>> SignatureDoesNotMatch error? >>> >>> On further investigations I'm also convinced that is nothing to do >>> with the certificates but more about the permissions of the >>> 'dashboard' user. This is the exception that is thrown from the >>> mgr logs: >>> >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-01.domain.com, port=4443, ssl=True >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-03.domain.com, port=4443, ssl=True >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-02.domain.com, port=4443, ssl=True >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-04.domain.com, port=4443, ssl=True >>>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>>> host=node-05.domain.com, port=4443, ssl=True >>>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>>> [0.005s] [admin] [308.0B] /api/health/snapshot >>>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>>> [0.007s] [admin] [22.0B] /api/prometheus/alertgroup >>>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>>> [0.012s] [admin] [376.0B] /api/multi-cluster/get_config >>>> [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] >>>> [0.028s] [admin] [1.2K] /api/summary >>>> [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 >>>> [dashboard ERROR rgw_client] RGW REST API failed request with >>>> status code 403 >>>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >>>> b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') >>>> Traceback (most recent call last): >>>> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", >>>> line 413, in __init__ >>>> self.userid = self._get_user_id(self.admin_path) if >>>> self.got_keys_from_config \ >>>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, >>>> in func_wrapper >>>> return func( >>>> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", >>>> line 448, in _get_user_id >>>> response = request() >>>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, >>>> in __call__ >>>> resp = self.rest_client.do_request(method, self._gen_path(), params, >>>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, >>>> in do_request >>>> raise RequestException( >>>> dashboard.rest_client.RequestException: RGW REST API failed >>>> request with status code 403 >>>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >>>> b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') >>> >>> >>>> >>>> Could it be mismatching dashboard-rgw-api settings? Have you >>>> checked these settings? >>>> >>>> ceph dashboard get-rgw-api-admin-resource >>> >>> It's the default, 'admin'. I also reset the value with >>> "reset-rgw-api- admin-resource", so technically now is 'unset'. >>> Question what should be the value if not 'admin'? >>> >>>> >>>> ceph dashboard get-rgw-api-access-key >>>> >>>> ceph dashboard get-rgw-api-secret-key >>>> >>>> They should match with: >>>> >>>> radosgw-admin user info --uid dashboard | jq -r '.keys' >>> >>> The values are matching, I'see in both get-rgw-api-secret-key and >>> get- rgw-api-access-key something like this: >>> {'<REALM>': '<(ACCESS}SECRET)_KEY>'} >>> >>>> >>>> But as I wrote, it might something else, I would expect the mgr >>>> log to contain more details. >>>> >>> >>> Indeed, I'll tried to backtrace the error in the code and I see >>> that is failing in the rgw_client.py when is trying to get the >>> user_id "of the user that is used to communicate with the RGW >>> Admin Ops API". I suppose that is the 'admin' user from >>> rgw-api-admin-resource, am I right? >>> >>> I didn't had time till now to investigate further. >>> >>> >>> Thanks! >>> Iztok >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 27 May '26

27 May '26

On 27/05/26 09:36, Eugen Block via ceph-users wrote: > The description in this tracker [0] looks identical, but it is supposed > to be fixed. Do you have rgw_dns_name set? Yes, at certain point I configured client.rgw rgw_dns_name to the FQDN belonging to the virtual IP of the ingress service. BTW, how you 'unset' a configuration variable, you just put a empty string or you use the rm option? ceph config set client.rgw rgw_dns_name '' or ceph config rm client.rgw rgw_dns_name ? > > [0] https://tracker.ceph.com/issues/62396 > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > >> Hi Eugen! >> >> >> On 23/05/26 23:54, Eugen Block via ceph-users wrote: >>> Hi, >>> >>> I might be wrong, but I don't think it's a cert issue. If you look in >>> the mgr log, do you see more information that just the >>> SignatureDoesNotMatch error? >> >> On further investigations I'm also convinced that is nothing to do >> with the certificates but more about the permissions of the >> 'dashboard' user. This is the exception that is thrown from the mgr logs: >> >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-01.domain.com, port=4443, ssl=True >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-03.domain.com, port=4443, ssl=True >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-02.domain.com, port=4443, ssl=True >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-04.domain.com, port=4443, ssl=True >>> [dashboard INFO rgw_client] Found RGW daemon with configuration: >>> host=node-05.domain.com, port=4443, ssl=True >>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>> [0.005s] [admin] [308.0B] /api/health/snapshot >>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>> [0.007s] [admin] [22.0B] /api/prometheus/alertgroup >>> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >>> [0.012s] [admin] [376.0B] /api/multi-cluster/get_config >>> [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] >>> [0.028s] [admin] [1.2K] /api/summary >>> [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 >>> [dashboard ERROR rgw_client] RGW REST API failed request with status >>> code 403 >>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >>> b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') >>> Traceback (most recent call last): >>> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line >>> 413, in __init__ >>> self.userid = self._get_user_id(self.admin_path) if >>> self.got_keys_from_config \ >>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in >>> func_wrapper >>> return func( >>> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line >>> 448, in _get_user_id >>> response = request() >>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, in >>> __call__ >>> resp = self.rest_client.do_request(method, self._gen_path(), params, >>> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, in >>> do_request >>> raise RequestException( >>> dashboard.rest_client.RequestException: RGW REST API failed request >>> with status code 403 >>> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >>> b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') >> >> >>> >>> Could it be mismatching dashboard-rgw-api settings? Have you checked >>> these settings? >>> >>> ceph dashboard get-rgw-api-admin-resource >> >> It's the default, 'admin'. I also reset the value with "reset-rgw-api- >> admin-resource", so technically now is 'unset'. Question what should >> be the value if not 'admin'? >> >>> >>> ceph dashboard get-rgw-api-access-key >>> >>> ceph dashboard get-rgw-api-secret-key >>> >>> They should match with: >>> >>> radosgw-admin user info --uid dashboard | jq -r '.keys' >> >> The values are matching, I'see in both get-rgw-api-secret-key and get- >> rgw-api-access-key something like this: >> {'<REALM>': '<(ACCESS}SECRET)_KEY>'} >> >>> >>> But as I wrote, it might something else, I would expect the mgr log >>> to contain more details. >>> >> >> Indeed, I'll tried to backtrace the error in the code and I see that >> is failing in the rgw_client.py when is trying to get the user_id "of >> the user that is used to communicate with the RGW Admin Ops API". I >> suppose that is the 'admin' user from rgw-api-admin-resource, am I right? >> >> I didn't had time till now to investigate further. >> >> >> Thanks! >> Iztok >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Eugen Block 27 May '26

27 May '26

The description in this tracker [0] looks identical, but it is supposed to be fixed. Do you have rgw_dns_name set? [0] https://tracker.ceph.com/issues/62396 Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi Eugen! > > > On 23/05/26 23:54, Eugen Block via ceph-users wrote: >> Hi, >> >> I might be wrong, but I don't think it's a cert issue. If you look >> in the mgr log, do you see more information that just the >> SignatureDoesNotMatch error? > > On further investigations I'm also convinced that is nothing to do > with the certificates but more about the permissions of the > 'dashboard' user. This is the exception that is thrown from the mgr > logs: > >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-01.domain.com, port=4443, ssl=True >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-03.domain.com, port=4443, ssl=True >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-02.domain.com, port=4443, ssl=True >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-04.domain.com, port=4443, ssl=True >> [dashboard INFO rgw_client] Found RGW daemon with configuration: >> host=node-05.domain.com, port=4443, ssl=True >> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >> [0.005s] [admin] [308.0B] /api/health/snapshot >> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >> [0.007s] [admin] [22.0B] /api/prometheus/alertgroup >> [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] >> [0.012s] [admin] [376.0B] /api/multi-cluster/get_config >> [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] >> [0.028s] [admin] [1.2K] /api/summary >> [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 >> [dashboard ERROR rgw_client] RGW REST API failed request with >> status code 403 >> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >> b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') >> Traceback (most recent call last): >> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line >> 413, in __init__ >> self.userid = self._get_user_id(self.admin_path) if >> self.got_keys_from_config \ >> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in >> func_wrapper >> return func( >> File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line >> 448, in _get_user_id >> response = request() >> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, in __call__ >> resp = self.rest_client.do_request(method, self._gen_path(), params, >> File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, in >> do_request >> raise RequestException( >> dashboard.rest_client.RequestException: RGW REST API failed request >> with status code 403 >> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' >> b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') > > >> >> Could it be mismatching dashboard-rgw-api settings? Have you >> checked these settings? >> >> ceph dashboard get-rgw-api-admin-resource > > It's the default, 'admin'. I also reset the value with > "reset-rgw-api-admin-resource", so technically now is 'unset'. > Question what should be the value if not 'admin'? > >> >> ceph dashboard get-rgw-api-access-key >> >> ceph dashboard get-rgw-api-secret-key >> >> They should match with: >> >> radosgw-admin user info --uid dashboard | jq -r '.keys' > > The values are matching, I'see in both get-rgw-api-secret-key and > get-rgw-api-access-key something like this: > {'<REALM>': '<(ACCESS}SECRET)_KEY>'} > >> >> But as I wrote, it might something else, I would expect the mgr log >> to contain more details. >> > > Indeed, I'll tried to backtrace the error in the code and I see that > is failing in the rgw_client.py when is trying to get the user_id > "of the user that is used to communicate with the RGW Admin Ops > API". I suppose that is the 'admin' user from > rgw-api-admin-resource, am I right? > > I didn't had time till now to investigate further. > > > Thanks! > Iztok > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

NVMe-oF (14 msgs, 4 threads) is barely on the radar
by C U 27 May '26

27 May '26

did a little research into nvmeof gateway. tldr, more write throughput. protocol translation overhead, not a concern. mon managed gateway failover, stable. https://github.com/tu503/nvmeof-gateway regards,

1 0

Re: Cluster Dead After Network Failure. Connection Timeout.
by Eugen Block 26 May '26

26 May '26

Hi, this sound a bit like a "classic" quorum loss, resulting in cascading daemon failures. There seems to be one surviving MON (blade3n2), that could be a good starting point for disaster recovery by reducing the monmap to this one MON. This should give you back qourum and a working cluster. You might need to run 'systemctl reset-failed...' to let systemd start the containers. Although the product has been discontinued, the section of the SUSE docs [0] is still relevant ("Restoring the MONs quorum"). Back up the mon store of each node before you do this, just in case. The procedure itself has worked many times for me, but maybe there's an easier way, especially if you're not too familiar with cephadm or this procedure. But before you do that, do you have MON logs with an explanation why they refuse to start? Regarding Ceph images, your cluster uses af0c5903e901 for the Ceph services, what does 'docker images | grep af0c5903e901' show? I doubt it's "ceph-mon:latest", at least I haven't seen those in use with cephadm (I have the impression that this is a "regular" cephadm cluster). Regards, Eugen [0] https://documentation.suse.com/pt-br/ses/7.1/html/ses-all/bp-troubleshootin… Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > Hi, > > The network my 4-node cluster uses broke down after a driver issue a week > ago. Now, as the network resumed normal operation, but my Ceph 19 cluster > first said HEALTH_WARN and informed me of a lengthy recovery process, but > some one hour later, I only found this error message: > > mixtile@blade3n1:~$ sudo ceph -s > [sudo] password for mixtile: > 2026-05-25T13:35:51.685+0200 ffff9701f180 0 monclient(hunting): > authenticate timed out after 300 > [errno 110] RADOS timed out (error connecting to the cluster) > > A restart of all nodes did *not* help. Even worse: The Docker containers > with the various processes (mon, mrg, crash,…) started disappearing one by > one! Here is what remained: > > mixtile@blade3n1:~$ docker ps -a > CONTAINER ID IMAGE COMMAND > CREATED STATUS PORTS NAMES > 2039b18ba392 quay.io/prometheus/node-exporter:v1.7.0 > "/bin/node_exporter …" 47 minutes ago Up 47 minutes > ceph-8aad3073-39a1-11f1-b > f6e-f2704a1efa9b-node-exporter-blade3n1 > 16cb4a6822f2 af0c5903e901 > "/usr/bin/ceph-crash…" 47 minutes ago Up 47 > minutes ceph-8aad3073-39a1-11f1-b > f6e-f2704a1efa9b-crash-blade3n1 > > mixtile@blade3n2:~$ docker ps -a > CONTAINER ID IMAGE COMMAND > CREATED STATUS PORTS NAMES > 42361a694abf quay.io/prometheus/prometheus:v2.51.0 "/bin/prometheus > --c…" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-prometheus-blade3n2 > 88b085d000a8 quay.io/prometheus/node-exporter:v1.7.0 > "/bin/node_exporter …" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-node-exporter-blade3n2 > 7bb17808bdd8 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager > -…" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-alertmanager-blade3n2 > a43cfe36da29 quay.io/ceph/grafana:10.4.0 "/run.sh" > 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-grafana-blade3n2 > a95140b707ee af0c5903e901 > "/usr/bin/ceph-crash…" 2 hours ago Up 2 > hours ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-crash-blade3n2 > 38d4ca4035b0 af0c5903e901 "/usr/bin/ceph-mon > -…" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-mon-blade3n2 > > mixtile@blade3n3:~$ docker ps -a > CONTAINER ID IMAGE COMMAND > CREATED STATUS PORTS NAMES > d664dfe30bd8 af0c5903e901 > "/usr/bin/ceph-crash…" 2 hours ago Up 2 > hours ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-crash-blade3n3 > a64ac00dc28b quay.io/prometheus/node-exporter:v1.7.0 > "/bin/node_exporter …" 2 hours ago Up 2 hours > ceph-8aad3073-39a1-11f1-bf6 > e-f2704a1efa9b-node-exporter-blade3n3 > f7de98403a10 netdata/netdata "/usr/sbin/run.sh" > > mixtile@blade3n4:~$ docker ps -a > CONTAINER ID IMAGE COMMAND > CREATED STATUS PORTS NAMES > d437cec7d6bf quay.io/prometheus/node-exporter:v1.7.0 > "/bin/node_exporter …" 54 minutes ago Up 53 minutes > ceph-8aad3073-39a1-11f1-b > f6e-f2704a1efa9b-node-exporter-blade3n4 > c6d5ac595857 af0c5903e901 > "/usr/bin/ceph-crash…" 54 minutes ago Up 53 > minutes ceph-8aad3073-39a1-11f1-b > f6e-f2704a1efa9b-crash-blade3n4 > > As you can see, there are much less Ceph-related processes than expected. > The rest hasn't only crashed: In fact, the corresponding images have also > disappeared! Pulling the missing containers didn't work: > > mixtile@blade3n1:~$ docker run ceph-mon > Unable to find image 'ceph-mon:latest' locally > docker: Error response from daemon: pull access denied for ceph-mon, > repository does not exist or may require 'docker login' > > This is my exact system and Ceph version BTW: > > mixtile@blade3n1:~$ ceph -v > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > (stable) > > mixtile@blade3n1:~$ uname -a > Linux blade3n1 6.1.0-1027-rockchip #27 SMP Sun Apr 27 01:54:34 UTC 2025 > aarch64 aarch64 aarch64 GNU/Linux > > The drives my data are stored on seem to be still there (as lsblk said), > but as all most OSD processes are gone, I can no longer access them. I've > got four hosts, of which #1 is the admin node. #2 also hosts Ganesha NFS > for external clients. > > So: What can I do to bring my cluster back to life without endangering my > data? Thank you. > > Kind regards > Jacek Rużyczka > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph Days Seattle this week
by Anthony Middleton 26 May '26

26 May '26

Hey Ceph Friends! If you're in the Seattle area this week, please note that Ceph Days Seattle is this Thursday. If you haven't secured your ticket yet, you're still in luck! Check out the event schedule and ticket information here: https://t.ly/TicketsSeattleCeph. Please spread the word within your circles and help me make this Ceph Day a success! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Cluster Dead After Network Failure. Connection Timeout.
by Jacek Rużyczka 26 May '26

26 May '26

Hi, The network my 4-node cluster uses broke down after a driver issue a week ago. Now, as the network resumed normal operation, but my Ceph 19 cluster first said HEALTH_WARN and informed me of a lengthy recovery process, but some one hour later, I only found this error message: mixtile@blade3n1:~$ sudo ceph -s [sudo] password for mixtile: 2026-05-25T13:35:51.685+0200 ffff9701f180 0 monclient(hunting): authenticate timed out after 300 [errno 110] RADOS timed out (error connecting to the cluster) A restart of all nodes did *not* help. Even worse: The Docker containers with the various processes (mon, mrg, crash,…) started disappearing one by one! Here is what remained: mixtile@blade3n1:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2039b18ba392 quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 47 minutes ago Up 47 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-node-exporter-blade3n1 16cb4a6822f2 af0c5903e901 "/usr/bin/ceph-crash…" 47 minutes ago Up 47 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-crash-blade3n1 mixtile@blade3n2:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 42361a694abf quay.io/prometheus/prometheus:v2.51.0 "/bin/prometheus --c…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-prometheus-blade3n2 88b085d000a8 quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-node-exporter-blade3n2 7bb17808bdd8 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager -…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-alertmanager-blade3n2 a43cfe36da29 quay.io/ceph/grafana:10.4.0 "/run.sh" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-grafana-blade3n2 a95140b707ee af0c5903e901 "/usr/bin/ceph-crash…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-crash-blade3n2 38d4ca4035b0 af0c5903e901 "/usr/bin/ceph-mon -…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-mon-blade3n2 mixtile@blade3n3:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d664dfe30bd8 af0c5903e901 "/usr/bin/ceph-crash…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-crash-blade3n3 a64ac00dc28b quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-node-exporter-blade3n3 f7de98403a10 netdata/netdata "/usr/sbin/run.sh" mixtile@blade3n4:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d437cec7d6bf quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 54 minutes ago Up 53 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-node-exporter-blade3n4 c6d5ac595857 af0c5903e901 "/usr/bin/ceph-crash…" 54 minutes ago Up 53 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-crash-blade3n4 As you can see, there are much less Ceph-related processes than expected. The rest hasn't only crashed: In fact, the corresponding images have also disappeared! Pulling the missing containers didn't work: mixtile@blade3n1:~$ docker run ceph-mon Unable to find image 'ceph-mon:latest' locally docker: Error response from daemon: pull access denied for ceph-mon, repository does not exist or may require 'docker login' This is my exact system and Ceph version BTW: mixtile@blade3n1:~$ ceph -v ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) mixtile@blade3n1:~$ uname -a Linux blade3n1 6.1.0-1027-rockchip #27 SMP Sun Apr 27 01:54:34 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux The drives my data are stored on seem to be still there (as lsblk said), but as all most OSD processes are gone, I can no longer access them. I've got four hosts, of which #1 is the admin node. #2 also hosts Ganesha NFS for external clients. So: What can I do to bring my cluster back to life without endangering my data? Thank you. Kind regards Jacek Rużyczka

1 0

Ceph Day Seattle 2026: Still time to register and attend!
by Anthony D'Atri 26 May '26

26 May '26

Ceph Day Seattle 2026 happens this Thursday May 28. https://ceph.io/en/community/events/2026/ceph-days-seattle/ Register and attend for all the latest squiggly goodness. Proxmox-fu! CephFS-fu! RGW-fu! Anthony-Bob sez check it out. -- Anthony D'Atri, Ceph Ambassador and Master of Tentacles

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 26 May '26

26 May '26

Hi Eugen! On 23/05/26 23:54, Eugen Block via ceph-users wrote: > Hi, > > I might be wrong, but I don't think it's a cert issue. If you look in > the mgr log, do you see more information that just the > SignatureDoesNotMatch error? On further investigations I'm also convinced that is nothing to do with the certificates but more about the permissions of the 'dashboard' user. This is the exception that is thrown from the mgr logs: > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-01.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-03.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-02.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-04.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-05.domain.com, port=4443, ssl=True > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.005s] [admin] [308.0B] /api/health/snapshot > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.007s] [admin] [22.0B] /api/prometheus/alertgroup > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.012s] [admin] [376.0B] /api/multi-cluster/get_config > [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] [0.028s] [admin] [1.2K] /api/summary > [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 > [dashboard ERROR rgw_client] RGW REST API failed request with status code 403 > (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' > b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') > Traceback (most recent call last): > File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 413, in __init__ > self.userid = self._get_user_id(self.admin_path) if self.got_keys_from_config \ > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in func_wrapper > return func( > File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 448, in _get_user_id > response = request() > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, in __call__ > resp = self.rest_client.do_request(method, self._gen_path(), params, > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, in do_request > raise RequestException( > dashboard.rest_client.RequestException: RGW REST API failed request with status code 403 > (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' > b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') > > Could it be mismatching dashboard-rgw-api settings? Have you checked > these settings? > > ceph dashboard get-rgw-api-admin-resource It's the default, 'admin'. I also reset the value with "reset-rgw-api-admin-resource", so technically now is 'unset'. Question what should be the value if not 'admin'? > > ceph dashboard get-rgw-api-access-key > > ceph dashboard get-rgw-api-secret-key > > They should match with: > > radosgw-admin user info --uid dashboard | jq -r '.keys' The values are matching, I'see in both get-rgw-api-secret-key and get-rgw-api-access-key something like this: {'<REALM>': '<(ACCESS}SECRET)_KEY>'} > > But as I wrote, it might something else, I would expect the mgr log to > contain more details. > Indeed, I'll tried to backtrace the error in the code and I see that is failing in the rgw_client.py when is trying to get the user_id "of the user that is used to communicate with the RGW Admin Ops API". I suppose that is the 'admin' user from rgw-api-admin-resource, am I right? I didn't had time till now to investigate further. Thanks! Iztok

1 2

Re: Looking for recommendations on Ceph node specifications
by Tony Liu 26 May '26

26 May '26

It depends on how RGW pools are configured. https://docs.ceph.com/en/squid/radosgw/pools/ There are two layers, OSD and pool. OSD can be 1) pure HDD, 2) pure SSD, 3) HDD with db on SSD. By setting crush rule, you may choose which OSD is for which pool, like #1 for rgw.data and #2 for rgw.index. Tony ________________________________________ From: Adam Prycki via ceph-users <ceph-users(a)ceph.io> Sent: May 26, 2026 08:57 AM To: Tony Liu; Anthony D'Atri Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Looking for recommendations on Ceph node specifications In the past we have deployed HDD OSDs (without SSD block.db) and SSD OSDs for S3 cluster. It worked fine. From our understanding, on a ceph consisting of HDDs with SSD block.db most of RGW meta (index shards) will end up on SSD block.db. So, if we can afford it, HDD+block.db setup for S3 would be better for general performance and recovery speeds. Is this assumption correct? Best regards Adam Prycki On 19/05/2026 18:44, Tony Liu wrote: > For RBD on HDD, with DB on SSD (typically shared by multiple HDDs), getting better write performance, > but SSD will be worn out fast. For S3, index on SSD, data and DB on HDD, it will work fine. For density, > it will be fine normally, but in case of failure or maintenance, recovery on dense HDD will be very slow. > For EC, 8+3 is fine on HDD without performance requirement. We tried 6+3 on NVMe, CPU usage is very > high which affects performance. For networking, 2x50G or 2x100G is too much for HDD. 2x25G is sufficient. > > > Tony > ________________________________________ > From: Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> > Sent: May 19, 2026 08:45 AM > To: Adam Prycki > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Looking for recommendations on Ceph node specifications > >>> . >> >> At least 16 or 32 nodes >> in 16 racks. >> Erasure coding 8+3 with failure domain on rack level. >> > Ack. > > >> We initially selected 8+3 over 4+2 because we expect rebuilds to take very long with nodes this big and we don't want to loose redundancy > > Fair enough. You get more nines with m=3 for sure, though the wider profile itself will mean slower scrubs and recovery. I suspect you set mon_osd_down_out_subtree_limit? > >> >>>> >>>> >>>> Splitting JBOD logically into 2 servers isn't an issue for use because we will replicate data on rack level and not host level. >>>> >>>> >>>> Common specifications for all variants >>>> >>>> 5-6GB of RAM per 1 HDD >>> Plus more for mons and other daemons? Especially MDS? >> >> Other daemons will be on some dedicated non-storage servers. > > Ack. Had to ask. > >> We aim for low RAM/HDD on storage nodes. Other daemons won't fit there. >> >>>> 2% of HDD capacity in NVMe devices for block.db (or none) >>>> 2x 50Gb or 2x 100Gb Ethernet per server (active-backup bonded interfaces) >>>> (CPU per OSD to be determined) >>>> >>>> >>>> Variant A1 is very unlikely to happen but we are curious what network interface speeds would you suggest for so many HDDs in one node. >>> 100GE bonded at the least. Depends on your workload. >>>> >>>> Variant A2 is the most likely the one we will choose for large deployment. >>>> >>>> Variant B1/B2 for smaller deployments. >>>> >>>> Does anyone of you run ceph on similar setups? Did you find any pitfall with it? >>>> >>>> What are your minimal recommendations for network speed per HDD, cpu per HDD, etc? >>>> >>>> In our experience most of our servers, even in large clusters, never max out the network interfaces or CPUs. We almost never rebuild or rebalance whole servers. 27 HDD nodes of our biggest CephFS cluster with EC usually have only 2-3Gbps of network traffic. >>> Your workload is archival? >> >> Yes, mostly archival. >> We have big demand for S3 and CephFS. >> But we may move to pure s3 cluster in the future. >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: tentacle 20.2.2 QE validation status
by Afreen 26 May '26

26 May '26

Dashboard approved On Tue, May 26, 2026 at 10:00 PM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Resending because of emails problems > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/75878#note-6 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya > > nvmeof: Aviv, Vallari > > > > upgrade: Patrick > > > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > > > perf-basic: PASSED > > > > crimson-rados: N/A > > > > powercycle: PASSED > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 26 May '26

26 May '26

Hi Eugen! On 23/05/26 23:54, Eugen Block via ceph-users wrote: > Hi, > > I might be wrong, but I don't think it's a cert issue. If you look in > the mgr log, do you see more information that just the > SignatureDoesNotMatch error? On further investigations I'm also convinced that is nothing to do with the certificates but more about the permissions of the 'dashboard' user. This is the exception that is thrown from the mgr logs: > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-01.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-03.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-02.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-04.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-05.domain.com, port=4443, ssl=True > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.005s] [admin] [308.0B] /api/health/snapshot > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.007s] [admin] [22.0B] /api/prometheus/alertgroup > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.012s] [admin] [376.0B] /api/multi-cluster/get_config > [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] [0.028s] [admin] [1.2K] /api/summary > [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 > [dashboard ERROR rgw_client] RGW REST API failed request with status code 403 > (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' > b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') > Traceback (most recent call last): > File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 413, in __init__ > self.userid = self._get_user_id(self.admin_path) if self.got_keys_from_config \ > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in func_wrapper > return func( > File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 448, in _get_user_id > response = request() > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, in __call__ > resp = self.rest_client.do_request(method, self._gen_path(), params, > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, in do_request > raise RequestException( > dashboard.rest_client.RequestException: RGW REST API failed request with status code 403 > (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' > b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') > > Could it be mismatching dashboard-rgw-api settings? Have you checked > these settings? > > ceph dashboard get-rgw-api-admin-resource It's the default, 'admin'. I also reset the value with "reset-rgw-api-admin-resource", so technically now is 'unset'. Question what should be the value if not 'admin'? > > ceph dashboard get-rgw-api-access-key > > ceph dashboard get-rgw-api-secret-key > > They should match with: > > radosgw-admin user info --uid dashboard | jq -r '.keys' The values are matching, I'see in both get-rgw-api-secret-key and get-rgw-api-access-key something like this: {'<REALM>': '<(ACCESS}SECRET)_KEY>'} > > But as I wrote, it might something else, I would expect the mgr log to > contain more details. > Indeed, I'll tried to backtrace the error in the code and I see that is failing in the rgw_client.py when is trying to get the user_id "of the user that is used to communicate with the RGW Admin Ops API". I suppose that is the 'admin' user from rgw-api-admin-resource, am I right? I didn't had time till now to investigate further. Thanks! Iztok

1 0

otel, rgw -> bluestore
by C U 26 May '26

26 May '26

did some stuff with claude, maybe it's interesting if your thing is otel tracing rgw -> bluestore. https://github.com/tu503/warp-otel/blob/main/README.md regards,

1 0

CSC minutes 2026-05-25
by Venky Shankar 26 May '26

26 May '26

- Low turnout due to the US holiday; the agenda moved to the CSC meeting on 8th June. Radek discussed only one item (see below) - [Radek] Umbrella release update Blocker: The fix is in progress and currently being retested. See https://github.com/ceph/ceph/pull/66459 for details. -- Cheers, Venky

1 0

Re: Looking for recommendations on Ceph node specifications
by Tony Liu 25 May '26

25 May '26

For RBD on HDD, with DB on SSD (typically shared by multiple HDDs), getting better write performance, but SSD will be worn out fast. For S3, index on SSD, data and DB on HDD, it will work fine. For density, it will be fine normally, but in case of failure or maintenance, recovery on dense HDD will be very slow. For EC, 8+3 is fine on HDD without performance requirement. We tried 6+3 on NVMe, CPU usage is very high which affects performance. For networking, 2x50G or 2x100G is too much for HDD. 2x25G is sufficient. Tony ________________________________________ From: Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> Sent: May 19, 2026 08:45 AM To: Adam Prycki Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Looking for recommendations on Ceph node specifications >> . > > At least 16 or 32 nodes > in 16 racks. > Erasure coding 8+3 with failure domain on rack level. > Ack. > We initially selected 8+3 over 4+2 because we expect rebuilds to take very long with nodes this big and we don't want to loose redundancy Fair enough. You get more nines with m=3 for sure, though the wider profile itself will mean slower scrubs and recovery. I suspect you set mon_osd_down_out_subtree_limit? > >>> >>> >>> Splitting JBOD logically into 2 servers isn't an issue for use because we will replicate data on rack level and not host level. >>> >>> >>> Common specifications for all variants >>> >>> 5-6GB of RAM per 1 HDD >> Plus more for mons and other daemons? Especially MDS? > > Other daemons will be on some dedicated non-storage servers. Ack. Had to ask. > We aim for low RAM/HDD on storage nodes. Other daemons won't fit there. > >>> 2% of HDD capacity in NVMe devices for block.db (or none) >>> 2x 50Gb or 2x 100Gb Ethernet per server (active-backup bonded interfaces) >>> (CPU per OSD to be determined) >>> >>> >>> Variant A1 is very unlikely to happen but we are curious what network interface speeds would you suggest for so many HDDs in one node. >> 100GE bonded at the least. Depends on your workload. >>> >>> Variant A2 is the most likely the one we will choose for large deployment. >>> >>> Variant B1/B2 for smaller deployments. >>> >>> Does anyone of you run ceph on similar setups? Did you find any pitfall with it? >>> >>> What are your minimal recommendations for network speed per HDD, cpu per HDD, etc? >>> >>> In our experience most of our servers, even in large clusters, never max out the network interfaces or CPUs. We almost never rebuild or rebalance whole servers. 27 HDD nodes of our biggest CephFS cluster with EC usually have only 2-3Gbps of network traffic. >> Your workload is archival? > > Yes, mostly archival. > We have big demand for S3 and CephFS. > But we may move to pure s3 cluster in the future. > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Re: Looking for recommendations on Ceph node specifications
by Anthony D'Atri 25 May '26

25 May '26

> We have 4 potential variants for OSD nodes. > > Variant A1 > ~100 HDD JOBD + x86_64 server with block.db NVMe > > > Variant A2 > ~100 HDD JOBD + 2x x86_64 server with block.db NVMe > JBOD split in half, each node gets 50 HDDs > > > Variant B1 > ~60 HDD JOBD + x86_64 server with block.db NVMe > > > Variant B2 > ~60 HDD JOBD + 2x x86_64 server with block.db NVMe > JBOD split in half, each node gets 30 HDDs How big of a cluster are you planning? Nodes that dense can present various problems, such that of your variants I would reluctantly pick B2. > > > Splitting JBOD logically into 2 servers isn't an issue for use because we will replicate data on rack level and not host level. > > > Common specifications for all variants > > 5-6GB of RAM per 1 HDD Plus more for mons and other daemons? Especially MDS? > 2% of HDD capacity in NVMe devices for block.db (or none) > 2x 50Gb or 2x 100Gb Ethernet per server (active-backup bonded interfaces) > (CPU per OSD to be determined) > > > Variant A1 is very unlikely to happen but we are curious what network interface speeds would you suggest for so many HDDs in one node. 100GE bonded at the least. Depends on your workload. > > Variant A2 is the most likely the one we will choose for large deployment. > > Variant B1/B2 for smaller deployments. > > Does anyone of you run ceph on similar setups? Did you find any pitfall with it? > > What are your minimal recommendations for network speed per HDD, cpu per HDD, etc? > > In our experience most of our servers, even in large clusters, never max out the network interfaces or CPUs. We almost never rebuild or rebalance whole servers. 27 HDD nodes of our biggest CephFS cluster with EC usually have only 2-3Gbps of network traffic. Your workload is archival? > > Best regards > Adam Prycki > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 4

Ceph Day Seattle 2026: Still time to register and attend!
by Anthony D'Atri 25 May '26

25 May '26

Ceph Day Seattle 2026 happens this Thursday May 28. https://ceph.io/en/community/events/2026/ceph-days-seattle/ Register and attend for all the latest squiggly goodness. Proxmox-fu! CephFS-fu! RGW-fu! Anthony-Bob sez check it out. -- Anthony D'Atri, Ceph Ambassador and Master of Tentacles

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by C U 25 May '26

25 May '26

fqdn on the dashboard needs working certificate validation. "Squid did cert work inline → bad cert raised crypto.Error → caught and rewrapped as ServerConfigException cleanly. Tentacle does cert work via a subprocess; failures can now come from (a) the fork itself, (b) PyO3 init crashing in the child (the documented subinterpreter collision), or (c) genuine cert errors marshalled back as JSON. When (b) happens, the dashboard module's cert provisioning never completes, the module ends up half-initialized, and downstream dashboard→RGW calls fail with SignatureDoesNotMatch because the dashboard is operating without properly loaded credentials/sigv4 plumbing." On 5/23/26 17:54, Eugen Block via ceph-users wrote: > Hi, > > I might be wrong, but I don't think it's a cert issue. If you look in > the mgr log, do you see more information that just the > SignatureDoesNotMatch error? > > Could it be mismatching dashboard-rgw-api settings? Have you checked > these settings? > > ceph dashboard get-rgw-api-admin-resource > > ceph dashboard get-rgw-api-access-key > > ceph dashboard get-rgw-api-secret-key > > They should match with: > > radosgw-admin user info --uid dashboard | jq -r '.keys' > > But as I wrote, it might something else, I would expect the mgr log to > contain more details. > > Regards > Eugen > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > >> Hi to all! >> >> After upgrading my cluster from squid to tentacle (now on 20.2.1) I'm >> getting the following error when I try to access any 'page' in the >> Object section of the Ceph Dashboard: >> >> Error connecting to Object Gateway: RGW REST API failed request with >> status code 403 >> (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx0000000b5bf42356' >> b'85174-006a0d95b0-308299-eros","HostId":"308299-zone-default"}') >> >> I'm pretty sure that in Squid it was working (I don't usually access >> the Dashboard, but I did after the upgrade to check if everything is >> ok). >> >> The error 'SignatureDoesNotMatch' leads me to believe that there is a >> problem somewhere with the SSL certificates. But I put >> RGW_API_SSL_VERIFY to false... >> >> I have a SSL certificate (issued by harica.gr) which has different >> SANs including the hostname of all the nodes where RGW daemon is >> running (plus a cluster hostname and a wildcard hostname). I used >> this certificate for the ingress service and for the rgw daemons. >> Everything is done with a spec file using ceph orchestrator. I'm >> using the fullchain for the certificates and the the rgw/ingress >> services are running without any problem >> >> Or I'm completely wrong and the problem itself is with the access >> permissions (error code 403)? >> >> Any ideas? >> >> Thanks >> Iztok >> >> -- >> Iztok Gregori >> ICT Systems and Services >> Elettra - Sincrotrone Trieste S.C.p.A. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Eugen Block 23 May '26

23 May '26

Hi, I might be wrong, but I don't think it's a cert issue. If you look in the mgr log, do you see more information that just the SignatureDoesNotMatch error? Could it be mismatching dashboard-rgw-api settings? Have you checked these settings? ceph dashboard get-rgw-api-admin-resource ceph dashboard get-rgw-api-access-key ceph dashboard get-rgw-api-secret-key They should match with: radosgw-admin user info --uid dashboard | jq -r '.keys' But as I wrote, it might something else, I would expect the mgr log to contain more details. Regards Eugen Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi to all! > > After upgrading my cluster from squid to tentacle (now on 20.2.1) > I'm getting the following error when I try to access any 'page' in > the Object section of the Ceph Dashboard: > > Error connecting to Object Gateway: RGW REST API failed request with > status code 403 > (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx0000000b5bf42356' > b'85174-006a0d95b0-308299-eros","HostId":"308299-zone-default"}') > > I'm pretty sure that in Squid it was working (I don't usually access > the Dashboard, but I did after the upgrade to check if everything is > ok). > > The error 'SignatureDoesNotMatch' leads me to believe that there is > a problem somewhere with the SSL certificates. But I put > RGW_API_SSL_VERIFY to false... > > I have a SSL certificate (issued by harica.gr) which has different > SANs including the hostname of all the nodes where RGW daemon is > running (plus a cluster hostname and a wildcard hostname). I used > this certificate for the ingress service and for the rgw daemons. > Everything is done with a spec file using ceph orchestrator. I'm > using the fullchain for the certificates and the the rgw/ingress > services are running without any problem > > Or I'm completely wrong and the problem itself is with the access > permissions (error code 403)? > > Any ideas? > > Thanks > Iztok > > -- > Iztok Gregori > ICT Systems and Services > Elettra - Sincrotrone Trieste S.C.p.A. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph User + Dev Meetup: Help Us Find a Meetup Time – Quick Survey
by Anthony Middleton 22 May '26

22 May '26

Hey Ceph Friend! During this week's Ceph User + Developer Monthly Meetup, we discussed possibly moving our meeting time to a different time to serve participants across time zones. If you attend (or have wanted to attend but couldn't), we'd love your input! Please take a moment to fill out this short survey: https://forms.gle/g2PxYxC87PjKDJyu5. The survey also includes an option to express interest in leading a meetup in your time zone. If that's something you'd like to explore, let us know! For questions, feel free to reach out to me anytime. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

May Ceph Science User Group Virtual Meeting
by Mattia 21 May '26

21 May '26

Dear all, We will be having a Ceph science/research/big cluster call on Tuesday May 26th as per ceph.io calendar (https://ceph.io/en/community/meetups/) We will start our meeting with Theofilos Mouratidis from switch.ch presenting "Switch Cloud Portal S3: a look into the future S3 service and it's current status" This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, ... If you are interested in giving a short talk on one of the next meetings (it can be a status report, an introduction of your infra/ceph usecase or anything else of interest) you are welcome to reach out! If you would like to discuss something specific you can add it to the pad linked below. Virtual event details: Tue 26 May 2026 PDT 7:00 – 8:00 UTC 14:00 - 15:00 CEST 16:00 - 17:00 Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_20260526 Pad Index: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ See you there! Mattia

1 0

Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 20 May '26

20 May '26

Hi to all! After upgrading my cluster from squid to tentacle (now on 20.2.1) I'm getting the following error when I try to access any 'page' in the Object section of the Ceph Dashboard: Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx0000000b5bf42356' b'85174-006a0d95b0-308299-eros","HostId":"308299-zone-default"}') I'm pretty sure that in Squid it was working (I don't usually access the Dashboard, but I did after the upgrade to check if everything is ok). The error 'SignatureDoesNotMatch' leads me to believe that there is a problem somewhere with the SSL certificates. But I put RGW_API_SSL_VERIFY to false... I have a SSL certificate (issued by harica.gr) which has different SANs including the hostname of all the nodes where RGW daemon is running (plus a cluster hostname and a wildcard hostname). I used this certificate for the ingress service and for the rgw daemons. Everything is done with a spec file using ceph orchestrator. I'm using the fullchain for the certificates and the the rgw/ingress services are running without any problem Or I'm completely wrong and the problem itself is with the access permissions (error code 403)? Any ideas? Thanks Iztok -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A.

1 0

Experience with D3N RGW Data Cache
by Eugen Block 19 May '26

19 May '26

Hi, I can't recall having read about it on this list, searching the archives only shows rgw logs where d3n is NOT enabled. Has anyone some experience to share with D3N Cache [0]? I'd be interested if people use it in production and if it meets their expectation. After reading [0] I feel like it could help if you have a HDD pool for RGW data but need to access frequently used objects fast. Since the old "cache tier" has been deprecated this could be a replacement. Any comments are appreciated! Thanks! Eugen [0] https://docs.ceph.com/en/latest/radosgw/d3n_datacache/

1 0

Looking for recommendations on Ceph node specifications
by Adam Prycki 19 May '26

19 May '26

Hello, we are preparing to buy hardware for our new ceph clusters. Because of current economic conditions we are considering buying nodes consisting of JBODs + servers. We have 4 potential variants for OSD nodes. Variant A1 ~100 HDD JOBD + x86_64 server with block.db NVMe Variant A2 ~100 HDD JOBD + 2x x86_64 server with block.db NVMe JBOD split in half, each node gets 50 HDDs Variant B1 ~60 HDD JOBD + x86_64 server with block.db NVMe Variant B2 ~60 HDD JOBD + 2x x86_64 server with block.db NVMe JBOD split in half, each node gets 30 HDDs Splitting JBOD logically into 2 servers isn't an issue for use because we will replicate data on rack level and not host level. Common specifications for all variants 5-6GB of RAM per 1 HDD 2% of HDD capacity in NVMe devices for block.db (or none) 2x 50Gb or 2x 100Gb Ethernet per server (active-backup bonded interfaces) (CPU per OSD to be determined) Variant A1 is very unlikely to happen but we are curious what network interface speeds would you suggest for so many HDDs in one node. Variant A2 is the most likely the one we will choose for large deployment. Variant B1/B2 for smaller deployments. Does anyone of you run ceph on similar setups? Did you find any pitfall with it? What are your minimal recommendations for network speed per HDD, cpu per HDD, etc? In our experience most of our servers, even in large clusters, never max out the network interfaces or CPUs. We almost never rebuild or rebalance whole servers. 27 HDD nodes of our biggest CephFS cluster with EC usually have only 2-3Gbps of network traffic. Best regards Adam Prycki

1 0

Re: NVMe/TCP + VMware
by Alan Murrell 18 May '26

18 May '26

Hi Aviv, I have just upgraded my cluster to tentacle and will try it again and report back here. Regards, Alan On Sun, 2026-05-17 at 05:49 +0000, Aviv Caro via ceph-users wrote: > Hi, > > Ceph officially supports nvme/tcp only since tentacle release. In > squid some important code is missing. We can discuss in more in > #nvmeof channel in ceph workspace in slack. > > Aviv > > -----Original Message----- > From: Alan Murrell via ceph-users <ceph-users(a)ceph.io> > Sent: Saturday, 16 May 2026 3:18 > To: ceph-users(a)ceph.io > Subject: [EXTERNAL] [ceph-users] NVMe/TCP + VMware > > Hello, > > I am hoping someone might be able to help, or point me to a > "bulletproof" step-by-step howto of connecting my Ceph 19.2.3 cluster > to my VMware 8.0 cluster as I want to do some testing. > > I have spent the last day and half with Gemini trying to get this to > work, and have been able to get as far as VMware seemingly to see my > RBD images that I am exposing but just simply not seeing them as > disks to create a datastore on -- thye just simply do not show up at > all. > > I am a bit frustrated, but hoping that someone who has done this > (Gemini assures me that it is possible!) can provide some > guidance/pointiners/pont to a step-by-step howto. > > Thanks! :-) > > Regards, > > ALan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: NVMe/TCP + VMware
by Aviv Caro 17 May '26

17 May '26

Hi, Ceph officially supports nvme/tcp only since tentacle release. In squid some important code is missing. We can discuss in more in #nvmeof channel in ceph workspace in slack. Aviv -----Original Message----- From: Alan Murrell via ceph-users <ceph-users(a)ceph.io> Sent: Saturday, 16 May 2026 3:18 To: ceph-users(a)ceph.io Subject: [EXTERNAL] [ceph-users] NVMe/TCP + VMware Hello, I am hoping someone might be able to help, or point me to a "bulletproof" step-by-step howto of connecting my Ceph 19.2.3 cluster to my VMware 8.0 cluster as I want to do some testing. I have spent the last day and half with Gemini trying to get this to work, and have been able to get as far as VMware seemingly to see my RBD images that I am exposing but just simply not seeing them as disks to create a datastore on -- thye just simply do not show up at all. I am a bit frustrated, but hoping that someone who has done this (Gemini assures me that it is possible!) can provide some guidance/pointiners/pont to a step-by-step howto. Thanks! :-) Regards, ALan _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

NVMe/TCP + VMware
by Alan Murrell 16 May '26

16 May '26

Hello, I am hoping someone might be able to help, or point me to a "bulletproof" step-by-step howto of connecting my Ceph 19.2.3 cluster to my VMware 8.0 cluster as I want to do some testing. I have spent the last day and half with Gemini trying to get this to work, and have been able to get as far as VMware seemingly to see my RBD images that I am exposing but just simply not seeing them as disks to create a datastore on -- thye just simply do not show up at all. I am a bit frustrated, but hoping that someone who has done this (Gemini assures me that it is possible!) can provide some guidance/pointiners/pont to a step-by-step howto. Thanks! :-) Regards, ALan

1 0

Birds of a Feather Proposal for SuperComputing 2026
by Anthony Middleton 15 May '26

15 May '26

Hey Ceph friends, I hope all is well. This year, the Ceph Foundation plans to submit a Birds of a Feather (BoF) proposal for SuperComputing 2026 (SC26), and we want this to be a genuinely community-driven session, shaped by the people who are deploying, developing, and using Ceph in HPC environments. A BoF is an open, interactive session at SC where practitioners and developers gather to share experiences, discuss challenges, and influence future direction. It's less of a talk and more of a structured community conversation. We'd love your input on two things: 1. TOPIC IDEAS: What should we discuss? We're looking for the most pressing topics, questions, and challenges facing Ceph in HPC right now. Some starting points: - CephFS performance and tuning for HPC workloads - NVMe/SSD tiering and all-flash deployments - Data encryption and security at scale - Interoperability with HPC job schedulers and parallel filesystems - Operational challenges at petabyte scale - What's missing in Ceph for HPC use cases? What would YOU want to talk about? Reply to this thread or fill out this short form: https://forms.gle/SEhgjYz9pda8SgT88 2. PARTICIPATION: Are you attending SC26? We're looking for: - Lightning talk speakers (5–10 min) who can share a real deployment story or lesson learned - Co-session leaders from different organizations (especially HPC centers, national labs, or universities) - Community members who want to show up and join the discussion SC26 takes place November 15–20, 2026, in Chicago, IL. The BoF proposal deadline is July 2nd. This session works best when it reflects a wide range of voices and organizations within the Ceph community, so the more input we receive, the stronger the proposal and the session will be. Please reply here, reach out directly, or share this with anyone in your network who works with Ceph in HPC contexts. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Ceph Tech Talk Recording for May and June Announcement!
by Konold, Martin 14 May '26

14 May '26

Am 2026-05-12 17:01, schrieb Anthony Middleton via ceph-users: Hi, YouTube gives me "This video has been removed by the uploader". Regards --martin > Hello Ceph Friends! > > I hope your week is off to a great start! You can find the recording of > yesterday's Ceph Tech Talk on Containerized Deployment of Ceph with > KCLI > here: https://youtu.be/vPOYyNtpHp4. -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

1 0

CFS IO500 ISC'26 Deadline 12 June 2026
by IO500 Committee 13 May '26

13 May '26

https://io500.org/submission Deadline 12 June 2026 AoE The IO500 is now accepting and encouraging submissions for the upcoming semi-annual IO500 Production and Research lists, in conjunction with ISC HPC 26. Once again, we are also accepting submissions to the 10 Client Node Challenge to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Submissions Webpage. The new ranked lists will be announced at the "IO500: High Performance Storage Community" BoF. We hope to see many new results. The GitHub repository has a new tag io500-isc26 for the current version of the benchmark for this list. Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community-accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes, giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: - Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance, and the documentation of how the numbers are achieved. - Support small to extreme-scale Research and Production HPC systems using flexible storage APIs - Maximizing simplicity in running the benchmark suite - Specifically, the benchmark suite includes a hero run of both IOR and mdtest, configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly prescribed parameters in an attempt to determine a lower performance bound. New in this release is a random 4KB read phase to better understand the performance of non-sequential IO patterns used in AI workloads. Finally, it includes a namespace search, as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. - The goals of the community are also multi-fold: - Gather historical data for the sake of analysis and to aid predictions of storage futures - Collect tuning information to share valuable performance optimizations across the community - Encourage vendors and designers to optimize for workloads beyond "hero runs" - Establish bounded expectations for users, procurers, and administrators - Understand and be able to reproduce performance on storage systems - Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. - The IO500 follows a two-stage approach. First, there will be a two-week stabilization period during which we encourage the community to verify that the benchmark runs properly on a variety of storage systems. During this period, the benchmark may be updated based upon feedback from the community. The final benchmark will then be released. We expect that runs compliant with the rules made during the stabilization period will be valid as a final submission unless a significant defect is found. 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather Once again, we encourage you to submit to join our community, and to attend the ISC26 BoF on ISC26, where we will announce the new IO500 Production and Research lists and their 10 client node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage features customizable sorting, allowing you to submit on a small system and still achieve a very good per-client score, for example. We will also highlight new and interesting results at the BoF.

1 0

Ceph Tech Talk May
by Anthony Middleton 12 May '26

12 May '26

There was an issue with the video, so I deleted it and reuploaded it. Sorry for the confusion. Please enjoy Shraddha's presentation - https://www.youtube.com/watch?v=_zPR0s5H468 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph Tech Talk Recording for May and June Announcement!
by Anthony Middleton 12 May '26

12 May '26

Hello Ceph Friends! I hope your week is off to a great start! You can find the recording of yesterday's Ceph Tech Talk on Containerized Deployment of Ceph with KCLI here: https://youtu.be/vPOYyNtpHp4. Our next session is scheduled for June 9th. The topic will be "Building Ultra-Resilient Ceph Clusters with Xinnor and OSNexus," featuring Steven Umbehocker, CEO of OSNexus, and Dmitry Livshits, CEO of Xinnor. RSVP here: https://www.linkedin.com/events/7459977670458757120?viewAsMember=true Subscribed to the Ceph Community Calendar <https://calendar.google.com/calendar/b/1?cid=OXRzOWM3bHQ3dTF2aWMyaWp2dnFxbG…> for more details. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

RGW/Beast logs show the proxy IP instead of the real client IP
by farhad kh 12 May '26

12 May '26

Hi, i try to enable RGW access log for calculation byte_send, byte_reseved, operation_count, audit_detail, request_tracing and ... with set these options ceph config set client.rgw.vanak rgw_enable_usage_log true ceph config set client.rgw rgw_enable_usage_log true ceph config set client.rgw.vanak rgw_ops_log_rados true RGW Beast access logs show the HAProxy node IP instead of the real client IP, despite X-Forwarded-For being correctly set by the cephadm ingress service. Root Cause Analysis: 1. Logging Mechanism: By design, the Beast frontend logs the TCP source address (Layer 4) rather than parsing the X-Forwarded-For HTTP header (Layer 7). This ensures audit logs reflect the trusted transport peer. 2. Missing PROXY Protocol: To preserve the real client IP at the transport layer, PROXY protocol must be used (send-proxy-v2 in HAProxy and proxy_protocol=true in RGW). 3. Orchestration Blocker: The current version of cephadm ingress (e.g., Reef) does not support the enable_proxy_protocol parameter in its service specification. Consequently, it cannot generate the required send-proxy-v2 directive in the managed HAProxy configuration. Technical Impact: - Observability: RGW logs are insufficient for client-level auditing, forensics, or per-IP bandwidth accounting. - Workaround Needed: Client attribution must currently be performed at the HAProxy log layer rather than the RGW log layer. Proposed Resolution / Solution Paths: 1. Upgrade: Verify and move to a Ceph version where cephadm ingress supports enable_proxy_protocol. 2. Manual Ingress: Replace the cephadm-managed ingress with a manual HAProxy deployment configured with send-proxy-v2 to satisfy RGW’s proxy_protocol=true requirement. 3. Upstream Change: Propose a feature update for cephadm to include first-class PROXY protocol support in the ingress spec to ensure parity with Beast’s logging architecture. Captured evidence: - tcpdump confirms X-Forwarded-For is present but no PROXY preamble is sent. - RGW is configured for proxy_protocol=true but receives plain HTTP streams. 07:09:44.247960 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [.], ack 1, win 502, options [nop,nop,TS val 735749891 ecr 4199462555], length 0 E..4..@.@.;..........,.@.C@f... ....#...... +....N.. 07:09:44.248068 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [P.], seq 1:902, ack 1, win 502, options [nop,nop,TS val 735749891 ecr 4199462555], length 901 E.....@.@.83.........,.@.C@f... .....G..... +....N..PUT /test/test2 HTTP/1.1 host: 172.29.244.8 accept-encoding: identity user-agent: aws-cli/2.31.18 md/awscrt#0.27.6 ua/2.1 os/linux#6.8.0-86-generic md/arch#x86_64 lang/python#3.13.7 md/pyimpl#CPython m/Z,E,n,b,W,N cfg/retry-mode#standard md/installer#exe md/distr ib#ubuntu.24 md/prompt#off md/command#s3api.put-object expect: 100-continue x-amz-checksum-crc64nvme: DxVUd/EiBAY= x-amz-sdk-checksum-algorithm: CRC64NVME x-amz-date: 20260510T070943Z x-amz-content-sha256: e951419ac0d10cc76f3e5eb415e68f7d7d6dd99e016d1463f0cb1ce18680d8e1 authorization: AWS4-HMAC-SHA256 Credential=MPQOYU9DYBV012QXHQ0H/20260510/tehran/s3/aws4_request, SignedHeaders=host;x-amz-checksum-crc64nvme;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-a lgorithm, Signature=59da15f49a75490adf3e90829616416ed0abac757272146692c3a6149849e1c3 content-length: 52428800 x-forwarded-for: 172.29.244.19 connection: close 07:09:44.248088 ens160 Out IP ceph-csb-02.8000 > 172.29.244.7.44332: Flags [.], ack 902, win 503, options [nop,nop,TS val 4199462556 ecr 735749891], length 0 E..4I.@.@............@.,... .CC.....@p..... .N..+... 07:09:44.253519 ens160 Out IP ceph-csb-02.8000 > 172.29.244.7.44332: Flags [P.], seq 1:26, ack 902, win 503, options [nop,nop,TS val 4199462561 ecr 735749891], length 25 E..MI.@.@............@.,... .CC.....@...... .N..+...HTTP/1.1 100 CONTINUE 07:09:44.253912 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [.], ack 26, win 502, options [nop,nop,TS val 735749897 ecr 4199462561], length 0 E..4..@.@.;..........,.@.CC....9........... +.. .N.. 07:09:44.255059 ens160 In IP 172.29.244.7.44332 > ceph-csb-02.8000: Flags [P.], seq 902:8142, ack 26, win 502, options [nop,nop,TS val 735749898 ecr 4199462561], length 7240 E..|>.@.@..n.........,.@.CC....9....\...... +.. .N.._>.z..?....b.T}g1Wa^yc..wWq$...S.....(..0...yf''...`..O....P.....Y.H.....?#.:f...z..v....{w_fjx..a....v...>+.............J;...M.......e...OE.....#.. .h...U.UW.`...d.,..6...2e .i(.;.... + my client_ip is '172.29.244.19' but show in log haproxy ip '192.29.244.7' :| {"log":"debug 2026-05-09T11:09:27.398+0000 7fc897ac4640 1 beast: 0x7fc7b18d6310: 172.29.244.7 - farhad$root [09/May/2026:11:09:27.394 +0000] \"GET / HTTP/1.1\" 200 510 - \"aws-cli/1.22.34 Python/3.10.12 Linux/5.15.0-164-generic botocore/1.23.34\" - latency=0.004000204s\n","stream":"stdout","time":"2026-05-09T11:09:27.405683471Z"} how can i fix this problem ? anyone have solution?

1 0

Exploring Interest- Ceph Days Chicago 2026
by Lee-Ann Pullar 11 May '26

11 May '26

Hey All, I’m reaching out to start some initial planning discussions around a potential Ceph Days Chicago event, tentatively scheduled for Monday, November 16, 2026. We would like to collaborate with the OpenInfra community and, as the event coincides with the Supercomputing Conference, bring together attendees from across the Ceph, OpenInfra, and SC26 communities. Before moving further with planning, we’d love to gauge interest from the community. If you have a moment, please take this quick survey. https://forms.gle/jzrP7A4hkgiTL98X6 Your feedback will help shape the size, scope, and direction of the event. If there’s enough interest in the date and location, we’ll move forward with planning to make this a great community event! Thank you, Lee-Ann Pullar

1 0

OSDs and RGW busy without traffic since Tentacle Upgrade
by Phillip Schichtel 10 May '26

10 May '26

Hi everyone, I've previously posted about this in Slack, but it was suggested to bring this to the mailing list, so here I am. The post in slack: [1]. I've recently upgraded a small Squid cluster to Tentacle (from 19.2.3 straight to 20.2.1). The cluster has 3 nodes (Debian Bookworm, 2x EPYC 7371, 256gig memory) and is primarily used for RBD (VM images) and RGW/S3. There are 43 OSDs, 31 Intel SATA SSDs (10 per node, but the first node has 1 extra) and 12 Seagate SAS SSDs (4 per node). The SATA ssds have device class "ssd" with ~50% average utilization, the SAS SSDs have device class "sas-ssd" with ~15% average utilization and serve different pools. RGW only utilizes the ssd class. The cluster has been continuously upgraded from (I think) Mimic or Nautilus. It was originally deployed using ceph-deploy and switched to cephadm as soon as that was stable. Under Squid the cluster was usually running around 1-3% CPU most of the time, since the applications aren't super IO heavy. During off-hours we only see about 5-8k iops usually. With the upgrade to Tentacle the nodes jumped to about 60% CPU utilization and 150k iops (Hosts Overall Performance dashboard). All that while ceph status and all the metrics I could find in Grafana/Prometheus don't report increased throughput, iops or even latency. Checking on the nodes directly (e.g. with htop's IO view) all the OSDs in the ssd class consistently generate quite a bit throughput. The iops are generated by the OSDs, the CPU load is generated both by the OSDs and RGW. RGW is usually deployed with 3 instances (1 per node), but the issue also appears when running with just 1. It is currently single-site, but I have toyed around with multi-site a while back on Squid. Increasing the logging (debug_rgw 20/5 & debug_ms 1/5) showed that it often attempted to get data_log.* objects but failed to do so. The .rgw.log pool only contained a data_log.0 object, blindly creating empty data_log.{1..127} objects also didn't silence the log line. An example log: osd.2 v2:_._._._:6843/2369122728 33152999 ==== osd_op_reply(286992957 data_log.20 [call] v109598'11820244296 uv0 ondisk = -61 ((61) No data available)) ==== 155+0+0 (crc 0 0 0) 0x55e65cff3400 con 0x55e656ad7800 When I restart the single instance, the load returns back to normal for exactly 20 minutes, after which it returns to roughly the same level. This pointed me to the `rgw_sync_log_trim_interval` config. Both the log lines and this setting seem to be related to multi-site replication logs. Changing the option to another value also changes the delay until the issue appears, which makes me pretty certain the issue is triggered by one of the coroutines spawned by RGWSyncLogTrimThread. Luckily setting the interval to 0 doesn't start that thread, which I have set now to prevent RGW from burning through my flash cells and to keep power usage in check. I've also temporarily swapped out the RGW container image for quay.ceph.io/ceph- ci/ceph:tentacle-debug@sha256:1b89786ce81c900227a2077be33dadda8722006b0 45b409e91e1ed89d7547be1 (I hope that was the right image to get the latest backports for tentacle), but that version also has this issue. I've also created a flamegraph with perf (executed perf outside the container and then converted to svg inside the ceph container), but it doesn't seem super useful to me since the interesting symbols seems to be missing. I've attached it anyway. Should I go ahead and report this to tracker.ceph.com or does anyone have ideas on what to do here? Thanks in advance for any input! ~ Phillip [1]: https://ceph-storage.slack.com/archives/C1HFU4JK1/p1776029060801259

1 0

Re: InfluxDB deprecated?
by Anthony D'Atri 10 May '26

10 May '26

This may be a function of packages available on the underlying EL distributions used as the base for package builds. I believe that there may be an RPM in EPEL, but I personally am hesitant to rely on EPEL for anything. Installation via pip3 should work. I'll forward this to someone involved in releases / builds. ceph.spec.in has %package mgr Summary: Ceph Manager Daemon %if 0%{?suse_version} Group: System/Filesystems %endif Requires: ceph-base = %{_epoch_prefix}%{version}-%{release} Requires: ceph-mgr-modules-core = %{_epoch_prefix}%{version}-%{release} Requires: libcephsqlite = %{_epoch_prefix}%{version}-%{release} %if 0%{?weak_deps} Recommends: ceph-mgr-dashboard = %{_epoch_prefix}%{version}-%{release} Recommends: ceph-mgr-diskprediction-local = %{_epoch_prefix}%{version}-%{release} Recommends: ceph-mgr-k8sevents = %{_epoch_prefix}%{version}-%{release} Recommends: ceph-mgr-cephadm = %{_epoch_prefix}%{version}-%{release} Recommends: python%{python3_pkgversion}-influxdb %endif I see an RPM package in EPEL9 but not EPEL10, so I suspect that's the root cause. I suspect that 18.2.7 might still work but do not have the wherewithal to test. The EPEL 9 RPM may well have been outdated. Possibly the build just needs to install via pip3. I'll see what I can do. > On May 7, 2026, at 3:02 PM, Robert Sander via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > Going from 16.2.15 in packages to cephadm managed containers and > upgrading to 18.2.8 we find that the InfluxDB MGR module does > not work any more: > > # ceph mgr module enable influx > Error ENOENT: module 'influx' reports that it cannot run on the active manager daemon: influxdb python module not found (pass --force to force enablement) > > The container image does not contain the necessary Python module. > > There is a tracker issue open for 18 months now: > https://tracker.ceph.com/issues/68892 > > Has the InfluxDB support been deprecated? I do not believe so, though the cool kids skate Prometheus ;) > Why is it still in the documentation for recent Ceph versions? See above. Oversights for soft dependencies on rarely-used bits can happen. > > Regards > -- > Robert Sander > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > http://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 220009 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Ceph deployment for HPC and research workflows
by Anthony D'Atri 09 May '26

09 May '26

> On May 6, 2026, at 9:55 AM, Max Breitmeyer via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello all, > At UMBC we were given a grant to run a ceph cluster as our primary research > storage for an HPC facility. The ceph cluster consists of the following > hardware: > Mon” nodes x3: > • 2x 25Gb interfaces > • 192GB RAM > • Storage: > -- 0 (none) OSDs/drives Why nodes dedicated only to mons? I suggest considering a deployment where mons are colocated on OSD or MDS nodes, either abstracting client config through DNS, by constraining placement to ensure that the orchestrator is failure-domain aware when scheduling, or possibly even just specifying all of the mons on the client mounts for resilience. The latter for sure doesn't scale well and is prone to not keeping up with changes. Clients need to be able to contact at least one mon to download the current monmap. > “HDD” nodes x16: > • 2x 25Gb interfaces > • 192GB RAM > • Storage: > -- 12 OSDs w/ 20TB HDDs > -- 2 local Write-Access-Logs (journal) w/ 8TB NVMe drives WAL+DB? > “NVMe” nodes x3: > • 2x 100Gb interfaces > • 384GB RAM > • Storage: > -- 16x OSDs w/ 8TB NVMe drives These could easily have colocated mons, sparing the dedicated mon nodes. > “MDS” nodes x3: > • 2x 100Gb interfaces > • 192GB RAM > • Storage: > -- 8x OSDs w/ 1TB NVMe drives A few thoughts: * I recommend 5 mon daemons vs only 3 * Since the MDS node spec above mentions 1TB (960 GB) SSDs and the NVMe node spec mentions 8 TB (7.6 TB) SSDs, I might infer that you constrain -- or intended to constrain -- the MDS metadata pool to only these 24x OSDs? And that the 48x 7.6 T SSD nodes host OSDs used for a separate CephFS filesystem or specific files/directories via layout xattrs? I might suggest a rearchitecture, which could be done incrementally without downtime: * Colocate 5 mon daemons on other nodes and either use these 3 systems for something else, locate some RGWs (if in use) there. If those nodes were ordered or could be made NVMe-capable, you might repurpose them as additional OSD nodes. If they have SATA bays, you could deploy HDD OSDs on them too. * It's best to not have OSDs within a given device class not be very different sizes, but Ceph can handle it. Setting mgr/balancer/upmap_max_deviation 1 will help the balancer do well with the smaller OSDs. * If the 1T and 7.6T SSD OSDs aren't in the same device class, consider consolidating them. Your workload -- and availability -- will benefit. The fine folks at OSNexus could help you with these ideas. "to use the full 192GB" There ya go. Note that the MDS cache size config value (default is 4GiB I think?) is like osd_memory_target: advisory, not a hard limit. Under heavy load an MDS can and will attempt to use more than the configured amount. You also need to allow some margin for other MDS internal uses, other daemons (node_exporter, the OSDs I presume you're running, etc), and the usual OS tasks. I might suggest with 192 GiB of physmem to never configure more than 128 GiB at most for MDS cache. Individual MDSes with sizes > 64 GiB may also be very slow to failover. So without seeing your actual deployment details this is speculation, but one thought is that the too-large MDS cache size DoSed or OOMed the colocated OSDs. > > In the six months since moving from a different storage solution (isilon), > we've had multiple crashes on the system that have completely taken down > our system as a result of multiple thousands of jobs being run at the same > time, causing an overload on the active MDS nodes. Has the use of ceph as a > primary research storage been used in an HPC with many dozens of different > workflows and filetypes being used at once? Or is a better to design to > come up with a tiered approach in combination with a second storage > solution? > > -- > V/R, > Maxwell Breitmeyer > UMBC HPCF Specialist > Graduate Student > (443) 835-8250 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Ceph deployment for HPC and research workflows
by Max Breitmeyer 08 May '26

08 May '26

Hey Mark, Initially we had it set to use the full 192GB, but after an issue where starting up the MDS service was going straight to OOM, we brought it down to 64GB. This has prevented any further crashes. We have a support vendor that we're working with (OSNexus) at the moment. On Wed, May 6, 2026 at 10:40 AM Mark Nelson via ceph-users < ceph-users(a)ceph.io> wrote: > Hi Max, > > Probably the first order of business is to understand the nature of the > crashes you are seeing. Just as a guess, if you have thousands of jobs > being run at the same time, you may have a lot of memory pressure on the > MDSses. If the crashes are just the MDSes going OOM, you might want to > verify how much of the 192GB of RAM on each node the MDSes are allowed > to use for cache. > > Beyond that, it might take some digging to understand the specifics of > your issues. There are other institutions that do use CephFS for HPC use > cases though. I gave a talk at the IO500 BoF at SC24 in collaboration > with GWDG in Germany regarding their Ceph deployment. > > Shameless plug: Clyso does offer academic support packages for upstream > Ceph if that is of interest to you. > > Thanks, > Mark > > On 5/6/26 6:54 AM, Max Breitmeyer via ceph-users wrote: > > Hello all, > > At UMBC we were given a grant to run a ceph cluster as our primary > research > > storage for an HPC facility. The ceph cluster consists of the following > > hardware: > > Mon” nodes x3: > > • 2x 25Gb interfaces > > • 192GB RAM > > • Storage: > > -- 0 (none) OSDs/drives > > “HDD” nodes x16: > > • 2x 25Gb interfaces > > • 192GB RAM > > • Storage: > > -- 12 OSDs w/ 20TB HDDs > > -- 2 local Write-Access-Logs (journal) w/ 8TB NVMe drives > > “NVMe” nodes x3: > > • 2x 100Gb interfaces > > • 384GB RAM > > • Storage: > > -- 16x OSDs w/ 8TB NVMe drives > > “MDS” nodes x3: > > • 2x 100Gb interfaces > > • 192GB RAM > > • Storage: > > -- 8x OSDs w/ 1TB NVMe drives > > > > > > In the six months since moving from a different storage solution > (isilon), > > we've had multiple crashes on the system that have completely taken down > > our system as a result of multiple thousands of jobs being run at the > same > > time, causing an overload on the active MDS nodes. Has the use of ceph > as a > > primary research storage been used in an HPC with many dozens of > different > > workflows and filetypes being used at once? Or is a better to design to > > come up with a tiered approach in combination with a second storage > > solution? > > > -- > Best Regards, > Mark Nelson > Co-Founder and Head of R&D > > Clyso Inc. > p: +49 89 21552391 12 | a: North Vancouver, B.C. > w: https://clyso.com | e: mark.nelson(a)clyso.com > > We are hiring: https://www.clyso.com/jobs/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- V/R, Maxwell Breitmeyer UMBC HPCF Specialist Graduate Student (443) 835-8250

2 1

Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Adiga, Anantha 08 May '26

08 May '26

Thank you Eugen. Truly appreciate your quick response as always. Regards, Anantha -----Original Message----- From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Friday, May 8, 2026 6:15 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty Hi, I'm afraid I can't really help any further. I don't have any CephFS snapshot mirroring enabled in any of my test clusters at the moment. The only thing I could quickly check is that on a Squid cluster, I was able to create a snapshot via dashboard of a directory with these permissions: /mnt# ll volumes/_nogroup/ total 0 drwxr-sr-x 3 root root 1 May 8 12:43 ./ drwxr-xr-x 3 root root 2 May 8 12:43 ../ d-----S--- 3 root root 2 May 8 12:43 test-sub-restrictive/ /mnt# ceph fs subvolume snapshot ls cephfs test-sub-restrictive [ { "name": "2026-05-08T14:45:13.823+02:00" } This works a bit differently in Pacific, I created a subvolume in the same way, changed directory permissions like above, but the Dashboard can't show me the path with this error: cephfs.OSError: opendir failed: Permission denied [Errno 13] I can create a snapshot via command line though, so I'm not sure how this error would come into play here. I would expect/hope that upgrading your primary site could resolve this, but of course that's just a guess. Regards, Eugen Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>: > Hi Eugen, > > Regarding this issue, we have 18TB of data successfully mirrored, but > the process is completely blocked by these permission restrictions. > CephFS mirroring daemon cannot access directories with restrictive > permissions, and I'm seeking guidance on the proper solution. > > We also find that this issue is not seen on Ceph Reef ( both backup > source and target ) > > Environment: > • Primary Cluster: Ceph Pacific 16.2.5 > • Backup Cluster: Ceph Reef 18.2.7 > • Data Size: ~18TB successfully mirrored before hitting permission issues > > Problem Description: > CephFS mirroring fails when encountering directories with highly > restrictive permissions (d-----S---). The mirror daemon cannot access > these directories, causing snapshot synchronization to fail entirely. > > Error logs: > cephfs::mirror::PeerReplayer() should_sync_entry: failed to stat prev > entry= <path/to/file>: (13) Permission denied > cephfs::mirror::PeerReplayer() do_sync_snaps: failed to synchronize > dir_root=/volumes/capi_subvolumegroup/capi_subvolume, > snapshot=scheduled-2026-04-29-22_00_00 > > Root Cause Analysis: > The cephfs-mirror daemon runs with --setuser ceph --setgroup ceph > arguments, causing it to drop privileges to the ceph user. Even with > full Ceph capabilities, the MDS enforces POSIX permissions, preventing > access to directories with no read/execute permissions. > > Directory permissions: > • Many directories with d-----S--- permissions (no access for anyone) > • These directories contain critical data that must be backed up > • Cannot modify source file permissions (production system) > > Attempted solutions: > > [client.cephfs-mirror.a001s017.aoolcx] > key = <key> > caps mds = "allow *" > caps mgr = "allow *" > caps mon = "allow *" > caps osd = "allow *" > > root@a001s016:~# ceph auth get client.mirror_gfs_src exported keyring > for client.mirror_gfs_src [client.mirror_gfs_src] > key = <key> > caps mds = "allow *" > caps mgr = "allow *" > caps mon = "allow *" > caps osd = "allow *" > root@a001s016:~# > > ceph config set client.cephfs-mirror client_mount_uid 0 ceph config > set client.cephfs-mirror client_mount_gid 0 > > ceph config set client.cephfs-mirror setuser 0 ceph config set > client.cephfs-mirror setgroup 0 > > Attempted to use extra_entrypoint_args in service spec, but Pacific returns: > Error EINVAL: ServiceSpec: __init__() got an unexpected keyword > argument 'extra_entrypoint_args' > > Is there a supported way to run cephfs-mirror with root privileges to > bypass POSIX permission checks? > Are there service spec parameters in Pacific that allow overriding the > --setuser/--setgroup arguments? > Is this a known limitation of CephFS mirroring with restrictive file > permissions? > What is the recommended approach for mirroring filesystems containing > directories that require root access? > > Request guidance on: > • Proper configuration for root-level CephFS mirroring access > • Whether this is a design limitation or configuration issue > • Recommended best practices for mirroring restricted filesystems > > Regards, > Anantha > > -----Original Message----- > From: Eugen Block <eblock(a)nde.ag> > Sent: Thursday, April 23, 2026 1:11 PM > To: Adiga, Anantha <anantha.adiga(a)intel.com> > Cc: ceph-users(a)ceph.io > Subject: Re: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists > already an active peer" Error (EEXIST) Despite Empty > > Awesome, glad I could help! > > Zitat von "Adiga, Anantha" <anantha.adiga(a)intel.com>: > >> Hi Eugen, >> >> Thank you so much. Yes, that fixed the issue: >> removed the peer uuid from the primary cluster. This removed the >> attribute from the backup cluster as well. Then import peer bootstrap >> token worked. >> >> Regards, >> Anantha >> >> -----Original Message----- >> From: Eugen Block via ceph-users <ceph-users(a)ceph.io> >> Sent: Thursday, April 23, 2026 12:56 AM >> To: ceph-users(a)ceph.io >> Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists >> already an active peer" Error (EEXIST) Despite Empty >> >> Hi, >> >> a quick look into the code [0] shows (note that I'm not a developer): >> >> def set_mirror_info(local_cluster_id, local_fsid, remote_fs): >> log.info(f'setting {local_cluster_id}::{local_fsid} on remote') >> try: >> remote_fs.setxattr('/', 'ceph.mirror.info', >> f'cluster_id={local_cluster_id} >> fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE) >> except cephfs.Error as e: >> if e.errno == errno.EEXIST: >> >> >> My interpretation is that the setxattr command fails on the remote >> site. Could this be an auth caps issue? Did you verify the keyrings >> and their permissions? >> >> This is the only lead I got. >> >> Regards, >> Eugen >> >> [0] >> https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20 >> c >> bd6a/src/pybind/mgr/mirroring/fs/snapshot_mirror.py#L411 >> >> Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>: >> >>> Hi, >>> >>> This is issue with CephFS snapshot mirroring setup between two >>> clusters and would appreciate your assistance. >>> >>> Environment: >>> >>> * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) >>> [cid:image001.png@01DCD199.2933E2E0] >>> * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) >>> [cid:image002.png@01DCD199.2933E2E0] >>> * Filesystem: cephfs >>> * Issue: Unable to import peer bootstrap token >>> >>> Problem Description: >>> When attempting to import a peer bootstrap token using: >>> ceph fs snapshot mirror peer_bootstrap import cephfs <token> I >>> receive the error: >>> Error EEXIST: already an active peer mgr.server reply reply (17) >>> File exists already an active peer However, ceph fs snapshot mirror >>> peer_list cephfs returns an empty list [], indicating no peers are >>> configured. >>> >>> Steps Performed: >>> Initial Setup: >>> 1. Enabled mirroring module: ceph mgr module enable mirroring >>> 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror >>> 3. Enabled mirroring on filesystem: ceph fs snapshot mirror >>> enable cephfs >>> 4. Created remote user and bootstrap token on backup cluster >>> 5. Verified mirroring daemons are running: ceph fs snapshot >>> mirror daemon status >>> Troubleshooting Attempts: >>> 1. Reset filesystem mirroring: >>> ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror >>> enable cephfs >>> 2. Module reset: >>> ceph mgr module disable mirroring >>> ceph mgr module enable mirroring >>> 3. Service recreation: >>> ceph orch rm cephfs-mirror >>> ceph orch apply cephfs-mirror >>> 4. Checked for stale configuration: >>> o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror >>> o ceph fs snapshot mirror dirmap cephfs returns empty >>> o ceph fs snapshot mirror status cephfs shows no active mirroring >>> >>> Current Status: >>> >>> * Mirroring daemons: Running and visible in daemon status >>> * Peer list: Empty [] >>> * Filesystem mirroring: Enabled >>> * Error persists: EEXIST despite no visible peers >>> >>> Diagnostic Information: >>> >>> * MGR logs show: mgr.server reply reply (17) File exists already >>> an active peer >>> * No connectivity issues between clusters (ports 6789, 3300, >>> 6800+ tested) >>> * Both clusters are healthy with no other issues >>> >>> Regards, >>> Anantha >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Eugen Block 08 May '26

08 May '26

Hi, I'm afraid I can't really help any further. I don't have any CephFS snapshot mirroring enabled in any of my test clusters at the moment. The only thing I could quickly check is that on a Squid cluster, I was able to create a snapshot via dashboard of a directory with these permissions: /mnt# ll volumes/_nogroup/ total 0 drwxr-sr-x 3 root root 1 May 8 12:43 ./ drwxr-xr-x 3 root root 2 May 8 12:43 ../ d-----S--- 3 root root 2 May 8 12:43 test-sub-restrictive/ /mnt# ceph fs subvolume snapshot ls cephfs test-sub-restrictive [ { "name": "2026-05-08T14:45:13.823+02:00" } This works a bit differently in Pacific, I created a subvolume in the same way, changed directory permissions like above, but the Dashboard can't show me the path with this error: cephfs.OSError: opendir failed: Permission denied [Errno 13] I can create a snapshot via command line though, so I'm not sure how this error would come into play here. I would expect/hope that upgrading your primary site could resolve this, but of course that's just a guess. Regards, Eugen Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>: > Hi Eugen, > > Regarding this issue, we have 18TB of data successfully mirrored, > but the process is completely blocked by these permission > restrictions. CephFS mirroring daemon cannot access directories with > restrictive permissions, and I'm seeking guidance on the proper > solution. > > We also find that this issue is not seen on Ceph Reef ( both backup > source and target ) > > Environment: > • Primary Cluster: Ceph Pacific 16.2.5 > • Backup Cluster: Ceph Reef 18.2.7 > • Data Size: ~18TB successfully mirrored before hitting permission issues > > Problem Description: > CephFS mirroring fails when encountering directories with highly > restrictive permissions (d-----S---). The mirror daemon cannot > access these directories, causing snapshot synchronization to fail > entirely. > > Error logs: > cephfs::mirror::PeerReplayer() should_sync_entry: failed to stat > prev entry= <path/to/file>: (13) Permission denied > cephfs::mirror::PeerReplayer() do_sync_snaps: failed to synchronize > dir_root=/volumes/capi_subvolumegroup/capi_subvolume, > snapshot=scheduled-2026-04-29-22_00_00 > > Root Cause Analysis: > The cephfs-mirror daemon runs with --setuser ceph --setgroup ceph > arguments, causing it to drop privileges to the ceph user. Even with > full Ceph capabilities, the MDS enforces POSIX permissions, > preventing access to directories with no read/execute permissions. > > Directory permissions: > • Many directories with d-----S--- permissions (no access for anyone) > • These directories contain critical data that must be backed up > • Cannot modify source file permissions (production system) > > Attempted solutions: > > [client.cephfs-mirror.a001s017.aoolcx] > key = <key> > caps mds = "allow *" > caps mgr = "allow *" > caps mon = "allow *" > caps osd = "allow *" > > root@a001s016:~# ceph auth get client.mirror_gfs_src > exported keyring for client.mirror_gfs_src > [client.mirror_gfs_src] > key = <key> > caps mds = "allow *" > caps mgr = "allow *" > caps mon = "allow *" > caps osd = "allow *" > root@a001s016:~# > > ceph config set client.cephfs-mirror client_mount_uid 0 > ceph config set client.cephfs-mirror client_mount_gid 0 > > ceph config set client.cephfs-mirror setuser 0 > ceph config set client.cephfs-mirror setgroup 0 > > Attempted to use extra_entrypoint_args in service spec, but Pacific returns: > Error EINVAL: ServiceSpec: __init__() got an unexpected keyword > argument 'extra_entrypoint_args' > > Is there a supported way to run cephfs-mirror with root privileges > to bypass POSIX permission checks? > Are there service spec parameters in Pacific that allow overriding > the --setuser/--setgroup arguments? > Is this a known limitation of CephFS mirroring with restrictive file > permissions? > What is the recommended approach for mirroring filesystems > containing directories that require root access? > > Request guidance on: > • Proper configuration for root-level CephFS mirroring access > • Whether this is a design limitation or configuration issue > • Recommended best practices for mirroring restricted filesystems > > Regards, > Anantha > > -----Original Message----- > From: Eugen Block <eblock(a)nde.ag> > Sent: Thursday, April 23, 2026 1:11 PM > To: Adiga, Anantha <anantha.adiga(a)intel.com> > Cc: ceph-users(a)ceph.io > Subject: Re: [ceph-users] Re: CephFS Snapshot Mirroring - "File > exists already an active peer" Error (EEXIST) Despite Empty > > Awesome, glad I could help! > > Zitat von "Adiga, Anantha" <anantha.adiga(a)intel.com>: > >> Hi Eugen, >> >> Thank you so much. Yes, that fixed the issue: >> removed the peer uuid from the primary cluster. This removed the >> attribute from the backup cluster as well. Then import peer bootstrap >> token worked. >> >> Regards, >> Anantha >> >> -----Original Message----- >> From: Eugen Block via ceph-users <ceph-users(a)ceph.io> >> Sent: Thursday, April 23, 2026 12:56 AM >> To: ceph-users(a)ceph.io >> Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists >> already an active peer" Error (EEXIST) Despite Empty >> >> Hi, >> >> a quick look into the code [0] shows (note that I'm not a developer): >> >> def set_mirror_info(local_cluster_id, local_fsid, remote_fs): >> log.info(f'setting {local_cluster_id}::{local_fsid} on remote') >> try: >> remote_fs.setxattr('/', 'ceph.mirror.info', >> f'cluster_id={local_cluster_id} >> fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE) >> except cephfs.Error as e: >> if e.errno == errno.EEXIST: >> >> >> My interpretation is that the setxattr command fails on the remote >> site. Could this be an auth caps issue? Did you verify the keyrings >> and their permissions? >> >> This is the only lead I got. >> >> Regards, >> Eugen >> >> [0] >> https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20c >> bd6a/src/pybind/mgr/mirroring/fs/snapshot_mirror.py#L411 >> >> Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>: >> >>> Hi, >>> >>> This is issue with CephFS snapshot mirroring setup between two >>> clusters and would appreciate your assistance. >>> >>> Environment: >>> >>> * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) >>> [cid:image001.png@01DCD199.2933E2E0] >>> * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) >>> [cid:image002.png@01DCD199.2933E2E0] >>> * Filesystem: cephfs >>> * Issue: Unable to import peer bootstrap token >>> >>> Problem Description: >>> When attempting to import a peer bootstrap token using: >>> ceph fs snapshot mirror peer_bootstrap import cephfs <token> I >>> receive the error: >>> Error EEXIST: already an active peer mgr.server reply reply (17) File >>> exists already an active peer However, ceph fs snapshot mirror >>> peer_list cephfs returns an empty list [], indicating no peers are >>> configured. >>> >>> Steps Performed: >>> Initial Setup: >>> 1. Enabled mirroring module: ceph mgr module enable mirroring >>> 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror >>> 3. Enabled mirroring on filesystem: ceph fs snapshot mirror >>> enable cephfs >>> 4. Created remote user and bootstrap token on backup cluster >>> 5. Verified mirroring daemons are running: ceph fs snapshot >>> mirror daemon status >>> Troubleshooting Attempts: >>> 1. Reset filesystem mirroring: >>> ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror enable >>> cephfs >>> 2. Module reset: >>> ceph mgr module disable mirroring >>> ceph mgr module enable mirroring >>> 3. Service recreation: >>> ceph orch rm cephfs-mirror >>> ceph orch apply cephfs-mirror >>> 4. Checked for stale configuration: >>> o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror >>> o ceph fs snapshot mirror dirmap cephfs returns empty >>> o ceph fs snapshot mirror status cephfs shows no active mirroring >>> >>> Current Status: >>> >>> * Mirroring daemons: Running and visible in daemon status >>> * Peer list: Empty [] >>> * Filesystem mirroring: Enabled >>> * Error persists: EEXIST despite no visible peers >>> >>> Diagnostic Information: >>> >>> * MGR logs show: mgr.server reply reply (17) File exists already >>> an active peer >>> * No connectivity issues between clusters (ports 6789, 3300, >>> 6800+ tested) >>> * Both clusters are healthy with no other issues >>> >>> Regards, >>> Anantha >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Ceph Dashboard fails on tenant RGW buckets (NoSuchBucket → 500)
by Eugen Block 08 May '26

08 May '26

Hi, I think this worked in 19.2.3, but I'm not entirely sure. Have you tried with latest Tentacle (20.2.1) as well? Was the bucket also created via dashboard or by the client? Zitat von farhad kh via ceph-users <ceph-users(a)ceph.io>: > Ceph Dashboard fails with 500 Internal Server Error when operating on RGW > buckets that belong to a *tenant*, even though the bucket exists and is > accessible via radosgw-admin. > > RGW returns NoSuchBucket because the Dashboard queries the bucket *without > the tenant prefix* > ----------- > 13:02:06.832428866Z"} {"log":"debug 2026-05-04T13:02:06.838+0000 > 7fbc29896640 0 [dashboard ERROR rest_client] RGW REST API failed GET req > status: 404\n","stream":"stderr","time":"2026-05-04T13:02:06.841073842Z"} > {"log":"debug 2026-05-04T13:02:06.838+0000 7fbc29896640 0 [dashboard ERROR > exception] Internal Server > Error\n","stream":"stderr","time":"2026-05-04T13:02:06.842785189Z"} > {"log":"Traceback (most recent call > last):\n","stream":"stderr","time":"2026-05-04T13:02:06.842818184Z"} > {"log":" File \"/usr/share/ceph/mgr/dashboard/services/exception.py\", line > 47, in > dashboard_exception_handler\n","stream":"stderr","time":"2026-05-04T13:02:06.842831624Z"} > {"log":" return handler(*args, > **kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.842841016Z"} > {"log":" File \"/lib/python3.9/site-packages/cherrypy/_cpdispatch.py\", > line 54, in > __call__\n","stream":"stderr","time":"2026-05-04T13:02:06.842851035Z"} > {"log":" return self.callable(*self.args, > **self.kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.842983312Z"} > {"log":" File > \"/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py\", line > 264, in inner\n","stream":"stderr","time":"2026-05-04T13:02:06.843004859Z"} > {"log":" ret = func(*args, > **kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.843015376Z"} > {"log":" File > \"/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py\", line > 193, in > wrapper\n","stream":"stderr","time":"2026-05-04T13:02:06.843024549Z"} > {"log":" return func(*vpath, > **params)\n","stream":"stderr","time":"2026-05-04T13:02:06.84303376Z"} > {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\", line > 702, in set\n","stream":"stderr","time":"2026-05-04T13:02:06.843050003Z"} > {"log":" locking = self._get_locking(uid, daemon_name, > bucket_name)\n","stream":"stderr","time":"2026-05-04T13:02:06.843059741Z"} > {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\", line > 443, in > _get_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843069083Z"} > {"log":" return > rgw_client.get_bucket_locking(bucket_name)\n","stream":"stderr","time":"2026-05-04T13:02:06.843078375Z"} > {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 538, > in > func_wrapper\n","stream":"stderr","time":"2026-05-04T13:02:06.843087524Z"} > {"log":" return > func(\n","stream":"stderr","time":"2026-05-04T13:02:06.843096712Z"} > {"log":" File \"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\", > line 899, in > get_bucket_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843105575Z"} > {"log":" raise > e\n","stream":"stderr","time":"2026-05-04T13:02:06.843114638Z"} {"log":" > File \"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\", line 881, in > get_bucket_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843123252Z"} > {"log":" result = request() # type: > ignore\n","stream":"stderr","time":"2026-05-04T13:02:06.843132275Z"} > {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 324, > in __call__\n","stream":"stderr","time":"2026-05-04T13:02:06.843141087Z"} > {"log":" resp = self.rest_client.do_request(method, self._gen_path(), > params,\n","stream":"stderr","time":"2026-05-04T13:02:06.843163504Z"} > {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 422, > in do_request\n","stream":"stderr","time":"2026-05-04T13:02:06.843173809Z"} > {"log":" raise > RequestException(\n","stream":"stderr","time":"2026-05-04T13:02:06.843367851Z"} > {"log":"dashboard.rest_client.RequestException: RGW REST API failed request > with status code > 404\n","stream":"stderr","time":"2026-05-04T13:02:06.843382042Z"} > {"log":"(b'{\"Code\":\"NoSuchBucket\",\"Message\":\"\",\"BucketName\":\"farhad\",\"RequestId\":\"tx000'\n","stream":"stderr","time":"2026-05-04T13:02:06.843391177Z"} > {"log":" > b'00ab80c789f95808ff-0069f898ce-110717032-vanak\",\"HostId\":\"110717032-vanak-teh'\n","stream":"stderr","time":"2026-05-04T13:02:06.843400934Z"} > {"log":" > b'ran\"}')\n","stream":"stderr","time":"2026-05-04T13:02:06.843410095Z"} > {"log":"debug 2026-05-04T13:02:06.842+0000 7fbc29896640 0 [dashboard ERROR > request] [::ffff:172.20.26.54:49876] [PUT] [500] [0.619s] [admin] [513.0B] > /api/rgw/bucket/khedrian%2Ffarhad\n","stream":"stderr","time":"2026-05-04T13:02:06.846732997Z"} > {"log":"debug 2026-05-04T13:02:06.842+0000 7fbc29896640 0 [dashboard ERROR > request] [b'{\"status\": \"500 Internal Server Error\", \"detail\": \"The > server encountered an unexpected condition which prevented it from > fulfilling the request.\", \"request_id\": > \"a3abea21-d73c-4d26-a8bb-1d99fc53a466\"} > ']\n","stream":"stderr","time":"2026-05-04T13:02:06.847608634Z"} > {"log":"debug 2026-05-04T13:02:06.846+0000 7fbc29896640 0 [dashboard INFO > request] [::ffff:172.20.26.54:49876] [PUT] [500] [0.622s] [admin] [513.0B] > /api/rgw/bucket/khedrian%2Ffarhad\n","stream":"stderr","time":"2026-05-04T13:02:06.849427429Z"} > {"log":"debug 2026-05-04T13:02:07.322+0000 7fbc28053640 0 [dashboard INFO > request] [::ffff:172.20.26.54:64634] [GET] [200] [0.691s] [admin] [295.0B] > /api/rgw/daemon\n","stream":"stderr","time":"2026-05-04T13:02:07.324402049Z"} > {"log":"debug 2026-05-04T13:02:07.850+0000 7fbc2504d640 0 [dashboard INFO > request] [::ffff:172.20.26.54:49870] [GET] [200] [0.004s] [admin] [22.0B] > /api/prometheus/notifications\n","stream":"stderr","time":"2026-05-04T13:02:07.855794264Z"} > {"log":"debug 2026-05-04T13:02:08.110+0000 7fbc5c2fb640 0 > log_channel(cluster) log [DBG] : pgmap v611376: 449 pgs: 449 active+clean; > 202 MiB data, 10 GiB used, 1.2 TiB / 1.2 TiB avail; 6.0 KiB/s rd, 0 B/s wr, > 8 op/s\n","stream":"stderr","time":"2026-05-04T13:02:08.113933657Z"} > {"log":"debug 2026-05-04T13:02:10.111+0000 7fbc5c2fb640 0 > log_channel(cluster) log [DBG] : pgmap v611377: 449 pgs: 449 active+clean; > 202 MiB data, 10 GiB used, 1.2 TiB / 1.2 TiB avail; 5.7 KiB/s rd, 0 B/s wr, > 8 op/s\n","stream":"stderr","time":"2026-05-04T13:02:10.115433742Z"} > {"log":"debug 2026-05-04T13:02:10.211+0000 7fbc2484c640 0 [dashboard INFO > request] [::ffff:172.20.26.54:49876] [GET] [200] [0.003s] [admin] [22.0B] > /api/prometheus/notifications\n","stream":"stderr","time":"2026-05-04T13:02:10.2151369Z"} > ---------------- > root@dev-ceph01:~# ceph -v > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle > (stable) > > { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", > "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": > "arn:aws:s3:::khedrian:farhad/*" } ] } > > > - Bucket exists and is accessible via radosgw-admin > - Issue only occurs for *tenant buckets* > - Non‑tenant buckets work correctly > - Appears to be a *Dashboard RGW tenant parsing bug* > - Possibly missing tenant propagation when calling get_bucket_locking() and > similar RGW APIs > can anyone explain this ? what should i do ? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 2

InfluxDB deprecated?
by Robert Sander 07 May '26

07 May '26

Hi, Going from 16.2.15 in packages to cephadm managed containers and upgrading to 18.2.8 we find that the InfluxDB MGR module does not work any more: # ceph mgr module enable influx Error ENOENT: module 'influx' reports that it cannot run on the active manager daemon: influxdb python module not found (pass --force to force enablement) The container image does not contain the necessary Python module. There is a tracker issue open for 18 months now: https://tracker.ceph.com/issues/68892 Has the InfluxDB support been deprecated? Why is it still in the documentation for recent Ceph versions? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Adiga, Anantha 07 May '26

07 May '26

Hi Eugen, Thank you so much. Yes, that fixed the issue: removed the peer uuid from the primary cluster. This removed the attribute from the backup cluster as well. Then import peer bootstrap token worked. Regards, Anantha -----Original Message----- From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Thursday, April 23, 2026 12:56 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty Hi, a quick look into the code [0] shows (note that I'm not a developer): def set_mirror_info(local_cluster_id, local_fsid, remote_fs): log.info(f'setting {local_cluster_id}::{local_fsid} on remote') try: remote_fs.setxattr('/', 'ceph.mirror.info', f'cluster_id={local_cluster_id} fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE) except cephfs.Error as e: if e.errno == errno.EEXIST: My interpretation is that the setxattr command fails on the remote site. Could this be an auth caps issue? Did you verify the keyrings and their permissions? This is the only lead I got. Regards, Eugen [0] https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20cbd6a/… Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>: > Hi, > > This is issue with CephFS snapshot mirroring setup between two > clusters and would appreciate your assistance. > > Environment: > > * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) > [cid:image001.png@01DCD199.2933E2E0] > * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) > [cid:image002.png@01DCD199.2933E2E0] > * Filesystem: cephfs > * Issue: Unable to import peer bootstrap token > > Problem Description: > When attempting to import a peer bootstrap token using: > ceph fs snapshot mirror peer_bootstrap import cephfs <token> I receive > the error: > Error EEXIST: already an active peer mgr.server reply reply (17) File > exists already an active peer However, ceph fs snapshot mirror > peer_list cephfs returns an empty list [], indicating no peers are > configured. > > Steps Performed: > Initial Setup: > 1. Enabled mirroring module: ceph mgr module enable mirroring > 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror > 3. Enabled mirroring on filesystem: ceph fs snapshot mirror > enable cephfs > 4. Created remote user and bootstrap token on backup cluster > 5. Verified mirroring daemons are running: ceph fs snapshot > mirror daemon status > Troubleshooting Attempts: > 1. Reset filesystem mirroring: > ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror enable > cephfs > 2. Module reset: > ceph mgr module disable mirroring > ceph mgr module enable mirroring > 3. Service recreation: > ceph orch rm cephfs-mirror > ceph orch apply cephfs-mirror > 4. Checked for stale configuration: > o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror > o ceph fs snapshot mirror dirmap cephfs returns empty > o ceph fs snapshot mirror status cephfs shows no active mirroring > > Current Status: > > * Mirroring daemons: Running and visible in daemon status > * Peer list: Empty [] > * Filesystem mirroring: Enabled > * Error persists: EEXIST despite no visible peers > > Diagnostic Information: > > * MGR logs show: mgr.server reply reply (17) File exists already > an active peer > * No connectivity issues between clusters (ports 6789, 3300, > 6800+ tested) > * Both clusters are healthy with no other issues > > Regards, > Anantha _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Ceph Steering Committee (CSC) Election Report 2026
by Anthony Middleton 06 May '26

06 May '26

Ceph Steering Committee (CSC) Election Report 2026 The CSC 2026 - Q1 election was conducted according to the guidelines set forth in section two, part six of the CSC Charter (Adopted Feb 12, 2026), available at https://github.com/ceph/ceph/blob/main/doc/technical-charter.rst. With support and input from current CSC members (Joseph Mundackal, Mark Nelson, Neha Ojha, and Patrick Donnelly), the next CSC election is scheduled for October 2026. This election served as the makeup for the missing 2025 election. This election cycle ran from March 19 through April 29, with results announced on May 4. The election was administered by Anthony Middleton, Ceph Community Manager, using ElectionBuddy as the voting platform, following a recommendation from the Linux Foundation. This report outlines the structure of the election, participation levels, outcomes, and key observations from both stages. Voting Tool ElectionBuddy ElectionBuddy was selected to manage the 2026 CSC election. The platform supported a multi-stage voting process and introduced capabilities that aligned more closely with CSC governance needs. Highlights - Supports ranked-choice voting, enabling more flexible and representative outcomes for leadership roles - Streamlined ballot distribution and tracking, with automated email delivery and vote confirmations - Clear audit trails and real-time result visibility for election administrators - Configurable ballots, allowing multiple question types across different election stages Election Structure The 2026 CSC election was conducted in two stages. Stage 1: Nominations and Governance Updates Stage 1 included two questions: 1. Governance Update (Removal Clause) This question proposed updating the CSC member responsibilities to include a removal clause. Due to unclear wording, the question caused confusion among voters and was ultimately discarded. Following a discussion at the March 16th CSC meeting <https://pad.ceph.com/p/csc-weekly-minutes>, it was agreed that Anthony Middleton would rewrite the removal section for clarity before reintroducing it as a GitHub RP. 2. Co-Chair Nominations Voting members were asked to nominate up to three CSC members for the Co-Chair ballot. A list of all active CSC members was provided. Participation - 21 ballots were submitted out of 36 eligible voters (58%) Outcome and Observations - Results from the governance question were not counted - Nomination results were retained and used to form the final ballot - There was confusion about the structure of the election, with some participants believing Stage 1 was the final vote - Several members did not receive ballots due to outdated contact information on the CSC Governance page - Voting was reopened for an additional three days to account for missed participation Following Stage 1, outreach was conducted to all individuals nominated. Of the 11 members nominated, 6 accepted placement on the final ballot. A new addition to this election cycle was the introduction of candidate statements. Each candidate was given up to two minutes during a CSC meeting to speak about their candidacy. This was the first time this format was used and was received positively by CSC members. Election Data ElectionBuddy Repost of Stage 1 <https://docs.google.com/spreadsheets/d/1E1H9GAO9PNGZ-Sl_bOb7PCzEaXhTKPDqZpy…> Stage 2: Co-Chair Election and Governance Votes Stage 2 ran for two weeks and included: - Two governance questions related to CSC membership removal - One ranked-choice vote to elect CSC Co-Chairs Participation - 27 ballots submitted out of 36 eligible voters (75%) - 1 spoiled ballot recorded Election Data - ElectionBuddyReport - Results <https://docs.google.com/spreadsheets/d/1Q3_jtNfrYcjg77_DGKEYY7-mxXwXTSKvlio…> - ElectionBuddyReport -VoteByVote <https://docs.google.com/spreadsheets/d/1YSw7fSxSO6ucJq58MZAEXhqA7pQE7qJdibY…> - ElectionBuddyReport - VoterStatus (voter participation by email) <https://docs.google.com/spreadsheets/d/1rg0LsAnNgBmIzJtkgSDY7aSA1R0mCSqsMfo…> Votes for removal of two members CSC members determined that these members were seen as inactive. - Ken Dreyer Removal Vote - 27 votes in favor - 0 abstentions - Jeff Layton Removal Vote - 25 votes in favor - 2 abstentions Co-Chair Election Results A ranked voting system was used to elect up to three Co-Chairs. A governance constraint was applied to ensure that no more than two Co-Chairs could come from the same organization. Vote Distribution - Patrick Donnelly — 20 votes (27.40%) - Neha Ojha — 16 votes (21.92%) - Dan van der Ster — 14 votes (19.18%) - Joseph Mundackal — 11 votes (15.07%) - Ernesto Puerta — 6 votes (8.22%) - Gregory Farnum — 6 votes (8.22%) Outcome The top three vote-getters were selected as CSC Co-Chairs, subject to the rule on organizational representation. Observations Stage 2 showed clear improvement over Stage 1: - Ballot structure and questions were better understood - No significant confusion was reported during voting - Participation increased from 58% to 75% However, some challenges remained: - A portion of eligible voters did not participate despite multiple follow-ups via email and Slack - Delayed responses from non-participating members slowed the overall process Summary The 2026 CSC election introduced several meaningful improvements, including ranked-choice voting and candidate statements, both of which contributed to a more transparent and participatory process. While Stage 1 revealed gaps in communication and ballot clarity, those issues were addressed in Stage 2, resulting in stronger engagement and cleaner execution. Future election cycles would benefit from clearer ballot language upfront, enforced updates to voter contact information, and continued use of candidate advocacy opportunities. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Ceph deployment for HPC and research workflows
by Mark Nelson 06 May '26

06 May '26

Hi Max, Probably the first order of business is to understand the nature of the crashes you are seeing. Just as a guess, if you have thousands of jobs being run at the same time, you may have a lot of memory pressure on the MDSses. If the crashes are just the MDSes going OOM, you might want to verify how much of the 192GB of RAM on each node the MDSes are allowed to use for cache. Beyond that, it might take some digging to understand the specifics of your issues. There are other institutions that do use CephFS for HPC use cases though. I gave a talk at the IO500 BoF at SC24 in collaboration with GWDG in Germany regarding their Ceph deployment. Shameless plug: Clyso does offer academic support packages for upstream Ceph if that is of interest to you. Thanks, Mark On 5/6/26 6:54 AM, Max Breitmeyer via ceph-users wrote: > Hello all, > At UMBC we were given a grant to run a ceph cluster as our primary research > storage for an HPC facility. The ceph cluster consists of the following > hardware: > Mon” nodes x3: > • 2x 25Gb interfaces > • 192GB RAM > • Storage: > -- 0 (none) OSDs/drives > “HDD” nodes x16: > • 2x 25Gb interfaces > • 192GB RAM > • Storage: > -- 12 OSDs w/ 20TB HDDs > -- 2 local Write-Access-Logs (journal) w/ 8TB NVMe drives > “NVMe” nodes x3: > • 2x 100Gb interfaces > • 384GB RAM > • Storage: > -- 16x OSDs w/ 8TB NVMe drives > “MDS” nodes x3: > • 2x 100Gb interfaces > • 192GB RAM > • Storage: > -- 8x OSDs w/ 1TB NVMe drives > > > In the six months since moving from a different storage solution (isilon), > we've had multiple crashes on the system that have completely taken down > our system as a result of multiple thousands of jobs being run at the same > time, causing an overload on the active MDS nodes. Has the use of ceph as a > primary research storage been used in an HPC with many dozens of different > workflows and filetypes being used at once? Or is a better to design to > come up with a tiered approach in combination with a second storage > solution? > -- Best Regards, Mark Nelson Co-Founder and Head of R&D Clyso Inc. p: +49 89 21552391 12 | a: North Vancouver, B.C. w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/

1 0

Ceph deployment for HPC and research workflows
by Max Breitmeyer 06 May '26

06 May '26

Hello all, At UMBC we were given a grant to run a ceph cluster as our primary research storage for an HPC facility. The ceph cluster consists of the following hardware: Mon” nodes x3: • 2x 25Gb interfaces • 192GB RAM • Storage: -- 0 (none) OSDs/drives “HDD” nodes x16: • 2x 25Gb interfaces • 192GB RAM • Storage: -- 12 OSDs w/ 20TB HDDs -- 2 local Write-Access-Logs (journal) w/ 8TB NVMe drives “NVMe” nodes x3: • 2x 100Gb interfaces • 384GB RAM • Storage: -- 16x OSDs w/ 8TB NVMe drives “MDS” nodes x3: • 2x 100Gb interfaces • 192GB RAM • Storage: -- 8x OSDs w/ 1TB NVMe drives In the six months since moving from a different storage solution (isilon), we've had multiple crashes on the system that have completely taken down our system as a result of multiple thousands of jobs being run at the same time, causing an overload on the active MDS nodes. Has the use of ceph as a primary research storage been used in an HPC with many dozens of different workflows and filetypes being used at once? Or is a better to design to come up with a tiered approach in combination with a second storage solution? -- V/R, Maxwell Breitmeyer UMBC HPCF Specialist Graduate Student (443) 835-8250

1 0

Re: could not find secret_id
by Eugen Block 06 May '26

06 May '26

Hi, I'm quite sure that we didn't investige MONs at that time, so no, we haven't tried a leader failover. But it might be possible that they rebooted nodes, so that would have eventually led to a failover. I remember that we also did try to rotate keys, that I do have in my email history, but it didn't have any impact. Waiting also didn't help, sorry for not having more memories of that incident. :-D But it's interesting that you could reproduce it with the mon leader, I wouldn't have expected that. Zitat von Andrea Martra via ceph-users <ceph-users(a)ceph.io>: > Hi Eugen, > Thanks a lot for chiming in — even a partial recollection helps > narrow the search space. > > A few notes from our side, in case it helps differentiate the cases: > > - No rbd-mirroring at the customer; single cluster, 39 nodes on Reef 18.2.x. > - Pattern affects 4 OSDs on 7 out of 39 nodes, persistent over days > (15M+ log lines in 3 days), no clock skew, quorum stable. > - We reproduced a very similar log signature on a small test cluster > (Reef 18.2.7) by isolating a node hosting MON leader + OSDs for >3h, > matching the original xiaowenhao111 scenario. On our test cluster > the affected OSDs eventually self-recovered after a couple of > key-rotation cycles. > - At the customer, the same signature is NOT clearing on its own. > > Two questions on your prior case, if anything comes back to you: > > 1. Was zap-and-readd actually required, or was it more of a > defensive cleanup after another action (e.g. a MON failover, key > rotation, or simply waiting)? In other words: did you ever observe > the OSDs unstick on their own, or were they truly stuck? > 2. Do you remember whether a MON leader failover alone was tried > before going to OSD rebuild, and whether it had any effect? > > The fact that our reproducer self-recovered while the production > case doesn't suggests there's something cluster-specific keeping the > rotation desynchronised — possibly stale state on the affected nodes > rather than a pure timing/quorum effect, which would line up with > your "remainder from previous tests" intuition. > > If nobody points to an existing tracker, we're prepared to open one > with logs from both the reproduction and the customer environment. > > Thanks again, > Andrea > > Il 24/04/26 22:06, Eugen Block via ceph-users ha scritto: >> Hi, >> >> how did you end up in that situation (did something happen in the >> cluster)? Is it affecting only OSDs or other daemons as well? >> >> We did investigate such an issue one and a half years ago for a >> customer. They added hosts to one of the two clusters, at that time >> the customer had two sites, with rbd mirroring enabled. And >> I don't recall the exact details, we did solve it in an online >> meeting and I don't think anyone wrote a protocoll. And the session >> definitely wasn't recored. :-) It might have some remainder on the >> new host from previous tests, or some misconfiguration, I really >> can't tell anymore. I think in the end we ended up removing the >> OSDs, zapping them and adding them back. Of course, it could be a >> different root cause, not sure which role the mon leader can play >> in this. But I thought I'd share this tiny bit at least. :-) >> >> Regards, >> Eugen >> >> >> Zitat von Andrea Martra via ceph-users <ceph-users(a)ceph.io>: >> >>> Hi all, >>> I'm reaching back to this old thread from December 2023 because >>> we're investigating a very similar pattern at a customer site >>> running Ceph Reef 18.2.x. Logs show identical shape: >>> >>> AuthRegistry get_handler peer_type 4 method 2 ... >>> cephx: verify_authorizer decrypted service osd secret_id=3524 >>> auth: could not find secret_id=3524 >>> auth: dump_rotating: (local range ~971-973) >>> cephx: verify_authorizer could not get service secret ... >>> >>> Affecting 4 OSDs on 7 out of 39 nodes, persistent over days >>> (15M+ log lines in 3 days). No clock skew, quorum stable. >>> >>> @xiaowenhao111 or anyone else who hit this: did you ever >>> find a root cause or a reliable fix? >>> The original thread >>> didn't seem to reach a conclusion. Specifically: >>> >>> - Did restarting the affected OSDs clear it, or did the >>> pattern come back? >>> - Was a MON leader failover enough, or did you need to >>> rebuild anything? >>> - Any tracker issue filed for this? >>> >>> We're trying to reproduce on a test cluster (Reef 18.2.7) >>> by isolating a node hosting MON leader + OSDs via network >>> for 3+ hours, to match the scenario described in the >>> original thread. Happy to share findings back to the list. >>> >>> Thanks, >>> Andrea >>> >>> Il 21/12/23 02:42, xiaowenhao111 ha scritto: >>>> >>>> >>>> Hello ceph-users: >>>> >>>> First ,sorry my english ... >>>> >>>> >>>> I find a bug (16.2.10 <tel:16210> and 18.2.1), but i do not know why >>>> >>>> test steps: >>>> >>>> 1、config all hosts contain chronyd service >>>> >>>> 2、cephadm --image quay.io/ceph/ceph:v16.2.10 >>>> <http://quay.io/ceph/ceph:v16.2.10> bootstrap >>>> --dashboard-password-noupdate --mon-ip 10.40.10.200 >>>> <http://10.40.10.200> --cluster-network=10.40.10.0/24 >>>> <http://10.40.10.0/24> --skip-pull --allow-overwrite >>>> --skip-monitoring-stack --allow-mismatched-release >>>> >>>> 3、add hosts >>>> >>>> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node2 >>>> >>>> ceph orch host add node2 10.40.10.202 <tel:2104010202> --labels=_admin >>>> >>>> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node-1 >>>> >>>> ceph orch host add node-1 10.40.10.202 >>>> <tel:1104010202> --labels=_admin >>>> >>>> 4、add osds: >>>> >>>> ceph orch daemon add osd node1:/dev/sda >>>> >>>> ceph orch daemon add osd node1:/dev/sdb >>>> >>>> ceph orch daemon add osd node1:/dev/sdc >>>> >>>> ceph orch daemon add osd node1:/dev/sdd >>>> >>>> ceph orch daemon add osd node2:/dev/sda >>>> >>>> ceph orch daemon add osd node2:/dev/sdb >>>> >>>> ceph orch daemon add osd node2:/dev/sdc >>>> >>>> ceph orch daemon add osd node2:/dev/sdd >>>> >>>> 5、ceph osd pool create test_pool 512 512 <tel:512512> --size=3 >>>> >>>> 6、disconnect node1 (which is monitor leader) cluster and public network >>>> >>>> ifdown enp35s0f0 >>>> >>>> ifdown enp35s0f1 >>>> >>>> 7、stay overnight（more than three hours） >>>> >>>> 8、connect node1 cluster and public network >>>> >>>> ifup enp35s0f0 >>>> >>>> ifup enp35s0f1 >>>> >>>> >>>> then, the osds of node1 call get_auth_session_key from mon.node1 >>>> which is not new leader，get the wrong secret_id=4. >>>> >>>> the others report error "could not find secret_id=4" more than 40 minutes >>>> >>>> >>>> >>>> >>>> the ceph.log content of node1: >>>> >>>> >>>> 2023-12-20 <tel:20231220>T09:48:30.199224 <tel:30199224>+0800 >>>> mon.node1 (mon.0) 1891 <tel:01891> : cluster [WRN] Health check >>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>>> >>>> 2023-12-20 <tel:20231220>T09:49:37.816260 <tel:37816260>+0800 >>>> mon.node1 (mon.0) 1901 <tel:01901> : cluster [INF] Health check >>>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 >>>> daemon(s)) >>>> >>>> 2023-12-20 <tel:20231220>T09:49:44.222046 <tel:44222046>+0800 >>>> mon.node1 (mon.0) 1902 <tel:01902> : cluster [WRN] Health check >>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>>> >>>> 2023-12-20 <tel:20231220>T09:50:00.000090 <tel:00000090>+0800 >>>> mon.node1 (mon.0) 1905 <tel:01905> : cluster [WRN] overall >>>> HEALTH_WARN Failed to place 1 daemon(s); 3 failed cephadm >>>> daemon(s); 1 pool(s) do not have an application enabled >>>> >>>> 2023-12-20 <tel:20231220>T09:50:51.916464 <tel:51916464>+0800 >>>> mon.node1 (mon.0) 1912 <tel:01912> : cluster [INF] Health check >>>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 >>>> daemon(s)) >>>> >>>> 2023-12-20 <tel:20231220>T09:51:07.127982 <tel:07127982>+0800 >>>> mon.node1 (mon.0) 1916 <tel:01916> : cluster [WRN] Health check >>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 >>>> osd.0 (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, >>>> but it is still running >>>> >>>> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >>>> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 >>>> calling monitor election >>>> >>>> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >>>> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >>>> calling monitor election >>>> >>>> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >>>> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >>>> election >>>> >>>> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >>>> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 >>>> calling monitor election >>>> >>>> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >>>> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is >>>> new leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >>>> >>>> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >>>> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >>>> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >>>> >>>> >>>> the ceph.log content of node2: >>>> >>>> >>>> 2023-12-21 <tel:20231221>T01:10:01.065051 <tel:01065051>+0800 >>>> mon.node2 (mon.1) 11522 <tel:111522> : cluster [WRN] Health check >>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>>> >>>> 2023-12-21 <tel:20231221>T01:11:11.092640 <tel:11092640>+0800 >>>> mon.node2 (mon.1) 11531 <tel:111531> : cluster [INF] Health check >>>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 >>>> daemon(s)) >>>> >>>> 2023-12-21 <tel:20231221>T01:11:19.197355 <tel:19197355>+0800 >>>> mon.node2 (mon.1) 11532 <tel:111532> : cluster [WRN] Health check >>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>>> >>>> 2023-12-21 <tel:20231221>T01:12:34.469403 <tel:34469403>+0800 >>>> mon.node2 (mon.1) 11547 <tel:111547> : cluster [INF] Health check >>>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 >>>> daemon(s)) >>>> >>>> 2023-12-21 <tel:20231221>T01:12:42.476793 <tel:42476793>+0800 >>>> mon.node2 (mon.1) 11549 <tel:111549> : cluster [WRN] Health check >>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.234163 <tel:55234163>+0800 >>>> mon.node2 (mon.1) 11572 <tel:111572> : cluster [INF] osd.5 marked >>>> itself dead as of e91 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.236348 <tel:55236348>+0800 >>>> mon.node2 (mon.1) 11573 <tel:111573> : cluster [INF] osd.4 marked >>>> itself dead as of e91 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.242209 <tel:55242209>+0800 >>>> mon.node2 (mon.1) 11574 <tel:111574> : cluster [INF] osd.0 marked >>>> itself dead as of e91 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.233774 <tel:55233774>+0800 >>>> osd.5 (osd.5) 3 : cluster [WRN] Monitor daemon marked osd.5 down, >>>> but it is still running >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.236038 <tel:55236038>+0800 >>>> osd.4 (osd.4) 3 : cluster [WRN] Monitor daemon marked osd.4 down, >>>> but it is still running >>>> >>>> 2023-12-21 <tel:20231221>T01:14:00.109136 <tel:109136>+0800 >>>> mon.node2 (mon.1) 11589 <tel:111589> : cluster [WRN] Health check >>>> update: Degraded data redundancy: 1/8 objects degraded (12.500 >>>> <tel:12500>%), 1 pg degraded, 188 pgs undersized (PG_DEGRADED) >>>> >>>> 2023-12-21 <tel:20231221>T01:14:00.311644 <tel:311644>+0800 >>>> mon.node2 (mon.1) 11590 <tel:111590> : cluster [INF] osd.3 marked >>>> itself dead as of e94 >>>> >>>> 2023-12-21 <tel:20231221>T01:14:00.311410 <tel:311410>+0800 osd.3 >>>> (osd.3) 3 : cluster [WRN] Monitor daemon marked osd.3 down, but >>>> it is still running >>>> >>>> 2023-12-21 <tel:20231221>T01:14:01.922553 <tel:01922553>+0800 >>>> mgr.node1.apderh (mgr.34343) 1 <tel:343431> : cluster [ERR] >>>> Failed to load ceph-mgr modules: k8sevents >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 >>>> osd.0 (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, >>>> but it is still running >>>> >>>> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >>>> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 >>>> calling monitor election >>>> >>>> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >>>> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >>>> calling monitor election >>>> >>>> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >>>> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >>>> election >>>> >>>> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >>>> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 >>>> calling monitor election >>>> >>>> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >>>> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is >>>> new leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >>>> >>>> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >>>> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >>>> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >>>> >>>> >>>> >>>> the osd log of node1: >>>> >>>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >>>> 10.40.10.202:6804 <http://10.40.10.202:6804> osd.6 ever on either >>>> front or back, first ping sent 2023-12-21 >>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest >>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457 >>>> <tel:43148457>+0800) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >>>> 10.40.10.202:6812 <http://10.40.10.202:6812> osd.7 ever on either >>>> front or back, first ping sent 2023-12-21 >>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest >>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457 >>>> <tel:43148457>+0800) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >>>> 10.40.10.202:6820 <http://10.40.10.202:6820> osd.8 ever on either >>>> front or back, first ping sent 2023-12-21 >>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest >>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457 >>>> <tel:43148457>+0800) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >>>> 10.40.10.202:6828 <http://10.40.10.202:6828> osd.9 ever on either >>>> front or back, first ping sent 2023-12-21 >>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest >>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457 >>>> <tel:43148457>+0800) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: reset >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: 0x562066 >>>> <tel:09562066>bbc000 handle_response ret = 0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: got initial server >>>> challenge 5fd5d9eca256ae69 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: validate_tickets: >>>> want=53 need=0 have=53 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have >>>> 0 need 53 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: build_request >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 20 <tel:864020> cephx client: old ticket len=96 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: get auth session key: >>>> client_challenge 7ca3f7c3ef30b2bf >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: 0x562066 >>>> <tel:09562066>bbc000 handle_response ret = 0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: get_auth_session_key >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> got 1 keys >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id auth >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> got encrypted ticket >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> ticket.secret_id=2 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> service auth secret_id 2 session_key >>>> AQDIIINlUlSmKBAAvL5rKyyIb6vt89rxUGlsxg== validity=259200.000000 >>>> <tel:259200000000> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> ticket expires=2023-12-24 <tel:20231224>T01:13:44.682217 >>>> <tel:44682217>+0800 renew_after=2023-12-23 >>>> <tel:20231223>T07:13:44.682217 <tel:44682217>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: got connection bl 84 >>>> and extra tickets 550 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: got connection_secret >>>> 64 bytes >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> got 3 keys >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mon >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> ticket.secret_id=4 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> service mon secret_id 4 session_key >>>> AQDIIINlWvamKBAAxOaZsuCMdgutax1n2ZRbZQ== validity=3600.000000 >>>> <tel:3600000000> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682256 >>>> <tel:44682256>+0800 renew_after=2023-12-21 >>>> <tel:20231221>T01:58:44.682256 <tel:44682256>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id osd >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> ticket.secret_id=4 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> service osd secret_id 4 session_key >>>> AQDIIINlXAinKBAAOwTlViH3CJRgSvj5Kn9qtA== validity=3600.000000 >>>> <tel:3600000000> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682270 >>>> <tel:44682270>+0800 renew_after=2023-12-21 >>>> <tel:20231221>T01:58:44.682270 <tel:44682270>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mgr >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> ticket.secret_id=4 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> service mgr secret_id 4 session_key >>>> AQDIIINlUiGnKBAAr6WBaSawzVyDoqEs8cENdg== validity=3600.000000 >>>> <tel:3600000000> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >>>> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682286 >>>> <tel:44682286>+0800 renew_after=2023-12-21 >>>> <tel:20231221>T01:58:44.682286 <tel:44682286>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: got extra >>>> service_tickets >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have >>>> 53 need 0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have >>>> 53 need 0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 20 <tel:864020> cephx client: need_tickets: want=53 >>>> have=53 need=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 -1 <tel:86401> monclient: _check_auth_rotating >>>> possible clock skew, rotating keys expired way too early (before >>>> 2023-12-21 <tel:20231221>T00:13:44.682376 <tel:44682376>+0800) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >>>> 7fcd2bed8640 10 <tel:864010> cephx client: build_rotating_request >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 >>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >>>> 10.40.10.201:6804 <http://10.40.10.201:6804> osd.1 ever on either >>>> front or back, first ping sent 2023-12-21 >>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest >>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457 >>>> <tel:43148457>+0800) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 >>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >>>> 10.40.10.201:6812 <http://10.40.10.201:6812> osd.2 ever on either >>>> front or back, first ping sent 2023-12-21 >>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest >>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457 >>>> <tel:43148457>+0800) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 >>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >>>> 10.40.10.200:6814 <http://10.40.10.200:6814> osd.3 ever on either >>>> front or back, first ping sent 2023-12-21 >>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest >>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457 >>>> <tel:43148457>+0800) >>>> >>>> >>>> >>>> >>>> the osd log of node2: >>>> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >>>> 7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have >>>> 53 need 0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >>>> 7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53 >>>> have=53 need=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >>>> 7fa573a6d640 10 <tel:64010> auth: dump_rotating: >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >>>> 7fa573a6d640 10 <tel:64010> auth: id 17 >>>> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21 >>>> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 >>>> 7fa573a6d640 10 <tel:64010> auth: id 18 >>>> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21 >>>> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 >>>> 7fa573a6d640 10 <tel:64010> auth: id 19 >>>> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21 >>>> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>>> 7fa581293640 20 <tel:58129364020> AuthRegistry(0x7ffc86c1e7b0) >>>> get_handler peer_type 4 method 2 cluster_methods [2] >>>> service_methods [2] client_methods [2] >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>>> 7fa581293640 10 <tel:58129364010> cephx: verify_authorizer >>>> decrypted service osd secret_id=4 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>>> 7fa581293640 0 <tel:5812936400> auth: could not find secret_id=4 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>>> 7fa581293640 10 <tel:58129364010> auth: dump_rotating: >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>>> 7fa581293640 10 <tel:58129364010> auth: id 17 >>>> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21 >>>> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>>> 7fa581293640 10 <tel:58129364010> auth: id 18 >>>> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21 >>>> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>>> 7fa581293640 10 <tel:58129364010> auth: id 19 >>>> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21 >>>> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>>> 7fa581293640 0 <tel:5812936400> cephx: verify_authorizer could >>>> not get service secret for service osd secret_id=4 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 >>>> 7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have >>>> 53 need 0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 >>>> 7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53 >>>> have=53 need=0 >>>> >>>> >>>> >>>> the mon log of node1: >>>> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> start_session >>>> entity_name=mgr.node1.apderh global_id=14184 >>>> <tel:14184> is_new_global_id=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >>>> start_session server_challenge 524ce5eea3e4dd75 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >>>> handle_request get_auth_session_key for mgr.node1.apderh >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh: >>>> checking key: req.key=f41952 <tel:41952>c0f02e6c0f >>>> expected_key=f41952 <tel:41952>c0f02e6c0f >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh: >>>> checking old_ticket: secret_id=2 len=112, >>>> old_ticket_may_be_omitted=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh: >>>> decoded old_ticket: global_id=14184 <tel:14184> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >>>> allowing reclaim of global_id 14184 <tel:14184> (valid ticket >>>> presented, will encrypt new ticket) >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply >>>> encoding 1 tickets with secret >>>> AQBmKIJlvHeYJRAAf5h+auT4c9Ilo/nlUqRJlg== >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >>>> auth secret_id 2 ticket_info.ticket.name >>>> <http://ticket_info.ticket.name>=mgr.node1.apderh >>>> ticket.global_id 14184 <tel:14184> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >>>> name=mgr.node1.apderh >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >>>> of caps=3 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >>>> adding key for service mon >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >>>> adding key for service mds >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >>>> name=mgr.node1.apderh >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >>>> of caps=3 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >>>> adding key for service osd >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >>>> name=mgr.node1.apderh >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >>>> of caps=3 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >>>> adding key for service mgr >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >>>> name=mgr.node1.apderh >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >>>> of caps=3 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply >>>> encoding 4 tickets with secret >>>> AQDIIINldY1mGRAApH04rNRZtuyAUrQn5PNG7A== >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >>>> mon secret_id 4 ticket_info.ticket.name >>>> <http://ticket_info.ticket.name>=mgr.node1.apderh >>>> ticket.global_id 14184 <tel:14184> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >>>> mds secret_id 4 ticket_info.ticket.name >>>> <http://ticket_info.ticket.name>=mgr.node1.apderh >>>> ticket.global_id 14184 <tel:14184> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >>>> osd secret_id 4 ticket_info.ticket.name >>>> <http://ticket_info.ticket.name>=mgr.node1.apderh >>>> ticket.global_id 14184 <tel:14184> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >>>> mgr secret_id 4 ticket_info.ticket.name >>>> <http://ticket_info.ticket.name>=mgr.node1.apderh >>>> ticket.global_id 14184 <tel:14184> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >>>> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 >>>> handle_command mon_command({"prefix":"config >>>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"} v 0) >>>> v1 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >>>> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >>>> from='mgr.14184 10.40.10.200:0/2514043697 >>>> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >>>> cmd=[{"prefix":"config >>>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"}]: >>>> dispatch >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >>>> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 >>>> handle_command mon_command({"prefix":"config >>>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"} v 0) >>>> v1 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >>>> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >>>> from='mgr.14184 10.40.10.200:0/2514043697 >>>> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >>>> cmd=[{"prefix":"config >>>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"}]: >>>> dispatch >>>> >>>> >>>> >>>> the mon log of node2: >>>> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:10.292 <tel:10292>+0800 >>>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >>>> {"time_micros": 1703092390293629, "job": 1145, "event": >>>> "table_file_deletion", "file_number": 2272} >>>> >>>> 2023-12-21 <tel:20231221>T01:13:10.295 <tel:10295>+0800 >>>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >>>> {"time_micros": 1703092390296385, "job": 1145, "event": >>>> "table_file_deletion", "file_number": 2270} >>>> >>>> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >>>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >>>> {"time_micros": 1703092390304703, "job": 1145, "event": >>>> "table_file_deletion", "file_number": 2269} >>>> >>>> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >>>> 7f88b66bb640 4 rocksdb: >>>> [db_impl/db_impl_compaction_flush.cc:1442 >>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>>> compaction starting >>>> >>>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>>> 7f88b66bb640 4 rocksdb: >>>> [db_impl/db_impl_compaction_flush.cc:1442 >>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>>> compaction starting >>>> >>>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>>> 7f88b66bb640 4 rocksdb: >>>> [db_impl/db_impl_compaction_flush.cc:1442 >>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>>> compaction starting >>>> >>>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>>> 7f88b66bb640 4 rocksdb: >>>> [db_impl/db_impl_compaction_flush.cc:1442 >>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>>> compaction starting >>>> >>>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>>> 7f88b66bb640 4 rocksdb: >>>> [db_impl/db_impl_compaction_flush.cc:1442 >>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>>> compaction starting >>>> >>>> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 >>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>>> _set_new_cache_sizes cache_size:1020054731 >>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 >>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >>>> prepare_rotating_update before: data.rotating_ver=17 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 >>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>>> _set_new_cache_sizes cache_size:1020054731 >>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 >>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >>>> prepare_rotating_update before: data.rotating_ver=17 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 >>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>>> _set_new_cache_sizes cache_size:1020054731 >>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 >>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >>>> prepare_rotating_update before: data.rotating_ver=17 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 >>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>>> _set_new_cache_sizes cache_size:1020054731 >>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 >>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >>>> prepare_rotating_update before: data.rotating_ver=17 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>>> mon_command({"prefix":"config >>>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"} v 0) >>>> v1 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >>>> from='mgr.14252 10.40.10.201:0/357060750 >>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>>> cmd=[{"prefix":"config >>>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"}]: >>>> dispatch >>>> >>>> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>>> mon_command({"prefix":"config >>>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"} v 0) >>>> v1 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >>>> from='mgr.14252 10.40.10.201:0/357060750 >>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>>> cmd=[{"prefix":"config >>>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"}]: >>>> dispatch >>>> >>>> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 >>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>>> _set_new_cache_sizes cache_size:1020054731 >>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 >>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >>>> prepare_rotating_update before: data.rotating_ver=17 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 >>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>>> _set_new_cache_sizes cache_size:1020054731 >>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 >>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >>>> prepare_rotating_update before: data.rotating_ver=17 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 >>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>>> _set_new_cache_sizes cache_size:1020054731 >>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 >>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >>>> prepare_rotating_update before: data.rotating_ver=17 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 >>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>>> _set_new_cache_sizes cache_size:1020054731 >>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 >>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >>>> prepare_rotating_update before: data.rotating_ver=17 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.142 <tel:51142>+0800 >>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>>> mon_command({"prefix": "config dump", "format": "json"} v 0) v1 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.143 <tel:51143>+0800 >>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >>>> from='mgr.14252 10.40.10.201:0/357060750 >>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>>> cmd=[{"prefix": "config dump", "format": "json"}]: dispatch >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>>> mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >>>> from='mgr.14252 10.40.10.201:0/357060750 >>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>>> cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>>> mon_command({"prefix": "auth get", "entity": "client.admin"} v 0) >>>> v1 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >>>> from='mgr.14252 10.40.10.201:0/357060750 >>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>>> cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.247 <tel:52247>+0800 >>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>>> mon_command([{prefix=config-key set, key=mgr/cephadm/host.node1}] >>>> v 0) v1 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:52.260 <tel:52260>+0800 >>>> 7f88b96c1640 0 <tel:16400> log_channel(audit) log [INF] : >>>> from='mgr.14252 10.40.10.201:0/357060750 >>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>>> >>>> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 >>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>>> _set_new_cache_sizes cache_size:1020054731 >>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>>> >>>> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 >>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >>>> prepare_rotating_update before: data.rotating_ver=17 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >>>> 7f88beecc640 10 <tel:64010> start_session entity_name=osd.0 >>>> global_id=14205 <tel:14205> is_new_global_id=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >>>> 7f88beecc640 10 <tel:64010> cephx server osd.0: start_session >>>> server_challenge dba982c2bced072d >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >>>> 7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.3 >>>> global_id=14379 <tel:14379> is_new_global_id=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >>>> 7f88bf6cd640 10 <tel:64010> cephx server osd.3: start_session >>>> server_challenge e9db92f3f5af188c >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >>>> 7f88beecc640 10 <tel:64010> start_session >>>> entity_name=mgr.node1.apderh global_id=14190 >>>> <tel:14190> is_new_global_id=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >>>> 7f88beecc640 10 <tel:64010> cephx server mgr.node1.apderh: >>>> start_session server_challenge b9809292 <tel:9809292>ea2e62f6 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >>>> 7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.5 >>>> global_id=24253 <tel:24253> is_new_global_id=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >>>> 7f88bf6cd640 10 <tel:64010> cephx server osd.5: start_session >>>> server_challenge 1be9f89b4bd9cdbc >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 >>>> 7f88b9ec2640 10 <tel:264010> start_session entity_name=osd.4 >>>> global_id=14406 <tel:14406> is_new_global_id=0 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 >>>> 7f88b9ec2640 10 <tel:264010> cephx server osd.4: start_session >>>> server_challenge cca4694d496c39a8 >>>> >>>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 >>>> 7f88beecc640 10 <tel:64010> cephx server osd.0: handle_request >>>> get_auth_session_key for osd.0 >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list --ceph-users(a)ceph.io >>>> To unsubscribe send an email toceph-users-leave(a)ceph.io >>> >>> -- >>> -- >>> >>> Andrea Martra >>> Senior Manager >>> +39 393 9048451 >>> >>> Open Technologies Sas >>> Via Michele Buniva, 26 - 10064 Pinerolo (TO) >>> Web:http://www.oscct.it >>> >>> ** Open Source Cloud Computing Technologies ** >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > -- > > Andrea Martra > Senior Manager > +39 393 9048451 > > Open Technologies Sas > Via Michele Buniva, 26 - 10064 Pinerolo (TO) > Web: http://www.oscct.it > > ** Open Source Cloud Computing Technologies ** > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: recent ceph-common can not be installed on rhel9.6
by Massimo Sgaravatto 06 May '26

06 May '26

If you update to 9.7, it provides the needed openssl version Regards, Massimo On Wed, May 6, 2026 at 12:48 PM Kenneth Waegeman via ceph-users < ceph-users(a)ceph.io> wrote: > Hi all, > > I’m trying to get some input on this issue: > <https://tracker.ceph.com/issues/75353> > Tasks #75353: recent ceph-common can not be installed on rhel9.6 - Ceph - > Ceph<https://tracker.ceph.com/issues/75353> > tracker.ceph.com<https://tracker.ceph.com/issues/75353> > [X]<https://tracker.ceph.com/issues/75353> > > Has anyone has a workaround or fix for this? > > Thank you! > > Kenneth > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

recent ceph-common can not be installed on rhel9.6
by Kenneth Waegeman 06 May '26

06 May '26

Hi all, I’m trying to get some input on this issue: <https://tracker.ceph.com/issues/75353> Tasks #75353: recent ceph-common can not be installed on rhel9.6 - Ceph - Ceph<https://tracker.ceph.com/issues/75353> tracker.ceph.com<https://tracker.ceph.com/issues/75353> [X]<https://tracker.ceph.com/issues/75353> Has anyone has a workaround or fix for this? Thank you! Kenneth

1 0

Re: could not find secret_id
by Andrea Martra 06 May '26

06 May '26

Hi Eugen, Thanks a lot for chiming in — even a partial recollection helps narrow the search space. A few notes from our side, in case it helps differentiate the cases: - No rbd-mirroring at the customer; single cluster, 39 nodes on Reef 18.2.x. - Pattern affects 4 OSDs on 7 out of 39 nodes, persistent over days (15M+ log lines in 3 days), no clock skew, quorum stable. - We reproduced a very similar log signature on a small test cluster (Reef 18.2.7) by isolating a node hosting MON leader + OSDs for >3h, matching the original xiaowenhao111 scenario. On our test cluster the affected OSDs eventually self-recovered after a couple of key-rotation cycles. - At the customer, the same signature is NOT clearing on its own. Two questions on your prior case, if anything comes back to you: 1. Was zap-and-readd actually required, or was it more of a defensive cleanup after another action (e.g. a MON failover, key rotation, or simply waiting)? In other words: did you ever observe the OSDs unstick on their own, or were they truly stuck? 2. Do you remember whether a MON leader failover alone was tried before going to OSD rebuild, and whether it had any effect? The fact that our reproducer self-recovered while the production case doesn't suggests there's something cluster-specific keeping the rotation desynchronised — possibly stale state on the affected nodes rather than a pure timing/quorum effect, which would line up with your "remainder from previous tests" intuition. If nobody points to an existing tracker, we're prepared to open one with logs from both the reproduction and the customer environment. Thanks again, Andrea Il 24/04/26 22:06, Eugen Block via ceph-users ha scritto: > Hi, > > how did you end up in that situation (did something happen in the > cluster)? Is it affecting only OSDs or other daemons as well? > > We did investigate such an issue one and a half years ago for a > customer. They added hosts to one of the two clusters, at that time > the customer had two sites, with rbd mirroring enabled. And > I don't recall the exact details, we did solve it in an online meeting > and I don't think anyone wrote a protocoll. And the session definitely > wasn't recored. :-) It might have some remainder on the new host from > previous tests, or some misconfiguration, I really can't tell anymore. > I think in the end we ended up removing the OSDs, zapping them and > adding them back. Of course, it could be a different root cause, not > sure which role the mon leader can play in this. But I thought I'd > share this tiny bit at least. :-) > > Regards, > Eugen > > > Zitat von Andrea Martra via ceph-users <ceph-users(a)ceph.io>: > >> Hi all, >> I'm reaching back to this old thread from December 2023 because >> we're investigating a very similar pattern at a customer site >> running Ceph Reef 18.2.x. Logs show identical shape: >> >> AuthRegistry get_handler peer_type 4 method 2 ... >> cephx: verify_authorizer decrypted service osd secret_id=3524 >> auth: could not find secret_id=3524 >> auth: dump_rotating: (local range ~971-973) >> cephx: verify_authorizer could not get service secret ... >> >> Affecting 4 OSDs on 7 out of 39 nodes, persistent over days >> (15M+ log lines in 3 days). No clock skew, quorum stable. >> >> @xiaowenhao111 or anyone else who hit this: did you ever >> find a root cause or a reliable fix? >> The original thread >> didn't seem to reach a conclusion. Specifically: >> >> - Did restarting the affected OSDs clear it, or did the >> pattern come back? >> - Was a MON leader failover enough, or did you need to >> rebuild anything? >> - Any tracker issue filed for this? >> >> We're trying to reproduce on a test cluster (Reef 18.2.7) >> by isolating a node hosting MON leader + OSDs via network >> for 3+ hours, to match the scenario described in the >> original thread. Happy to share findings back to the list. >> >> Thanks, >> Andrea >> >> Il 21/12/23 02:42, xiaowenhao111 ha scritto: >>> >>> >>> Hello ceph-users: >>> >>> First ,sorry my english ... >>> >>> >>> I find a bug (16.2.10 <tel:16210> and 18.2.1), but i do not know why >>> >>> test steps: >>> >>> 1、config all hosts contain chronyd service >>> >>> 2、cephadm --image quay.io/ceph/ceph:v16.2.10 >>> <http://quay.io/ceph/ceph:v16.2.10> bootstrap >>> --dashboard-password-noupdate --mon-ip 10.40.10.200 >>> <http://10.40.10.200> --cluster-network=10.40.10.0/24 >>> <http://10.40.10.0/24> --skip-pull --allow-overwrite >>> --skip-monitoring-stack --allow-mismatched-release >>> >>> 3、add hosts >>> >>> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node2 >>> >>> ceph orch host add node2 10.40.10.202 >>> <tel:2104010202> --labels=_admin >>> >>> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node-1 >>> >>> ceph orch host add node-1 10.40.10.202 >>> <tel:1104010202> --labels=_admin >>> >>> 4、add osds: >>> >>> ceph orch daemon add osd node1:/dev/sda >>> >>> ceph orch daemon add osd node1:/dev/sdb >>> >>> ceph orch daemon add osd node1:/dev/sdc >>> >>> ceph orch daemon add osd node1:/dev/sdd >>> >>> ceph orch daemon add osd node2:/dev/sda >>> >>> ceph orch daemon add osd node2:/dev/sdb >>> >>> ceph orch daemon add osd node2:/dev/sdc >>> >>> ceph orch daemon add osd node2:/dev/sdd >>> >>> 5、ceph osd pool create test_pool 512 512 <tel:512512> --size=3 >>> >>> 6、disconnect node1 (which is monitor leader) cluster and public >>> network >>> >>> ifdown enp35s0f0 >>> >>> ifdown enp35s0f1 >>> >>> 7、stay overnight（more than three hours） >>> >>> 8、connect node1 cluster and public network >>> >>> ifup enp35s0f0 >>> >>> ifup enp35s0f1 >>> >>> >>> then, the osds of node1 call get_auth_session_key from mon.node1 >>> which is not new leader，get the wrong secret_id=4. >>> >>> the others report error "could not find secret_id=4" more than 40 >>> minutes >>> >>> >>> >>> >>> the ceph.log content of node1: >>> >>> >>> 2023-12-20 <tel:20231220>T09:48:30.199224 <tel:30199224>+0800 >>> mon.node1 (mon.0) 1891 <tel:01891> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-20 <tel:20231220>T09:49:37.816260 <tel:37816260>+0800 >>> mon.node1 (mon.0) 1901 <tel:01901> : cluster [INF] Health check >>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >>> >>> 2023-12-20 <tel:20231220>T09:49:44.222046 <tel:44222046>+0800 >>> mon.node1 (mon.0) 1902 <tel:01902> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-20 <tel:20231220>T09:50:00.000090 <tel:00000090>+0800 >>> mon.node1 (mon.0) 1905 <tel:01905> : cluster [WRN] overall >>> HEALTH_WARN Failed to place 1 daemon(s); 3 failed cephadm daemon(s); >>> 1 pool(s) do not have an application enabled >>> >>> 2023-12-20 <tel:20231220>T09:50:51.916464 <tel:51916464>+0800 >>> mon.node1 (mon.0) 1912 <tel:01912> : cluster [INF] Health check >>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >>> >>> 2023-12-20 <tel:20231220>T09:51:07.127982 <tel:07127982>+0800 >>> mon.node1 (mon.0) 1916 <tel:01916> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0 >>> (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >>> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 calling >>> monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >>> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >>> calling monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >>> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >>> election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >>> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 calling >>> monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >>> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new >>> leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >>> >>> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >>> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >>> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >>> >>> >>> the ceph.log content of node2: >>> >>> >>> 2023-12-21 <tel:20231221>T01:10:01.065051 <tel:01065051>+0800 >>> mon.node2 (mon.1) 11522 <tel:111522> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-21 <tel:20231221>T01:11:11.092640 <tel:11092640>+0800 >>> mon.node2 (mon.1) 11531 <tel:111531> : cluster [INF] Health check >>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >>> >>> 2023-12-21 <tel:20231221>T01:11:19.197355 <tel:19197355>+0800 >>> mon.node2 (mon.1) 11532 <tel:111532> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-21 <tel:20231221>T01:12:34.469403 <tel:34469403>+0800 >>> mon.node2 (mon.1) 11547 <tel:111547> : cluster [INF] Health check >>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >>> >>> 2023-12-21 <tel:20231221>T01:12:42.476793 <tel:42476793>+0800 >>> mon.node2 (mon.1) 11549 <tel:111549> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-21 <tel:20231221>T01:13:55.234163 <tel:55234163>+0800 >>> mon.node2 (mon.1) 11572 <tel:111572> : cluster [INF] osd.5 marked >>> itself dead as of e91 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.236348 <tel:55236348>+0800 >>> mon.node2 (mon.1) 11573 <tel:111573> : cluster [INF] osd.4 marked >>> itself dead as of e91 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.242209 <tel:55242209>+0800 >>> mon.node2 (mon.1) 11574 <tel:111574> : cluster [INF] osd.0 marked >>> itself dead as of e91 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.233774 <tel:55233774>+0800 osd.5 >>> (osd.5) 3 : cluster [WRN] Monitor daemon marked osd.5 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:13:55.236038 <tel:55236038>+0800 osd.4 >>> (osd.4) 3 : cluster [WRN] Monitor daemon marked osd.4 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:14:00.109136 <tel:109136>+0800 >>> mon.node2 (mon.1) 11589 <tel:111589> : cluster [WRN] Health check >>> update: Degraded data redundancy: 1/8 objects degraded (12.500 >>> <tel:12500>%), 1 pg degraded, 188 pgs undersized (PG_DEGRADED) >>> >>> 2023-12-21 <tel:20231221>T01:14:00.311644 <tel:311644>+0800 >>> mon.node2 (mon.1) 11590 <tel:111590> : cluster [INF] osd.3 marked >>> itself dead as of e94 >>> >>> 2023-12-21 <tel:20231221>T01:14:00.311410 <tel:311410>+0800 osd.3 >>> (osd.3) 3 : cluster [WRN] Monitor daemon marked osd.3 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:14:01.922553 <tel:01922553>+0800 >>> mgr.node1.apderh (mgr.34343) 1 <tel:343431> : cluster [ERR] Failed >>> to load ceph-mgr modules: k8sevents >>> >>> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0 >>> (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >>> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 calling >>> monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >>> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >>> calling monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >>> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >>> election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >>> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 calling >>> monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >>> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new >>> leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >>> >>> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >>> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >>> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >>> >>> >>> >>> the osd log of node1: >>> >>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6804 >>> <http://10.40.10.202:6804> osd.6 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6812 >>> <http://10.40.10.202:6812> osd.7 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6820 >>> <http://10.40.10.202:6820> osd.8 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6828 >>> <http://10.40.10.202:6828> osd.9 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: reset >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: 0x562066 <tel:09562066>bbc000 >>> handle_response ret = 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: got initial server challenge >>> 5fd5d9eca256ae69 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: validate_tickets: want=53 need=0 have=53 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: validate_tickets want 53 have 0 need 53 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: want=53 need=53 have=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: build_request >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 20 <tel:864020> cephx client: old ticket len=96 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: get auth session key: client_challenge >>> 7ca3f7c3ef30b2bf >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: 0x562066 <tel:09562066>bbc000 >>> handle_response ret = 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: get_auth_session_key >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply got 1 keys >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: got key for service_id auth >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply got encrypted ticket >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=2 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply service auth >>> secret_id 2 session_key AQDIIINlUlSmKBAAvL5rKyyIb6vt89rxUGlsxg== >>> validity=259200.000000 <tel:259200000000> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket >>> expires=2023-12-24 <tel:20231224>T01:13:44.682217 >>> <tel:44682217>+0800 renew_after=2023-12-23 >>> <tel:20231223>T07:13:44.682217 <tel:44682217>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: want=53 need=53 have=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: got connection bl 84 and extra >>> tickets 550 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: got connection_secret 64 bytes >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply got 3 keys >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: got key for service_id mon >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply service mon >>> secret_id 4 session_key AQDIIINlWvamKBAAxOaZsuCMdgutax1n2ZRbZQ== >>> validity=3600.000000 <tel:3600000000> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket >>> expires=2023-12-21 <tel:20231221>T02:13:44.682256 >>> <tel:44682256>+0800 renew_after=2023-12-21 >>> <tel:20231221>T01:58:44.682256 <tel:44682256>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: got key for service_id osd >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply service osd >>> secret_id 4 session_key AQDIIINlXAinKBAAOwTlViH3CJRgSvj5Kn9qtA== >>> validity=3600.000000 <tel:3600000000> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket >>> expires=2023-12-21 <tel:20231221>T02:13:44.682270 >>> <tel:44682270>+0800 renew_after=2023-12-21 >>> <tel:20231221>T01:58:44.682270 <tel:44682270>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: got key for service_id mgr >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply service mgr >>> secret_id 4 session_key AQDIIINlUiGnKBAAr6WBaSawzVyDoqEs8cENdg== >>> validity=3600.000000 <tel:3600000000> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket >>> expires=2023-12-21 <tel:20231221>T02:13:44.682286 >>> <tel:44682286>+0800 renew_after=2023-12-21 >>> <tel:20231221>T01:58:44.682286 <tel:44682286>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: got extra service_tickets >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: validate_tickets want 53 have 53 need 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: validate_tickets want 53 have 53 need 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 20 <tel:864020> cephx client: need_tickets: want=53 have=53 need=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> -1 <tel:86401> monclient: _check_auth_rotating possible clock skew, >>> rotating keys expired way too early (before 2023-12-21 >>> <tel:20231221>T00:13:44.682376 <tel:44682376>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: build_rotating_request >>> >>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.201:6804 >>> <http://10.40.10.201:6804> osd.1 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.201:6812 >>> <http://10.40.10.201:6812> osd.2 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.200:6814 >>> <http://10.40.10.200:6814> osd.3 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> >>> >>> >>> the osd log of node2: >>> >>> >>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640 >>> 10 <tel:64010> cephx: validate_tickets want 53 have 53 need 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640 >>> 20 <tel:64020> cephx client: need_tickets: want=53 have=53 need=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640 >>> 10 <tel:64010> auth: dump_rotating: >>> >>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640 >>> 10 <tel:64010> auth: id 17 AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== >>> expires 2023-12-21 <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 7fa573a6d640 >>> 10 <tel:64010> auth: id 18 AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== >>> expires 2023-12-21 <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 7fa573a6d640 >>> 10 <tel:64010> auth: id 19 AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== >>> expires 2023-12-21 <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 20 <tel:58129364020> AuthRegistry(0x7ffc86c1e7b0) get_handler >>> peer_type 4 method 2 cluster_methods [2] service_methods [2] >>> client_methods [2] >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> cephx: verify_authorizer decrypted service osd >>> secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>> 7fa581293640 0 <tel:5812936400> auth: could not find secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> auth: dump_rotating: >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> auth: id 17 >>> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21 >>> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> auth: id 18 >>> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21 >>> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> auth: id 19 >>> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21 >>> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>> 7fa581293640 0 <tel:5812936400> cephx: verify_authorizer could not >>> get service secret for service osd secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 7fa573a6d640 >>> 10 <tel:64010> cephx: validate_tickets want 53 have 53 need 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 7fa573a6d640 >>> 20 <tel:64020> cephx client: need_tickets: want=53 have=53 need=0 >>> >>> >>> >>> the mon log of node1: >>> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> start_session entity_name=mgr.node1.apderh >>> global_id=14184 <tel:14184> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: start_session >>> server_challenge 524ce5eea3e4dd75 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: handle_request >>> get_auth_session_key for mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 20 <tel:964020> cephx server mgr.node1.apderh: checking key: >>> req.key=f41952 <tel:41952>c0f02e6c0f expected_key=f41952 >>> <tel:41952>c0f02e6c0f >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 20 <tel:964020> cephx server mgr.node1.apderh: checking old_ticket: >>> secret_id=2 len=112, old_ticket_may_be_omitted=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 20 <tel:964020> cephx server mgr.node1.apderh: decoded old_ticket: >>> global_id=14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: allowing reclaim of >>> global_id 14184 <tel:14184> (valid ticket presented, will encrypt >>> new ticket) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket_reply encoding 1 tickets >>> with secret AQBmKIJlvHeYJRAAf5h+auT4c9Ilo/nlUqRJlg== >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service auth secret_id 2 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: adding key for >>> service mon >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: adding key for >>> service mds >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: adding key for >>> service osd >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: adding key for >>> service mgr >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket_reply encoding 4 tickets >>> with secret AQDIIINldY1mGRAApH04rNRZtuyAUrQn5PNG7A== >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service mon secret_id 4 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service mds secret_id 4 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service osd secret_id 4 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service mgr secret_id 4 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >>> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command >>> mon_command({"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"} >>> v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >>> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >>> from='mgr.14184 10.40.10.200:0/2514043697 >>> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >>> cmd=[{"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"}]: >>> dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >>> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command >>> mon_command({"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"} >>> v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >>> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >>> from='mgr.14184 10.40.10.200:0/2514043697 >>> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >>> cmd=[{"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"}]: >>> dispatch >>> >>> >>> >>> the mon log of node2: >>> >>> >>> 2023-12-21 <tel:20231221>T01:13:10.292 <tel:10292>+0800 >>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >>> {"time_micros": 1703092390293629, "job": 1145, "event": >>> "table_file_deletion", "file_number": 2272} >>> >>> 2023-12-21 <tel:20231221>T01:13:10.295 <tel:10295>+0800 >>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >>> {"time_micros": 1703092390296385, "job": 1145, "event": >>> "table_file_deletion", "file_number": 2270} >>> >>> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >>> {"time_micros": 1703092390304703, "job": 1145, "event": >>> "table_file_deletion", "file_number": 2269} >>> >>> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"} >>> v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"}]: >>> dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"} >>> v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"}]: >>> dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.142 <tel:51142>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix": "config dump", "format": "json"} v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.143 <tel:51143>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix": "config dump", "format": "json"}]: dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix": "auth get", "entity": "client.admin"} v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:52.247 <tel:52247>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command([{prefix=config-key set, key=mgr/cephadm/host.node1}] v >>> 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.260 <tel:52260>+0800 >>> 7f88b96c1640 0 <tel:16400> log_channel(audit) log [INF] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> >>> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640 >>> 10 <tel:64010> start_session entity_name=osd.0 global_id=14205 >>> <tel:14205> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640 >>> 10 <tel:64010> cephx server osd.0: start_session server_challenge >>> dba982c2bced072d >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640 >>> 10 <tel:64010> start_session entity_name=osd.3 global_id=14379 >>> <tel:14379> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640 >>> 10 <tel:64010> cephx server osd.3: start_session server_challenge >>> e9db92f3f5af188c >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640 >>> 10 <tel:64010> start_session entity_name=mgr.node1.apderh >>> global_id=14190 <tel:14190> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640 >>> 10 <tel:64010> cephx server mgr.node1.apderh: start_session >>> server_challenge b9809292 <tel:9809292>ea2e62f6 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640 >>> 10 <tel:64010> start_session entity_name=osd.5 global_id=24253 >>> <tel:24253> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640 >>> 10 <tel:64010> cephx server osd.5: start_session server_challenge >>> 1be9f89b4bd9cdbc >>> >>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 7f88b9ec2640 >>> 10 <tel:264010> start_session entity_name=osd.4 global_id=14406 >>> <tel:14406> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 7f88b9ec2640 >>> 10 <tel:264010> cephx server osd.4: start_session server_challenge >>> cca4694d496c39a8 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 7f88beecc640 >>> 10 <tel:64010> cephx server osd.0: handle_request >>> get_auth_session_key for osd.0 >>> >>> >>> _______________________________________________ >>> ceph-users mailing list --ceph-users(a)ceph.io >>> To unsubscribe send an email toceph-users-leave(a)ceph.io >> >> -- >> -- >> >> Andrea Martra >> Senior Manager >> +39 393 9048451 >> >> Open Technologies Sas >> Via Michele Buniva, 26 - 10064 Pinerolo (TO) >> Web:http://www.oscct.it >> >> ** Open Source Cloud Computing Technologies ** >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- -- Andrea Martra Senior Manager +39 393 9048451 Open Technologies Sas Via Michele Buniva, 26 - 10064 Pinerolo (TO) Web: http://www.oscct.it ** Open Source Cloud Computing Technologies **

1 0

Re: Host buckets in crush map
by Iztok Gregori 06 May '26

06 May '26

Hi! Sometimes I find a solution just minutes after I click send... On 06/05/26 11:05, Iztok Gregori via ceph-users wrote: > Hi to all! > > I have 2 questions regarding host buckets in the crush map; > > 1. The name of a "host bucket" is just a label (which can have an > arbitrary value) or is used in some way or form (so it has to be a real > short or full hostname)? > > 2. The second question is more complex, how to avoid, upon host reboot > (or during the OSD activation), the creation of a new host bucket? I'll > try to explain myself better. > [cut] > > So, the question remains, how to "pass" to ceph the correct host bucket > name? > > I think that the solution is to put "host=node01-int" in ceph.conf, but > where? Under the "[osd]" sections ("[osd.*]"???) or I have to insert it > under every single osd ([osd.12], [osd.13]...)? Crush location can be defined in ceph.conf as: crush_location = root=default row=a rack=a2 chassis=a2a host=a2a1 Or the management can be disabled with (also in ceph.conf): osd_crush_update_on_start = false So, with this the question number 2 is answered, but for the first question I'm still not sure about it. Thanks Iztok

1 0

Host buckets in crush map
by Iztok Gregori 06 May '26

06 May '26

Hi to all! I have 2 questions regarding host buckets in the crush map; 1. The name of a "host bucket" is just a label (which can have an arbitrary value) or is used in some way or form (so it has to be a real short or full hostname)? 2. The second question is more complex, how to avoid, upon host reboot (or during the OSD activation), the creation of a new host bucket? I'll try to explain myself better. The host buckets in our environment have the name of the interface not the name of the host itself (if a node's "hostname -s" is "node01" this node is present in the crush map as "node01-int", to represent the interface where Ceph is communicating, cluster/private network). Furthermore each host bucket can be in a different crush root (the "default" crush root is empty). I just reinstalled some nodes with Rocky 9 and Ceph Quincy (see my previous messages for more info) and upon activation of the OSDs a new host bucket is created and placed (with all the OSDs) in the default root (The name of this host bucket is node "hostname -s", like "node01"). Because it is in the wrong root, I've renamed it to the interface name ("node01-int"), moved it to the correct root and deleted the "wrong" host bucket ("node01"). Everything seemed to be working fine, but to be sure I've rebooted (again) a freshly reinstalled node. Upon reboot the "wrong" host bucket (node01) is recreated in the default root and all the OSDs are moved there leaving the host bucket "node01-int" empty. Clearly upon boot the ceph-osd daemons pass the name "hostname -s" to the monitors, which (I suppose) check if this name exists in the crush map, if it doesn't a new host bucket is created (or something along this lines). In fact, when I changed the hostname of my node to "node01-int", after reboot, the OSDs were placed under the correct host bucket (and, more importantly, under the correct root). If the answer on my question number 1. is that the host bucket name is just a label, as long as its in the correct root, doesn't matter how the node is called (and I could just put all the host buckets name without the "*-int" suffix). But if is not a label I need to have it with the interface name. So, the question remains, how to "pass" to ceph the correct host bucket name? I think that the solution is to put "host=node01-int" in ceph.conf, but where? Under the "[osd]" sections ("[osd.*]"???) or I have to insert it under every single osd ([osd.12], [osd.13]...)? If you read this line, congratulation you reached the finish! :-) Thank you! Iztok P.S. I'm not using cephadm, all the commands are manual. -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. http://www.elettra.eu

1 0

Re: MDS and failure domains
by Gregory Farnum 06 May '26

06 May '26

Unfortunately, this was recently IDed as a major weakness of the current MDS rank configuration system. :/ You can follow https://tracker.ceph.com/issues/75892 which was filed to work on this. -Greg On Tue, May 5, 2026 at 11:34 PM Robert Sander via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > Given a cluster with multiple racks or rooms where the rack or room is > the failure domain for replicated pools. > > Given one or more CephFS volumes (filesystems) with at least one active > MDS each and a standby-replay for each active MDS. > > Each room or rack contains multiple MDS daemons (one per host). > > Is there a possibility to guarantee that the standby-replay MDS for each > active MDS does not run in the same failure domain (rack or room)? > > Because when both run in the same failure domain and that domain fails > there is no point in having a standby-replay. > > Are there any ideas on how to solve this? > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > <https://www.google.com/maps/search/Schwedter+Str.+8%2F9b,+10119+Berlin?entr…> > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

MDS and failure domains
by Robert Sander 06 May '26

06 May '26

Hi, Given a cluster with multiple racks or rooms where the rack or room is the failure domain for replicated pools. Given one or more CephFS volumes (filesystems) with at least one active MDS each and a standby-replay for each active MDS. Each room or rack contains multiple MDS daemons (one per host). Is there a possibility to guarantee that the standby-replay MDS for each active MDS does not run in the same failure domain (rack or room)? Because when both run in the same failure domain and that domain fails there is no point in having a standby-replay. Are there any ideas on how to solve this? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Ceph Dashboard fails on tenant RGW buckets (NoSuchBucket → 500)
by farhad kh 05 May '26

05 May '26

Ceph Dashboard fails with 500 Internal Server Error when operating on RGW buckets that belong to a *tenant*, even though the bucket exists and is accessible via radosgw-admin. RGW returns NoSuchBucket because the Dashboard queries the bucket *without the tenant prefix* ----------- 13:02:06.832428866Z"} {"log":"debug 2026-05-04T13:02:06.838+0000 7fbc29896640 0 [dashboard ERROR rest_client] RGW REST API failed GET req status: 404\n","stream":"stderr","time":"2026-05-04T13:02:06.841073842Z"} {"log":"debug 2026-05-04T13:02:06.838+0000 7fbc29896640 0 [dashboard ERROR exception] Internal Server Error\n","stream":"stderr","time":"2026-05-04T13:02:06.842785189Z"} {"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2026-05-04T13:02:06.842818184Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/services/exception.py\", line 47, in dashboard_exception_handler\n","stream":"stderr","time":"2026-05-04T13:02:06.842831624Z"} {"log":" return handler(*args, **kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.842841016Z"} {"log":" File \"/lib/python3.9/site-packages/cherrypy/_cpdispatch.py\", line 54, in __call__\n","stream":"stderr","time":"2026-05-04T13:02:06.842851035Z"} {"log":" return self.callable(*self.args, **self.kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.842983312Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py\", line 264, in inner\n","stream":"stderr","time":"2026-05-04T13:02:06.843004859Z"} {"log":" ret = func(*args, **kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.843015376Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py\", line 193, in wrapper\n","stream":"stderr","time":"2026-05-04T13:02:06.843024549Z"} {"log":" return func(*vpath, **params)\n","stream":"stderr","time":"2026-05-04T13:02:06.84303376Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\", line 702, in set\n","stream":"stderr","time":"2026-05-04T13:02:06.843050003Z"} {"log":" locking = self._get_locking(uid, daemon_name, bucket_name)\n","stream":"stderr","time":"2026-05-04T13:02:06.843059741Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\", line 443, in _get_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843069083Z"} {"log":" return rgw_client.get_bucket_locking(bucket_name)\n","stream":"stderr","time":"2026-05-04T13:02:06.843078375Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 538, in func_wrapper\n","stream":"stderr","time":"2026-05-04T13:02:06.843087524Z"} {"log":" return func(\n","stream":"stderr","time":"2026-05-04T13:02:06.843096712Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\", line 899, in get_bucket_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843105575Z"} {"log":" raise e\n","stream":"stderr","time":"2026-05-04T13:02:06.843114638Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\", line 881, in get_bucket_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843123252Z"} {"log":" result = request() # type: ignore\n","stream":"stderr","time":"2026-05-04T13:02:06.843132275Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 324, in __call__\n","stream":"stderr","time":"2026-05-04T13:02:06.843141087Z"} {"log":" resp = self.rest_client.do_request(method, self._gen_path(), params,\n","stream":"stderr","time":"2026-05-04T13:02:06.843163504Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 422, in do_request\n","stream":"stderr","time":"2026-05-04T13:02:06.843173809Z"} {"log":" raise RequestException(\n","stream":"stderr","time":"2026-05-04T13:02:06.843367851Z"} {"log":"dashboard.rest_client.RequestException: RGW REST API failed request with status code 404\n","stream":"stderr","time":"2026-05-04T13:02:06.843382042Z"} {"log":"(b'{\"Code\":\"NoSuchBucket\",\"Message\":\"\",\"BucketName\":\"farhad\",\"RequestId\":\"tx000'\n","stream":"stderr","time":"2026-05-04T13:02:06.843391177Z"} {"log":" b'00ab80c789f95808ff-0069f898ce-110717032-vanak\",\"HostId\":\"110717032-vanak-teh'\n","stream":"stderr","time":"2026-05-04T13:02:06.843400934Z"} {"log":" b'ran\"}')\n","stream":"stderr","time":"2026-05-04T13:02:06.843410095Z"} {"log":"debug 2026-05-04T13:02:06.842+0000 7fbc29896640 0 [dashboard ERROR request] [::ffff:172.20.26.54:49876] [PUT] [500] [0.619s] [admin] [513.0B] /api/rgw/bucket/khedrian%2Ffarhad\n","stream":"stderr","time":"2026-05-04T13:02:06.846732997Z"} {"log":"debug 2026-05-04T13:02:06.842+0000 7fbc29896640 0 [dashboard ERROR request] [b'{\"status\": \"500 Internal Server Error\", \"detail\": \"The server encountered an unexpected condition which prevented it from fulfilling the request.\", \"request_id\": \"a3abea21-d73c-4d26-a8bb-1d99fc53a466\"} ']\n","stream":"stderr","time":"2026-05-04T13:02:06.847608634Z"} {"log":"debug 2026-05-04T13:02:06.846+0000 7fbc29896640 0 [dashboard INFO request] [::ffff:172.20.26.54:49876] [PUT] [500] [0.622s] [admin] [513.0B] /api/rgw/bucket/khedrian%2Ffarhad\n","stream":"stderr","time":"2026-05-04T13:02:06.849427429Z"} {"log":"debug 2026-05-04T13:02:07.322+0000 7fbc28053640 0 [dashboard INFO request] [::ffff:172.20.26.54:64634] [GET] [200] [0.691s] [admin] [295.0B] /api/rgw/daemon\n","stream":"stderr","time":"2026-05-04T13:02:07.324402049Z"} {"log":"debug 2026-05-04T13:02:07.850+0000 7fbc2504d640 0 [dashboard INFO request] [::ffff:172.20.26.54:49870] [GET] [200] [0.004s] [admin] [22.0B] /api/prometheus/notifications\n","stream":"stderr","time":"2026-05-04T13:02:07.855794264Z"} {"log":"debug 2026-05-04T13:02:08.110+0000 7fbc5c2fb640 0 log_channel(cluster) log [DBG] : pgmap v611376: 449 pgs: 449 active+clean; 202 MiB data, 10 GiB used, 1.2 TiB / 1.2 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 8 op/s\n","stream":"stderr","time":"2026-05-04T13:02:08.113933657Z"} {"log":"debug 2026-05-04T13:02:10.111+0000 7fbc5c2fb640 0 log_channel(cluster) log [DBG] : pgmap v611377: 449 pgs: 449 active+clean; 202 MiB data, 10 GiB used, 1.2 TiB / 1.2 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s\n","stream":"stderr","time":"2026-05-04T13:02:10.115433742Z"} {"log":"debug 2026-05-04T13:02:10.211+0000 7fbc2484c640 0 [dashboard INFO request] [::ffff:172.20.26.54:49876] [GET] [200] [0.003s] [admin] [22.0B] /api/prometheus/notifications\n","stream":"stderr","time":"2026-05-04T13:02:10.2151369Z"} ---------------- root@dev-ceph01:~# ceph -v ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable) { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::khedrian:farhad/*" } ] } - Bucket exists and is accessible via radosgw-admin - Issue only occurs for *tenant buckets* - Non‑tenant buckets work correctly - Appears to be a *Dashboard RGW tenant parsing bug* - Possibly missing tenant propagation when calling get_bucket_locking() and similar RGW APIs can anyone explain this ? what should i do ?

1 0

[ Radosgw Admin API] - How to include user tenant in API requests
by wissem mimouna 05 May '26

05 May '26

Hi All, I'm trying to find out how to use the radsogw admin api to get user usage information ,hence the user has a tenant and I trying to pass : > /admin/bucket?uid=tenant$user >/admin/bucket?uid=user >/admin/bucket?uid=user&tenant=tenant >/admin/bucket?uid=tenant/user But I always get an empty response (https://docs.ceph.com/en/latest/radosgw/adminops/#id60) May be there is another way to call the api with user and tenant ? Thanks

1 0

Ceph Tech Talk and CFP Reminder!!
by Anthony Middleton 04 May '26

04 May '26

Hello, Ceph friends! I hope your week is off to a great start. I wanted to let you know that our next Ceph Tech Talk <https://ceph.io/en/community/tech-talks/> is on *Monday*, May 11* at 9 AM PDT* w/ *Shraddha Agrawal*. *Title:* Containerized deployment of Ceph with KCLI *Abstract*: Cephadm is the recommended approach for deploying Ceph clusters, but many developers still depend on bare metal environments for testing and development. In this session, Shraddha will introduce KCLI as a way to bridge that gap—making it easier to run Cephadm-powered clusters locally with minimal setup. This talk is especially useful for developers looking to streamline their environments and reduce setup overhead. The invite is on the Ceph Community Calendar: https://t.ly/CephTechTalks *ALSO!!!* The CFP for Ceph Days Seattle <https://ceph.io/en/community/events/2026/ceph-days-seattle/> closes *tomorrow*, May 5* (Revenge of the Fifth)!* This Ceph community event will be at the SURF Incubator on May 28. Submit now: https://t.ly/seattleCFP2026. Take care, and please reach out if you have any questions. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0