- ceph-users - lists.ceph.io

Re: recent ceph-common can not be installed on rhel9.6
by Massimo Sgaravatto 06 May '26

06 May '26

If you update to 9.7, it provides the needed openssl version Regards, Massimo On Wed, May 6, 2026 at 12:48 PM Kenneth Waegeman via ceph-users < ceph-users(a)ceph.io> wrote: > Hi all, > > I’m trying to get some input on this issue: > <https://tracker.ceph.com/issues/75353> > Tasks #75353: recent ceph-common can not be installed on rhel9.6 - Ceph - > Ceph<https://tracker.ceph.com/issues/75353> > tracker.ceph.com<https://tracker.ceph.com/issues/75353> > [X]<https://tracker.ceph.com/issues/75353> > > Has anyone has a workaround or fix for this? > > Thank you! > > Kenneth > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

recent ceph-common can not be installed on rhel9.6
by Kenneth Waegeman 06 May '26

06 May '26

Hi all, I’m trying to get some input on this issue: <https://tracker.ceph.com/issues/75353> Tasks #75353: recent ceph-common can not be installed on rhel9.6 - Ceph - Ceph<https://tracker.ceph.com/issues/75353> tracker.ceph.com<https://tracker.ceph.com/issues/75353> [X]<https://tracker.ceph.com/issues/75353> Has anyone has a workaround or fix for this? Thank you! Kenneth

1 0

Re: could not find secret_id
by Andrea Martra 06 May '26

06 May '26

Hi Eugen, Thanks a lot for chiming in — even a partial recollection helps narrow the search space. A few notes from our side, in case it helps differentiate the cases: - No rbd-mirroring at the customer; single cluster, 39 nodes on Reef 18.2.x. - Pattern affects 4 OSDs on 7 out of 39 nodes, persistent over days (15M+ log lines in 3 days), no clock skew, quorum stable. - We reproduced a very similar log signature on a small test cluster (Reef 18.2.7) by isolating a node hosting MON leader + OSDs for >3h, matching the original xiaowenhao111 scenario. On our test cluster the affected OSDs eventually self-recovered after a couple of key-rotation cycles. - At the customer, the same signature is NOT clearing on its own. Two questions on your prior case, if anything comes back to you: 1. Was zap-and-readd actually required, or was it more of a defensive cleanup after another action (e.g. a MON failover, key rotation, or simply waiting)? In other words: did you ever observe the OSDs unstick on their own, or were they truly stuck? 2. Do you remember whether a MON leader failover alone was tried before going to OSD rebuild, and whether it had any effect? The fact that our reproducer self-recovered while the production case doesn't suggests there's something cluster-specific keeping the rotation desynchronised — possibly stale state on the affected nodes rather than a pure timing/quorum effect, which would line up with your "remainder from previous tests" intuition. If nobody points to an existing tracker, we're prepared to open one with logs from both the reproduction and the customer environment. Thanks again, Andrea Il 24/04/26 22:06, Eugen Block via ceph-users ha scritto: > Hi, > > how did you end up in that situation (did something happen in the > cluster)? Is it affecting only OSDs or other daemons as well? > > We did investigate such an issue one and a half years ago for a > customer. They added hosts to one of the two clusters, at that time > the customer had two sites, with rbd mirroring enabled. And > I don't recall the exact details, we did solve it in an online meeting > and I don't think anyone wrote a protocoll. And the session definitely > wasn't recored. :-) It might have some remainder on the new host from > previous tests, or some misconfiguration, I really can't tell anymore. > I think in the end we ended up removing the OSDs, zapping them and > adding them back. Of course, it could be a different root cause, not > sure which role the mon leader can play in this. But I thought I'd > share this tiny bit at least. :-) > > Regards, > Eugen > > > Zitat von Andrea Martra via ceph-users <ceph-users(a)ceph.io>: > >> Hi all, >> I'm reaching back to this old thread from December 2023 because >> we're investigating a very similar pattern at a customer site >> running Ceph Reef 18.2.x. Logs show identical shape: >> >> AuthRegistry get_handler peer_type 4 method 2 ... >> cephx: verify_authorizer decrypted service osd secret_id=3524 >> auth: could not find secret_id=3524 >> auth: dump_rotating: (local range ~971-973) >> cephx: verify_authorizer could not get service secret ... >> >> Affecting 4 OSDs on 7 out of 39 nodes, persistent over days >> (15M+ log lines in 3 days). No clock skew, quorum stable. >> >> @xiaowenhao111 or anyone else who hit this: did you ever >> find a root cause or a reliable fix? >> The original thread >> didn't seem to reach a conclusion. Specifically: >> >> - Did restarting the affected OSDs clear it, or did the >> pattern come back? >> - Was a MON leader failover enough, or did you need to >> rebuild anything? >> - Any tracker issue filed for this? >> >> We're trying to reproduce on a test cluster (Reef 18.2.7) >> by isolating a node hosting MON leader + OSDs via network >> for 3+ hours, to match the scenario described in the >> original thread. Happy to share findings back to the list. >> >> Thanks, >> Andrea >> >> Il 21/12/23 02:42, xiaowenhao111 ha scritto: >>> >>> >>> Hello ceph-users: >>> >>> First ,sorry my english ... >>> >>> >>> I find a bug (16.2.10 <tel:16210> and 18.2.1), but i do not know why >>> >>> test steps: >>> >>> 1、config all hosts contain chronyd service >>> >>> 2、cephadm --image quay.io/ceph/ceph:v16.2.10 >>> <http://quay.io/ceph/ceph:v16.2.10> bootstrap >>> --dashboard-password-noupdate --mon-ip 10.40.10.200 >>> <http://10.40.10.200> --cluster-network=10.40.10.0/24 >>> <http://10.40.10.0/24> --skip-pull --allow-overwrite >>> --skip-monitoring-stack --allow-mismatched-release >>> >>> 3、add hosts >>> >>> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node2 >>> >>> ceph orch host add node2 10.40.10.202 >>> <tel:2104010202> --labels=_admin >>> >>> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node-1 >>> >>> ceph orch host add node-1 10.40.10.202 >>> <tel:1104010202> --labels=_admin >>> >>> 4、add osds: >>> >>> ceph orch daemon add osd node1:/dev/sda >>> >>> ceph orch daemon add osd node1:/dev/sdb >>> >>> ceph orch daemon add osd node1:/dev/sdc >>> >>> ceph orch daemon add osd node1:/dev/sdd >>> >>> ceph orch daemon add osd node2:/dev/sda >>> >>> ceph orch daemon add osd node2:/dev/sdb >>> >>> ceph orch daemon add osd node2:/dev/sdc >>> >>> ceph orch daemon add osd node2:/dev/sdd >>> >>> 5、ceph osd pool create test_pool 512 512 <tel:512512> --size=3 >>> >>> 6、disconnect node1 (which is monitor leader) cluster and public >>> network >>> >>> ifdown enp35s0f0 >>> >>> ifdown enp35s0f1 >>> >>> 7、stay overnight（more than three hours） >>> >>> 8、connect node1 cluster and public network >>> >>> ifup enp35s0f0 >>> >>> ifup enp35s0f1 >>> >>> >>> then, the osds of node1 call get_auth_session_key from mon.node1 >>> which is not new leader，get the wrong secret_id=4. >>> >>> the others report error "could not find secret_id=4" more than 40 >>> minutes >>> >>> >>> >>> >>> the ceph.log content of node1: >>> >>> >>> 2023-12-20 <tel:20231220>T09:48:30.199224 <tel:30199224>+0800 >>> mon.node1 (mon.0) 1891 <tel:01891> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-20 <tel:20231220>T09:49:37.816260 <tel:37816260>+0800 >>> mon.node1 (mon.0) 1901 <tel:01901> : cluster [INF] Health check >>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >>> >>> 2023-12-20 <tel:20231220>T09:49:44.222046 <tel:44222046>+0800 >>> mon.node1 (mon.0) 1902 <tel:01902> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-20 <tel:20231220>T09:50:00.000090 <tel:00000090>+0800 >>> mon.node1 (mon.0) 1905 <tel:01905> : cluster [WRN] overall >>> HEALTH_WARN Failed to place 1 daemon(s); 3 failed cephadm daemon(s); >>> 1 pool(s) do not have an application enabled >>> >>> 2023-12-20 <tel:20231220>T09:50:51.916464 <tel:51916464>+0800 >>> mon.node1 (mon.0) 1912 <tel:01912> : cluster [INF] Health check >>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >>> >>> 2023-12-20 <tel:20231220>T09:51:07.127982 <tel:07127982>+0800 >>> mon.node1 (mon.0) 1916 <tel:01916> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0 >>> (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >>> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 calling >>> monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >>> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >>> calling monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >>> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >>> election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >>> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 calling >>> monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >>> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new >>> leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >>> >>> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >>> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >>> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >>> >>> >>> the ceph.log content of node2: >>> >>> >>> 2023-12-21 <tel:20231221>T01:10:01.065051 <tel:01065051>+0800 >>> mon.node2 (mon.1) 11522 <tel:111522> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-21 <tel:20231221>T01:11:11.092640 <tel:11092640>+0800 >>> mon.node2 (mon.1) 11531 <tel:111531> : cluster [INF] Health check >>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >>> >>> 2023-12-21 <tel:20231221>T01:11:19.197355 <tel:19197355>+0800 >>> mon.node2 (mon.1) 11532 <tel:111532> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-21 <tel:20231221>T01:12:34.469403 <tel:34469403>+0800 >>> mon.node2 (mon.1) 11547 <tel:111547> : cluster [INF] Health check >>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >>> >>> 2023-12-21 <tel:20231221>T01:12:42.476793 <tel:42476793>+0800 >>> mon.node2 (mon.1) 11549 <tel:111549> : cluster [WRN] Health check >>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >>> >>> 2023-12-21 <tel:20231221>T01:13:55.234163 <tel:55234163>+0800 >>> mon.node2 (mon.1) 11572 <tel:111572> : cluster [INF] osd.5 marked >>> itself dead as of e91 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.236348 <tel:55236348>+0800 >>> mon.node2 (mon.1) 11573 <tel:111573> : cluster [INF] osd.4 marked >>> itself dead as of e91 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.242209 <tel:55242209>+0800 >>> mon.node2 (mon.1) 11574 <tel:111574> : cluster [INF] osd.0 marked >>> itself dead as of e91 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.233774 <tel:55233774>+0800 osd.5 >>> (osd.5) 3 : cluster [WRN] Monitor daemon marked osd.5 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:13:55.236038 <tel:55236038>+0800 osd.4 >>> (osd.4) 3 : cluster [WRN] Monitor daemon marked osd.4 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:14:00.109136 <tel:109136>+0800 >>> mon.node2 (mon.1) 11589 <tel:111589> : cluster [WRN] Health check >>> update: Degraded data redundancy: 1/8 objects degraded (12.500 >>> <tel:12500>%), 1 pg degraded, 188 pgs undersized (PG_DEGRADED) >>> >>> 2023-12-21 <tel:20231221>T01:14:00.311644 <tel:311644>+0800 >>> mon.node2 (mon.1) 11590 <tel:111590> : cluster [INF] osd.3 marked >>> itself dead as of e94 >>> >>> 2023-12-21 <tel:20231221>T01:14:00.311410 <tel:311410>+0800 osd.3 >>> (osd.3) 3 : cluster [WRN] Monitor daemon marked osd.3 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:14:01.922553 <tel:01922553>+0800 >>> mgr.node1.apderh (mgr.34343) 1 <tel:343431> : cluster [ERR] Failed >>> to load ceph-mgr modules: k8sevents >>> >>> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0 >>> (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it >>> is still running >>> >>> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >>> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 calling >>> monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >>> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >>> calling monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >>> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >>> election >>> >>> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >>> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 calling >>> monitor election >>> >>> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >>> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new >>> leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >>> >>> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >>> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >>> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >>> >>> >>> >>> the osd log of node1: >>> >>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6804 >>> <http://10.40.10.202:6804> osd.6 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6812 >>> <http://10.40.10.202:6812> osd.7 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6820 >>> <http://10.40.10.202:6820> osd.8 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6828 >>> <http://10.40.10.202:6828> osd.9 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: reset >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: 0x562066 <tel:09562066>bbc000 >>> handle_response ret = 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: got initial server challenge >>> 5fd5d9eca256ae69 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: validate_tickets: want=53 need=0 have=53 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: validate_tickets want 53 have 0 need 53 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: want=53 need=53 have=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: build_request >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 20 <tel:864020> cephx client: old ticket len=96 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: get auth session key: client_challenge >>> 7ca3f7c3ef30b2bf >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: 0x562066 <tel:09562066>bbc000 >>> handle_response ret = 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: get_auth_session_key >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply got 1 keys >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: got key for service_id auth >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply got encrypted ticket >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=2 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply service auth >>> secret_id 2 session_key AQDIIINlUlSmKBAAvL5rKyyIb6vt89rxUGlsxg== >>> validity=259200.000000 <tel:259200000000> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket >>> expires=2023-12-24 <tel:20231224>T01:13:44.682217 >>> <tel:44682217>+0800 renew_after=2023-12-23 >>> <tel:20231223>T07:13:44.682217 <tel:44682217>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: want=53 need=53 have=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: got connection bl 84 and extra >>> tickets 550 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: got connection_secret 64 bytes >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply got 3 keys >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: got key for service_id mon >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply service mon >>> secret_id 4 session_key AQDIIINlWvamKBAAxOaZsuCMdgutax1n2ZRbZQ== >>> validity=3600.000000 <tel:3600000000> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket >>> expires=2023-12-21 <tel:20231221>T02:13:44.682256 >>> <tel:44682256>+0800 renew_after=2023-12-21 >>> <tel:20231221>T01:58:44.682256 <tel:44682256>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: got key for service_id osd >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply service osd >>> secret_id 4 session_key AQDIIINlXAinKBAAOwTlViH3CJRgSvj5Kn9qtA== >>> validity=3600.000000 <tel:3600000000> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket >>> expires=2023-12-21 <tel:20231221>T02:13:44.682270 >>> <tel:44682270>+0800 renew_after=2023-12-21 >>> <tel:20231221>T01:58:44.682270 <tel:44682270>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: got key for service_id mgr >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply service mgr >>> secret_id 4 session_key AQDIIINlUiGnKBAAr6WBaSawzVyDoqEs8cENdg== >>> validity=3600.000000 <tel:3600000000> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: verify_service_ticket_reply ticket >>> expires=2023-12-21 <tel:20231221>T02:13:44.682286 >>> <tel:44682286>+0800 renew_after=2023-12-21 >>> <tel:20231221>T01:58:44.682286 <tel:44682286>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: got extra service_tickets >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: validate_tickets want 53 have 53 need 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx: validate_tickets want 53 have 53 need 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 20 <tel:864020> cephx client: need_tickets: want=53 have=53 need=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> -1 <tel:86401> monclient: _check_auth_rotating possible clock skew, >>> rotating keys expired way too early (before 2023-12-21 >>> <tel:20231221>T00:13:44.682376 <tel:44682376>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640 >>> 10 <tel:864010> cephx client: build_rotating_request >>> >>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.201:6804 >>> <http://10.40.10.201:6804> osd.1 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.201:6812 >>> <http://10.40.10.201:6812> osd.2 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 7fcd27c2e640 >>> -1 osd.0 83 heartbeat_check: no reply from 10.40.10.200:6814 >>> <http://10.40.10.200:6814> osd.3 ever on either front or back, first >>> ping sent 2023-12-21 <tel:20231221>T01:13:23.148457 >>> <tel:23148457>+0800 (oldest deadline 2023-12-21 >>> <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >>> >>> >>> >>> >>> the osd log of node2: >>> >>> >>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640 >>> 10 <tel:64010> cephx: validate_tickets want 53 have 53 need 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640 >>> 20 <tel:64020> cephx client: need_tickets: want=53 have=53 need=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640 >>> 10 <tel:64010> auth: dump_rotating: >>> >>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640 >>> 10 <tel:64010> auth: id 17 AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== >>> expires 2023-12-21 <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 7fa573a6d640 >>> 10 <tel:64010> auth: id 18 AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== >>> expires 2023-12-21 <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 7fa573a6d640 >>> 10 <tel:64010> auth: id 19 AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== >>> expires 2023-12-21 <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 20 <tel:58129364020> AuthRegistry(0x7ffc86c1e7b0) get_handler >>> peer_type 4 method 2 cluster_methods [2] service_methods [2] >>> client_methods [2] >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> cephx: verify_authorizer decrypted service osd >>> secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>> 7fa581293640 0 <tel:5812936400> auth: could not find secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> auth: dump_rotating: >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> auth: id 17 >>> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21 >>> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> auth: id 18 >>> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21 >>> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640 >>> 10 <tel:58129364010> auth: id 19 >>> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21 >>> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >>> 7fa581293640 0 <tel:5812936400> cephx: verify_authorizer could not >>> get service secret for service osd secret_id=4 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 7fa573a6d640 >>> 10 <tel:64010> cephx: validate_tickets want 53 have 53 need 0 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 7fa573a6d640 >>> 20 <tel:64020> cephx client: need_tickets: want=53 have=53 need=0 >>> >>> >>> >>> the mon log of node1: >>> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> start_session entity_name=mgr.node1.apderh >>> global_id=14184 <tel:14184> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: start_session >>> server_challenge 524ce5eea3e4dd75 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: handle_request >>> get_auth_session_key for mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 20 <tel:964020> cephx server mgr.node1.apderh: checking key: >>> req.key=f41952 <tel:41952>c0f02e6c0f expected_key=f41952 >>> <tel:41952>c0f02e6c0f >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 20 <tel:964020> cephx server mgr.node1.apderh: checking old_ticket: >>> secret_id=2 len=112, old_ticket_may_be_omitted=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 20 <tel:964020> cephx server mgr.node1.apderh: decoded old_ticket: >>> global_id=14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: allowing reclaim of >>> global_id 14184 <tel:14184> (valid ticket presented, will encrypt >>> new ticket) >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket_reply encoding 1 tickets >>> with secret AQBmKIJlvHeYJRAAf5h+auT4c9Ilo/nlUqRJlg== >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service auth secret_id 2 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: adding key for >>> service mon >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: adding key for >>> service mds >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: adding key for >>> service osd >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx server mgr.node1.apderh: adding key for >>> service mgr >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket_reply encoding 4 tickets >>> with secret AQDIIINldY1mGRAApH04rNRZtuyAUrQn5PNG7A== >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service mon secret_id 4 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service mds secret_id 4 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service osd secret_id 4 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640 >>> 10 <tel:964010> cephx: build_service_ticket service mgr secret_id 4 >>> ticket_info.ticket.name >>> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >>> 14184 <tel:14184> >>> >>> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >>> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command >>> mon_command({"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"} >>> v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >>> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >>> from='mgr.14184 10.40.10.200:0/2514043697 >>> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >>> cmd=[{"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"}]: >>> dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >>> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command >>> mon_command({"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"} >>> v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >>> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >>> from='mgr.14184 10.40.10.200:0/2514043697 >>> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >>> cmd=[{"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"}]: >>> dispatch >>> >>> >>> >>> the mon log of node2: >>> >>> >>> 2023-12-21 <tel:20231221>T01:13:10.292 <tel:10292>+0800 >>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >>> {"time_micros": 1703092390293629, "job": 1145, "event": >>> "table_file_deletion", "file_number": 2272} >>> >>> 2023-12-21 <tel:20231221>T01:13:10.295 <tel:10295>+0800 >>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >>> {"time_micros": 1703092390296385, "job": 1145, "event": >>> "table_file_deletion", "file_number": 2270} >>> >>> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >>> {"time_micros": 1703092390304703, "job": 1145, "event": >>> "table_file_deletion", "file_number": 2269} >>> >>> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >>> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >>> compaction starting >>> >>> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"} >>> v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"}]: >>> dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"} >>> v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix":"config >>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"}]: >>> dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.142 <tel:51142>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix": "config dump", "format": "json"} v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.143 <tel:51143>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix": "config dump", "format": "json"}]: dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command({"prefix": "auth get", "entity": "client.admin"} v 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch >>> >>> 2023-12-21 <tel:20231221>T01:13:52.247 <tel:52247>+0800 >>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >>> mon_command([{prefix=config-key set, key=mgr/cephadm/host.node1}] v >>> 0) v1 >>> >>> 2023-12-21 <tel:20231221>T01:13:52.260 <tel:52260>+0800 >>> 7f88b96c1640 0 <tel:16400> log_channel(audit) log [INF] : >>> from='mgr.14252 10.40.10.201:0/357060750 >>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >>> >>> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 >>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >>> _set_new_cache_sizes cache_size:1020054731 >>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >>> >>> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 7f88bd6c9640 >>> 20 <tel:964020> cephx keyserver: prepare_rotating_update before: >>> data.rotating_ver=17 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640 >>> 10 <tel:64010> start_session entity_name=osd.0 global_id=14205 >>> <tel:14205> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640 >>> 10 <tel:64010> cephx server osd.0: start_session server_challenge >>> dba982c2bced072d >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640 >>> 10 <tel:64010> start_session entity_name=osd.3 global_id=14379 >>> <tel:14379> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640 >>> 10 <tel:64010> cephx server osd.3: start_session server_challenge >>> e9db92f3f5af188c >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640 >>> 10 <tel:64010> start_session entity_name=mgr.node1.apderh >>> global_id=14190 <tel:14190> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640 >>> 10 <tel:64010> cephx server mgr.node1.apderh: start_session >>> server_challenge b9809292 <tel:9809292>ea2e62f6 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640 >>> 10 <tel:64010> start_session entity_name=osd.5 global_id=24253 >>> <tel:24253> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640 >>> 10 <tel:64010> cephx server osd.5: start_session server_challenge >>> 1be9f89b4bd9cdbc >>> >>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 7f88b9ec2640 >>> 10 <tel:264010> start_session entity_name=osd.4 global_id=14406 >>> <tel:14406> is_new_global_id=0 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 7f88b9ec2640 >>> 10 <tel:264010> cephx server osd.4: start_session server_challenge >>> cca4694d496c39a8 >>> >>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 7f88beecc640 >>> 10 <tel:64010> cephx server osd.0: handle_request >>> get_auth_session_key for osd.0 >>> >>> >>> _______________________________________________ >>> ceph-users mailing list --ceph-users(a)ceph.io >>> To unsubscribe send an email toceph-users-leave(a)ceph.io >> >> -- >> -- >> >> Andrea Martra >> Senior Manager >> +39 393 9048451 >> >> Open Technologies Sas >> Via Michele Buniva, 26 - 10064 Pinerolo (TO) >> Web:http://www.oscct.it >> >> ** Open Source Cloud Computing Technologies ** >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- -- Andrea Martra Senior Manager +39 393 9048451 Open Technologies Sas Via Michele Buniva, 26 - 10064 Pinerolo (TO) Web: http://www.oscct.it ** Open Source Cloud Computing Technologies **

1 0

Re: Host buckets in crush map
by Iztok Gregori 06 May '26

06 May '26

Hi! Sometimes I find a solution just minutes after I click send... On 06/05/26 11:05, Iztok Gregori via ceph-users wrote: > Hi to all! > > I have 2 questions regarding host buckets in the crush map; > > 1. The name of a "host bucket" is just a label (which can have an > arbitrary value) or is used in some way or form (so it has to be a real > short or full hostname)? > > 2. The second question is more complex, how to avoid, upon host reboot > (or during the OSD activation), the creation of a new host bucket? I'll > try to explain myself better. > [cut] > > So, the question remains, how to "pass" to ceph the correct host bucket > name? > > I think that the solution is to put "host=node01-int" in ceph.conf, but > where? Under the "[osd]" sections ("[osd.*]"???) or I have to insert it > under every single osd ([osd.12], [osd.13]...)? Crush location can be defined in ceph.conf as: crush_location = root=default row=a rack=a2 chassis=a2a host=a2a1 Or the management can be disabled with (also in ceph.conf): osd_crush_update_on_start = false So, with this the question number 2 is answered, but for the first question I'm still not sure about it. Thanks Iztok

1 0

Host buckets in crush map
by Iztok Gregori 06 May '26

06 May '26

Hi to all! I have 2 questions regarding host buckets in the crush map; 1. The name of a "host bucket" is just a label (which can have an arbitrary value) or is used in some way or form (so it has to be a real short or full hostname)? 2. The second question is more complex, how to avoid, upon host reboot (or during the OSD activation), the creation of a new host bucket? I'll try to explain myself better. The host buckets in our environment have the name of the interface not the name of the host itself (if a node's "hostname -s" is "node01" this node is present in the crush map as "node01-int", to represent the interface where Ceph is communicating, cluster/private network). Furthermore each host bucket can be in a different crush root (the "default" crush root is empty). I just reinstalled some nodes with Rocky 9 and Ceph Quincy (see my previous messages for more info) and upon activation of the OSDs a new host bucket is created and placed (with all the OSDs) in the default root (The name of this host bucket is node "hostname -s", like "node01"). Because it is in the wrong root, I've renamed it to the interface name ("node01-int"), moved it to the correct root and deleted the "wrong" host bucket ("node01"). Everything seemed to be working fine, but to be sure I've rebooted (again) a freshly reinstalled node. Upon reboot the "wrong" host bucket (node01) is recreated in the default root and all the OSDs are moved there leaving the host bucket "node01-int" empty. Clearly upon boot the ceph-osd daemons pass the name "hostname -s" to the monitors, which (I suppose) check if this name exists in the crush map, if it doesn't a new host bucket is created (or something along this lines). In fact, when I changed the hostname of my node to "node01-int", after reboot, the OSDs were placed under the correct host bucket (and, more importantly, under the correct root). If the answer on my question number 1. is that the host bucket name is just a label, as long as its in the correct root, doesn't matter how the node is called (and I could just put all the host buckets name without the "*-int" suffix). But if is not a label I need to have it with the interface name. So, the question remains, how to "pass" to ceph the correct host bucket name? I think that the solution is to put "host=node01-int" in ceph.conf, but where? Under the "[osd]" sections ("[osd.*]"???) or I have to insert it under every single osd ([osd.12], [osd.13]...)? If you read this line, congratulation you reached the finish! :-) Thank you! Iztok P.S. I'm not using cephadm, all the commands are manual. -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. http://www.elettra.eu

1 0

Re: MDS and failure domains
by Gregory Farnum 06 May '26

06 May '26

Unfortunately, this was recently IDed as a major weakness of the current MDS rank configuration system. :/ You can follow https://tracker.ceph.com/issues/75892 which was filed to work on this. -Greg On Tue, May 5, 2026 at 11:34 PM Robert Sander via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > Given a cluster with multiple racks or rooms where the rack or room is > the failure domain for replicated pools. > > Given one or more CephFS volumes (filesystems) with at least one active > MDS each and a standby-replay for each active MDS. > > Each room or rack contains multiple MDS daemons (one per host). > > Is there a possibility to guarantee that the standby-replay MDS for each > active MDS does not run in the same failure domain (rack or room)? > > Because when both run in the same failure domain and that domain fails > there is no point in having a standby-replay. > > Are there any ideas on how to solve this? > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > <https://www.google.com/maps/search/Schwedter+Str.+8%2F9b,+10119+Berlin?entr…> > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

MDS and failure domains
by Robert Sander 06 May '26

06 May '26

Hi, Given a cluster with multiple racks or rooms where the rack or room is the failure domain for replicated pools. Given one or more CephFS volumes (filesystems) with at least one active MDS each and a standby-replay for each active MDS. Each room or rack contains multiple MDS daemons (one per host). Is there a possibility to guarantee that the standby-replay MDS for each active MDS does not run in the same failure domain (rack or room)? Because when both run in the same failure domain and that domain fails there is no point in having a standby-replay. Are there any ideas on how to solve this? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Ceph Dashboard fails on tenant RGW buckets (NoSuchBucket → 500)
by farhad kh 05 May '26

05 May '26

Ceph Dashboard fails with 500 Internal Server Error when operating on RGW buckets that belong to a *tenant*, even though the bucket exists and is accessible via radosgw-admin. RGW returns NoSuchBucket because the Dashboard queries the bucket *without the tenant prefix* ----------- 13:02:06.832428866Z"} {"log":"debug 2026-05-04T13:02:06.838+0000 7fbc29896640 0 [dashboard ERROR rest_client] RGW REST API failed GET req status: 404\n","stream":"stderr","time":"2026-05-04T13:02:06.841073842Z"} {"log":"debug 2026-05-04T13:02:06.838+0000 7fbc29896640 0 [dashboard ERROR exception] Internal Server Error\n","stream":"stderr","time":"2026-05-04T13:02:06.842785189Z"} {"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2026-05-04T13:02:06.842818184Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/services/exception.py\", line 47, in dashboard_exception_handler\n","stream":"stderr","time":"2026-05-04T13:02:06.842831624Z"} {"log":" return handler(*args, **kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.842841016Z"} {"log":" File \"/lib/python3.9/site-packages/cherrypy/_cpdispatch.py\", line 54, in __call__\n","stream":"stderr","time":"2026-05-04T13:02:06.842851035Z"} {"log":" return self.callable(*self.args, **self.kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.842983312Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py\", line 264, in inner\n","stream":"stderr","time":"2026-05-04T13:02:06.843004859Z"} {"log":" ret = func(*args, **kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.843015376Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py\", line 193, in wrapper\n","stream":"stderr","time":"2026-05-04T13:02:06.843024549Z"} {"log":" return func(*vpath, **params)\n","stream":"stderr","time":"2026-05-04T13:02:06.84303376Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\", line 702, in set\n","stream":"stderr","time":"2026-05-04T13:02:06.843050003Z"} {"log":" locking = self._get_locking(uid, daemon_name, bucket_name)\n","stream":"stderr","time":"2026-05-04T13:02:06.843059741Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\", line 443, in _get_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843069083Z"} {"log":" return rgw_client.get_bucket_locking(bucket_name)\n","stream":"stderr","time":"2026-05-04T13:02:06.843078375Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 538, in func_wrapper\n","stream":"stderr","time":"2026-05-04T13:02:06.843087524Z"} {"log":" return func(\n","stream":"stderr","time":"2026-05-04T13:02:06.843096712Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\", line 899, in get_bucket_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843105575Z"} {"log":" raise e\n","stream":"stderr","time":"2026-05-04T13:02:06.843114638Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\", line 881, in get_bucket_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843123252Z"} {"log":" result = request() # type: ignore\n","stream":"stderr","time":"2026-05-04T13:02:06.843132275Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 324, in __call__\n","stream":"stderr","time":"2026-05-04T13:02:06.843141087Z"} {"log":" resp = self.rest_client.do_request(method, self._gen_path(), params,\n","stream":"stderr","time":"2026-05-04T13:02:06.843163504Z"} {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 422, in do_request\n","stream":"stderr","time":"2026-05-04T13:02:06.843173809Z"} {"log":" raise RequestException(\n","stream":"stderr","time":"2026-05-04T13:02:06.843367851Z"} {"log":"dashboard.rest_client.RequestException: RGW REST API failed request with status code 404\n","stream":"stderr","time":"2026-05-04T13:02:06.843382042Z"} {"log":"(b'{\"Code\":\"NoSuchBucket\",\"Message\":\"\",\"BucketName\":\"farhad\",\"RequestId\":\"tx000'\n","stream":"stderr","time":"2026-05-04T13:02:06.843391177Z"} {"log":" b'00ab80c789f95808ff-0069f898ce-110717032-vanak\",\"HostId\":\"110717032-vanak-teh'\n","stream":"stderr","time":"2026-05-04T13:02:06.843400934Z"} {"log":" b'ran\"}')\n","stream":"stderr","time":"2026-05-04T13:02:06.843410095Z"} {"log":"debug 2026-05-04T13:02:06.842+0000 7fbc29896640 0 [dashboard ERROR request] [::ffff:172.20.26.54:49876] [PUT] [500] [0.619s] [admin] [513.0B] /api/rgw/bucket/khedrian%2Ffarhad\n","stream":"stderr","time":"2026-05-04T13:02:06.846732997Z"} {"log":"debug 2026-05-04T13:02:06.842+0000 7fbc29896640 0 [dashboard ERROR request] [b'{\"status\": \"500 Internal Server Error\", \"detail\": \"The server encountered an unexpected condition which prevented it from fulfilling the request.\", \"request_id\": \"a3abea21-d73c-4d26-a8bb-1d99fc53a466\"} ']\n","stream":"stderr","time":"2026-05-04T13:02:06.847608634Z"} {"log":"debug 2026-05-04T13:02:06.846+0000 7fbc29896640 0 [dashboard INFO request] [::ffff:172.20.26.54:49876] [PUT] [500] [0.622s] [admin] [513.0B] /api/rgw/bucket/khedrian%2Ffarhad\n","stream":"stderr","time":"2026-05-04T13:02:06.849427429Z"} {"log":"debug 2026-05-04T13:02:07.322+0000 7fbc28053640 0 [dashboard INFO request] [::ffff:172.20.26.54:64634] [GET] [200] [0.691s] [admin] [295.0B] /api/rgw/daemon\n","stream":"stderr","time":"2026-05-04T13:02:07.324402049Z"} {"log":"debug 2026-05-04T13:02:07.850+0000 7fbc2504d640 0 [dashboard INFO request] [::ffff:172.20.26.54:49870] [GET] [200] [0.004s] [admin] [22.0B] /api/prometheus/notifications\n","stream":"stderr","time":"2026-05-04T13:02:07.855794264Z"} {"log":"debug 2026-05-04T13:02:08.110+0000 7fbc5c2fb640 0 log_channel(cluster) log [DBG] : pgmap v611376: 449 pgs: 449 active+clean; 202 MiB data, 10 GiB used, 1.2 TiB / 1.2 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 8 op/s\n","stream":"stderr","time":"2026-05-04T13:02:08.113933657Z"} {"log":"debug 2026-05-04T13:02:10.111+0000 7fbc5c2fb640 0 log_channel(cluster) log [DBG] : pgmap v611377: 449 pgs: 449 active+clean; 202 MiB data, 10 GiB used, 1.2 TiB / 1.2 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s\n","stream":"stderr","time":"2026-05-04T13:02:10.115433742Z"} {"log":"debug 2026-05-04T13:02:10.211+0000 7fbc2484c640 0 [dashboard INFO request] [::ffff:172.20.26.54:49876] [GET] [200] [0.003s] [admin] [22.0B] /api/prometheus/notifications\n","stream":"stderr","time":"2026-05-04T13:02:10.2151369Z"} ---------------- root@dev-ceph01:~# ceph -v ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable) { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::khedrian:farhad/*" } ] } - Bucket exists and is accessible via radosgw-admin - Issue only occurs for *tenant buckets* - Non‑tenant buckets work correctly - Appears to be a *Dashboard RGW tenant parsing bug* - Possibly missing tenant propagation when calling get_bucket_locking() and similar RGW APIs can anyone explain this ? what should i do ?

1 0

[ Radosgw Admin API] - How to include user tenant in API requests
by wissem mimouna 05 May '26

05 May '26

Hi All, I'm trying to find out how to use the radsogw admin api to get user usage information ,hence the user has a tenant and I trying to pass : > /admin/bucket?uid=tenant$user >/admin/bucket?uid=user >/admin/bucket?uid=user&tenant=tenant >/admin/bucket?uid=tenant/user But I always get an empty response (https://docs.ceph.com/en/latest/radosgw/adminops/#id60) May be there is another way to call the api with user and tenant ? Thanks

1 0

Ceph Tech Talk and CFP Reminder!!
by Anthony Middleton 04 May '26

04 May '26

Hello, Ceph friends! I hope your week is off to a great start. I wanted to let you know that our next Ceph Tech Talk <https://ceph.io/en/community/tech-talks/> is on *Monday*, May 11* at 9 AM PDT* w/ *Shraddha Agrawal*. *Title:* Containerized deployment of Ceph with KCLI *Abstract*: Cephadm is the recommended approach for deploying Ceph clusters, but many developers still depend on bare metal environments for testing and development. In this session, Shraddha will introduce KCLI as a way to bridge that gap—making it easier to run Cephadm-powered clusters locally with minimal setup. This talk is especially useful for developers looking to streamline their environments and reduce setup overhead. The invite is on the Ceph Community Calendar: https://t.ly/CephTechTalks *ALSO!!!* The CFP for Ceph Days Seattle <https://ceph.io/en/community/events/2026/ceph-days-seattle/> closes *tomorrow*, May 5* (Revenge of the Fifth)!* This Ceph community event will be at the SURF Incubator on May 28. Submit now: https://t.ly/seattleCFP2026. Take care, and please reach out if you have any questions. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0