- ceph-users - lists.ceph.io

Re: Best "modern" approach to drain/fill a OSD server
by Anthony D'Atri 27 Apr '26

27 Apr '26

> Hi to all! > > I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion...) without impacting on the performance of the cluster. You'll impact performance no matter what you do, but to differing degrees. The least impact is making the least change: set noout destroy one OSD redeploy that OSD unset noout wait for recovery At scale that doesn't, well, scale. > We are using the 'ceph-gentle-reweight' script, which is running fine, but we are interested to know if there is a newer approach for this job. What Ceph release are you running? That approach works, but will take a long time and you need to have enough free capacity on other nodes to hold that node's share of data. > > Searching the web I found a pdf of the talk of Dan van der Ster about "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about using the 'balancer' in 'upmap' mode If you're running, say, Mimic or later, you typically would already have the balancer on; upmap mode is the default. Run ceph balancer status When using upmaps, you want to avoid "ceph osd reweight-by-utilization" and ensure that the REWEIGHT column of "ceph osd df" or "ceph osd tree" is all 1.0000 > (and a "magic" script) to move the PGs from/to the OSDs in the background. > > So the balancer upmap approach can be used to drain a node following this procedure: Be sure to disable backfill and rebalancing, you want no backfilling in progress when you run upmap-remapped.py > - set balancer tunables (like 'target_max_misplaced_ratio') > - activate the balancer in upmap mode (if not already active) > - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain > - run the magic script 'upmap-remapped.py' > - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain > - do stuff on the node (remove, reinstall etc etc). If you are solving for safety over time, sure. Especially if you have replicated size=2 pools or EC N+1. Another approach is to set the flags, destroy the OSDs on a single host, redeploy them, set your backfill/recovery throttles down to 1, and unset the flags. This will be somewhat more risky as you're reducing redundancy until the recovery completes, and more will happen in parallel. Whether you can tolerate that depends in part on your hardware, CRUSH topology, how many hosts, replication vs EC, etc. > > But what is the implication for the whole cluster, how this background job affects its performance? We need more info. > > I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into? Yes. Setting that lower than default can help avoid certain intermediate states that can result in a fairly-full cluster's OSDs going nearfull/backfillfull. > > And there are other approaches for the job? > > Thanks a lot! > Iztok > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

CSC Meeting Minutes April 27th
by Patrick Donnelly 27 Apr '26

27 Apr '26

Good day, Here are the meeting minutes for today's short call: * [Anthony M and Emmanuel Ameh] Meet the next Tech Writer * [Anthony M] CSC/Board Q2 Meeting will be on Mon. Jun 1, 8:00 - 9:00 am PDT (we will invite the board to the CSC Meeting) Please add topics to the meeting pad - https://pad.ceph.com/p/CSC_-_board-meeting-Q2 by May 25th * [Anthony M] Voting for CSC Co-Leads ends Friday, May 1st at 11:59 pm PDT Have a good week, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Wannes Smet 27 Apr '26

27 Apr '26

Thanks for bringing this presentation to my attention. I'm in a similar place atm. where I'll likely introduce a change in our cluster which will cause "massive" data reloction. The "leap of faith" drawing seems very recognizable. I think here's the recording of the presentation. I'll definitily watch that one: https://www.youtube.com/watch?v=6PQYHlerJ8k&list=PLrBUGiINAakMz4n_oOeiueeUk… ________________________________ From: Iztok Gregori via ceph-users <ceph-users(a)ceph.io> Sent: Monday, April 27, 2026 12:12 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Best "modern" approach to drain/fill a OSD server Hi to all! I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion. . . ) without impacting on the performance of the cluster. We are using the 'ceph-gentle-reweight' ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi to all! I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion...) without impacting on the performance of the cluster. We are using the 'ceph-gentle-reweight' script, which is running fine, but we are interested to know if there is a newer approach for this job. Searching the web I found a pdf of the talk of Dan van der Ster about "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about using the 'balancer' in 'upmap' mode (and a "magic" script) to move the PGs from/to the OSDs in the background. So the balancer upmap approach can be used to drain a node following this procedure: - set balancer tunables (like 'target_max_misplaced_ratio') - activate the balancer in upmap mode (if not already active) - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain - run the magic script 'upmap-remapped.py' - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain - do stuff on the node (remove, reinstall etc etc). But what is the implication for the whole cluster, how this background job affects its performance? I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into? And there are other approaches for the job? Thanks a lot! Iztok _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Best "modern" approach to drain/fill a OSD server
by Iztok Gregori 27 Apr '26

27 Apr '26

Hi to all! I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion...) without impacting on the performance of the cluster. We are using the 'ceph-gentle-reweight' script, which is running fine, but we are interested to know if there is a newer approach for this job. Searching the web I found a pdf of the talk of Dan van der Ster about "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about using the 'balancer' in 'upmap' mode (and a "magic" script) to move the PGs from/to the OSDs in the background. So the balancer upmap approach can be used to drain a node following this procedure: - set balancer tunables (like 'target_max_misplaced_ratio') - activate the balancer in upmap mode (if not already active) - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain - run the magic script 'upmap-remapped.py' - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain - do stuff on the node (remove, reinstall etc etc). But what is the implication for the whole cluster, how this background job affects its performance? I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into? And there are other approaches for the job? Thanks a lot! Iztok

1 0

Re: Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')])
by Flemming Frandsen 27 Apr '26

27 Apr '26

let's encrypt certs, yes. On Fri, 24 Apr 2026 at 16:26, Konstantin Shalygin via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > Just guessing - it's Lets Encrypt certs? > Or may be something InternalCA when certificate issues by golang > application? > > > k > Sent from my iPhone > > > On 24 Apr 2026, at 15:54, Flemming Frandsen via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > I have a tentacle 20.2.1 cluster managed by cephadm, where I have > > configured a certificate for the rgw using something like: > > > > service_type: ingress > > service_id: rgw.default > > service_name: ingress.rgw.default > > placement: > > count: 3 > > label: rgw > > spec: > > backend_service: rgw.default > > first_virtual_router_id: 50 > > frontend_port: 443 > > monitor_port: 8080 > > virtual_ip: 10.0.40.3/24 > > networks: > > - 10.0.40.0/24 > > ssl_cert: | > > -----BEGIN CERTIFICATE----- > > *snip* > > -----END CERTIFICATE----- > > -----BEGIN CERTIFICATE----- > > *snip* > > -----END CERTIFICATE----- > > -----BEGIN RSA PRIVATE KEY----- > > *snip* > > -----END RSA PRIVATE KEY----- > > > > This initially worked perfectly, but I have just rotated the certificate > by > > generating the yaml with a fresh private key and associated certificate > > then installing it with: > > ceph orch apply -i new.yaml > > > > Clients accessing the rgw ingress get the correct certificate and work > > perfectly, both before and after the key rotation. > > > > > > I have a script that does this for me daily, and until the certificate > was > > rotated, everything was fine, but then I got this error: > > > > # ceph orch certmgr cert check > > > > Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > > valid (error: Invalid cert/key pair: [('x509 certificate routines', '', > > 'key values mismatch')]) > > > > After messing with the certificate a bunch I now get a completely > > different, but also worrying error: > > > > # ceph orch certmgr cert check > > - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > > valid (error: Invalid private key: [('DECODER routines', '', > > 'unsupported')]) > > > > The documentation offers no help: > > https://docs.ceph.com/en/tentacle/cephadm/certmgr/ > > > > The error seems to come from this code: > > > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > > > So the error is an amputated exception from somewhere in openssl but the > > actual exception is eaten by the code, so there is no way for me to debug > > the situation: > > > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > > > I have tried installing the old certificate and key, that only leads the > > the Invalid private key: [('DECODER routines', '', 'unsupported')] error. > > > > I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does > > nothing. > > > > I have even tried copying the key and cert pem files into the mgr > container > > and verified them using the openssl installed in the container. > > > > The certificates are generated by cert-manager, running in k8s and a > > cronjob that periodically pushes the new key/cert to the ceph cluster via > > ssh. > > > > Has anyone seen these types of errors before? > > > > -- > > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

2 1

Re: could not find secret_id
by Eugen Block 24 Apr '26

24 Apr '26

Hi, how did you end up in that situation (did something happen in the cluster)? Is it affecting only OSDs or other daemons as well? We did investigate such an issue one and a half years ago for a customer. They added hosts to one of the two clusters, at that time the customer had two sites, with rbd mirroring enabled. And I don't recall the exact details, we did solve it in an online meeting and I don't think anyone wrote a protocoll. And the session definitely wasn't recored. :-) It might have some remainder on the new host from previous tests, or some misconfiguration, I really can't tell anymore. I think in the end we ended up removing the OSDs, zapping them and adding them back. Of course, it could be a different root cause, not sure which role the mon leader can play in this. But I thought I'd share this tiny bit at least. :-) Regards, Eugen Zitat von Andrea Martra via ceph-users <ceph-users(a)ceph.io>: > Hi all, > I'm reaching back to this old thread from December 2023 because > we're investigating a very similar pattern at a customer site > running Ceph Reef 18.2.x. Logs show identical shape: > > AuthRegistry get_handler peer_type 4 method 2 ... > cephx: verify_authorizer decrypted service osd secret_id=3524 > auth: could not find secret_id=3524 > auth: dump_rotating: (local range ~971-973) > cephx: verify_authorizer could not get service secret ... > > Affecting 4 OSDs on 7 out of 39 nodes, persistent over days > (15M+ log lines in 3 days). No clock skew, quorum stable. > > @xiaowenhao111 or anyone else who hit this: did you ever > find a root cause or a reliable fix? > The original thread > didn't seem to reach a conclusion. Specifically: > > - Did restarting the affected OSDs clear it, or did the > pattern come back? > - Was a MON leader failover enough, or did you need to > rebuild anything? > - Any tracker issue filed for this? > > We're trying to reproduce on a test cluster (Reef 18.2.7) > by isolating a node hosting MON leader + OSDs via network > for 3+ hours, to match the scenario described in the > original thread. Happy to share findings back to the list. > > Thanks, > Andrea > > Il 21/12/23 02:42, xiaowenhao111 ha scritto: >> >> >> Hello ceph-users: >> >> First ,sorry my english ... >> >> >> I find a bug (16.2.10 <tel:16210> and 18.2.1), but i do not know why >> >> test steps: >> >> 1、config all hosts contain chronyd service >> >> 2、cephadm --image quay.io/ceph/ceph:v16.2.10 >> <http://quay.io/ceph/ceph:v16.2.10> bootstrap >> --dashboard-password-noupdate --mon-ip 10.40.10.200 >> <http://10.40.10.200> --cluster-network=10.40.10.0/24 >> <http://10.40.10.0/24> --skip-pull --allow-overwrite >> --skip-monitoring-stack --allow-mismatched-release >> >> 3、add hosts >> >> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node2 >> >> ceph orch host add node2 10.40.10.202 <tel:2104010202> --labels=_admin >> >> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node-1 >> >> ceph orch host add node-1 10.40.10.202 <tel:1104010202> --labels=_admin >> >> 4、add osds: >> >> ceph orch daemon add osd node1:/dev/sda >> >> ceph orch daemon add osd node1:/dev/sdb >> >> ceph orch daemon add osd node1:/dev/sdc >> >> ceph orch daemon add osd node1:/dev/sdd >> >> ceph orch daemon add osd node2:/dev/sda >> >> ceph orch daemon add osd node2:/dev/sdb >> >> ceph orch daemon add osd node2:/dev/sdc >> >> ceph orch daemon add osd node2:/dev/sdd >> >> 5、ceph osd pool create test_pool 512 512 <tel:512512> --size=3 >> >> 6、disconnect node1 (which is monitor leader) cluster and public network >> >> ifdown enp35s0f0 >> >> ifdown enp35s0f1 >> >> 7、stay overnight（more than three hours） >> >> 8、connect node1 cluster and public network >> >> ifup enp35s0f0 >> >> ifup enp35s0f1 >> >> >> then, the osds of node1 call get_auth_session_key from mon.node1 >> which is not new leader，get the wrong secret_id=4. >> >> the others report error "could not find secret_id=4" more than 40 minutes >> >> >> >> >> the ceph.log content of node1: >> >> >> 2023-12-20 <tel:20231220>T09:48:30.199224 <tel:30199224>+0800 >> mon.node1 (mon.0) 1891 <tel:01891> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-20 <tel:20231220>T09:49:37.816260 <tel:37816260>+0800 >> mon.node1 (mon.0) 1901 <tel:01901> : cluster [INF] Health check >> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >> >> 2023-12-20 <tel:20231220>T09:49:44.222046 <tel:44222046>+0800 >> mon.node1 (mon.0) 1902 <tel:01902> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-20 <tel:20231220>T09:50:00.000090 <tel:00000090>+0800 >> mon.node1 (mon.0) 1905 <tel:01905> : cluster [WRN] overall >> HEALTH_WARN Failed to place 1 daemon(s); 3 failed cephadm >> daemon(s); 1 pool(s) do not have an application enabled >> >> 2023-12-20 <tel:20231220>T09:50:51.916464 <tel:51916464>+0800 >> mon.node1 (mon.0) 1912 <tel:01912> : cluster [INF] Health check >> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >> >> 2023-12-20 <tel:20231220>T09:51:07.127982 <tel:07127982>+0800 >> mon.node1 (mon.0) 1916 <tel:01916> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0 >> (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >> election >> >> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new >> leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >> >> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >> >> >> the ceph.log content of node2: >> >> >> 2023-12-21 <tel:20231221>T01:10:01.065051 <tel:01065051>+0800 >> mon.node2 (mon.1) 11522 <tel:111522> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-21 <tel:20231221>T01:11:11.092640 <tel:11092640>+0800 >> mon.node2 (mon.1) 11531 <tel:111531> : cluster [INF] Health check >> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >> >> 2023-12-21 <tel:20231221>T01:11:19.197355 <tel:19197355>+0800 >> mon.node2 (mon.1) 11532 <tel:111532> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-21 <tel:20231221>T01:12:34.469403 <tel:34469403>+0800 >> mon.node2 (mon.1) 11547 <tel:111547> : cluster [INF] Health check >> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >> >> 2023-12-21 <tel:20231221>T01:12:42.476793 <tel:42476793>+0800 >> mon.node2 (mon.1) 11549 <tel:111549> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-21 <tel:20231221>T01:13:55.234163 <tel:55234163>+0800 >> mon.node2 (mon.1) 11572 <tel:111572> : cluster [INF] osd.5 marked >> itself dead as of e91 >> >> 2023-12-21 <tel:20231221>T01:13:55.236348 <tel:55236348>+0800 >> mon.node2 (mon.1) 11573 <tel:111573> : cluster [INF] osd.4 marked >> itself dead as of e91 >> >> 2023-12-21 <tel:20231221>T01:13:55.242209 <tel:55242209>+0800 >> mon.node2 (mon.1) 11574 <tel:111574> : cluster [INF] osd.0 marked >> itself dead as of e91 >> >> 2023-12-21 <tel:20231221>T01:13:55.233774 <tel:55233774>+0800 osd.5 >> (osd.5) 3 : cluster [WRN] Monitor daemon marked osd.5 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:13:55.236038 <tel:55236038>+0800 osd.4 >> (osd.4) 3 : cluster [WRN] Monitor daemon marked osd.4 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:14:00.109136 <tel:109136>+0800 >> mon.node2 (mon.1) 11589 <tel:111589> : cluster [WRN] Health check >> update: Degraded data redundancy: 1/8 objects degraded (12.500 >> <tel:12500>%), 1 pg degraded, 188 pgs undersized (PG_DEGRADED) >> >> 2023-12-21 <tel:20231221>T01:14:00.311644 <tel:311644>+0800 >> mon.node2 (mon.1) 11590 <tel:111590> : cluster [INF] osd.3 marked >> itself dead as of e94 >> >> 2023-12-21 <tel:20231221>T01:14:00.311410 <tel:311410>+0800 osd.3 >> (osd.3) 3 : cluster [WRN] Monitor daemon marked osd.3 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:14:01.922553 <tel:01922553>+0800 >> mgr.node1.apderh (mgr.34343) 1 <tel:343431> : cluster [ERR] Failed >> to load ceph-mgr modules: k8sevents >> >> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0 >> (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >> election >> >> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new >> leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >> >> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >> >> >> >> the osd log of node1: >> >> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.202:6804 <http://10.40.10.202:6804> osd.6 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.202:6812 <http://10.40.10.202:6812> osd.7 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.202:6820 <http://10.40.10.202:6820> osd.8 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.202:6828 <http://10.40.10.202:6828> osd.9 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: reset >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: 0x562066 >> <tel:09562066>bbc000 handle_response ret = 0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: got initial server >> challenge 5fd5d9eca256ae69 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: validate_tickets: >> want=53 need=0 have=53 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have 0 >> need 53 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: build_request >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 20 <tel:864020> cephx client: old ticket len=96 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: get auth session key: >> client_challenge 7ca3f7c3ef30b2bf >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: 0x562066 >> <tel:09562066>bbc000 handle_response ret = 0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: get_auth_session_key >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply got >> 1 keys >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id auth >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply got >> encrypted ticket >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket.secret_id=2 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> service auth secret_id 2 session_key >> AQDIIINlUlSmKBAAvL5rKyyIb6vt89rxUGlsxg== validity=259200.000000 >> <tel:259200000000> >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket expires=2023-12-24 <tel:20231224>T01:13:44.682217 >> <tel:44682217>+0800 renew_after=2023-12-23 >> <tel:20231223>T07:13:44.682217 <tel:44682217>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: got connection bl 84 >> and extra tickets 550 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: got connection_secret >> 64 bytes >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply got >> 3 keys >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mon >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket.secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> service mon secret_id 4 session_key >> AQDIIINlWvamKBAAxOaZsuCMdgutax1n2ZRbZQ== validity=3600.000000 >> <tel:3600000000> >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682256 >> <tel:44682256>+0800 renew_after=2023-12-21 >> <tel:20231221>T01:58:44.682256 <tel:44682256>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id osd >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket.secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> service osd secret_id 4 session_key >> AQDIIINlXAinKBAAOwTlViH3CJRgSvj5Kn9qtA== validity=3600.000000 >> <tel:3600000000> >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682270 >> <tel:44682270>+0800 renew_after=2023-12-21 >> <tel:20231221>T01:58:44.682270 <tel:44682270>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mgr >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket.secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> service mgr secret_id 4 session_key >> AQDIIINlUiGnKBAAr6WBaSawzVyDoqEs8cENdg== validity=3600.000000 >> <tel:3600000000> >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682286 >> <tel:44682286>+0800 renew_after=2023-12-21 >> <tel:20231221>T01:58:44.682286 <tel:44682286>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: got extra service_tickets >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have >> 53 need 0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have >> 53 need 0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 20 <tel:864020> cephx client: need_tickets: want=53 >> have=53 need=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 -1 <tel:86401> monclient: _check_auth_rotating >> possible clock skew, rotating keys expired way too early (before >> 2023-12-21 <tel:20231221>T00:13:44.682376 <tel:44682376>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: build_rotating_request >> >> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.201:6804 <http://10.40.10.201:6804> osd.1 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.201:6812 <http://10.40.10.201:6812> osd.2 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.200:6814 <http://10.40.10.200:6814> osd.3 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> >> >> >> the osd log of node2: >> >> >> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >> 7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have 53 >> need 0 >> >> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >> 7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53 >> have=53 need=0 >> >> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >> 7fa573a6d640 10 <tel:64010> auth: dump_rotating: >> >> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >> 7fa573a6d640 10 <tel:64010> auth: id 17 >> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21 >> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 >> 7fa573a6d640 10 <tel:64010> auth: id 18 >> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21 >> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 >> 7fa573a6d640 10 <tel:64010> auth: id 19 >> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21 >> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 20 <tel:58129364020> AuthRegistry(0x7ffc86c1e7b0) >> get_handler peer_type 4 method 2 cluster_methods [2] >> service_methods [2] client_methods [2] >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> cephx: verify_authorizer >> decrypted service osd secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 0 <tel:5812936400> auth: could not find secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> auth: dump_rotating: >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> auth: id 17 >> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21 >> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> auth: id 18 >> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21 >> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> auth: id 19 >> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21 >> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 0 <tel:5812936400> cephx: verify_authorizer could not >> get service secret for service osd secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 >> 7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have 53 >> need 0 >> >> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 >> 7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53 >> have=53 need=0 >> >> >> >> the mon log of node1: >> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> start_session >> entity_name=mgr.node1.apderh global_id=14184 >> <tel:14184> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >> start_session server_challenge 524ce5eea3e4dd75 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >> handle_request get_auth_session_key for mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh: >> checking key: req.key=f41952 <tel:41952>c0f02e6c0f >> expected_key=f41952 <tel:41952>c0f02e6c0f >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh: >> checking old_ticket: secret_id=2 len=112, old_ticket_may_be_omitted=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh: >> decoded old_ticket: global_id=14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >> allowing reclaim of global_id 14184 <tel:14184> (valid ticket >> presented, will encrypt new ticket) >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply >> encoding 1 tickets with secret >> AQBmKIJlvHeYJRAAf5h+auT4c9Ilo/nlUqRJlg== >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> auth secret_id 2 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >> name=mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >> of caps=3 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: adding >> key for service mon >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: adding >> key for service mds >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >> name=mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >> of caps=3 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: adding >> key for service osd >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >> name=mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >> of caps=3 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: adding >> key for service mgr >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >> name=mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >> of caps=3 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply >> encoding 4 tickets with secret >> AQDIIINldY1mGRAApH04rNRZtuyAUrQn5PNG7A== >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> mon secret_id 4 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> mds secret_id 4 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> osd secret_id 4 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> mgr secret_id 4 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command >> mon_command({"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"} v 0) >> v1 >> >> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >> from='mgr.14184 10.40.10.200:0/2514043697 >> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >> cmd=[{"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"}]: >> dispatch >> >> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command >> mon_command({"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"} >> v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >> from='mgr.14184 10.40.10.200:0/2514043697 >> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >> cmd=[{"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"}]: >> dispatch >> >> >> >> the mon log of node2: >> >> >> 2023-12-21 <tel:20231221>T01:13:10.292 <tel:10292>+0800 >> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >> {"time_micros": 1703092390293629, "job": 1145, "event": >> "table_file_deletion", "file_number": 2272} >> >> 2023-12-21 <tel:20231221>T01:13:10.295 <tel:10295>+0800 >> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >> {"time_micros": 1703092390296385, "job": 1145, "event": >> "table_file_deletion", "file_number": 2270} >> >> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >> {"time_micros": 1703092390304703, "job": 1145, "event": >> "table_file_deletion", "file_number": 2269} >> >> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"} >> v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"}]: >> dispatch >> >> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"} v 0) >> v1 >> >> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"}]: >> dispatch >> >> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:51.142 <tel:51142>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix": "config dump", "format": "json"} v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:51.143 <tel:51143>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix": "config dump", "format": "json"}]: dispatch >> >> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch >> >> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix": "auth get", "entity": "client.admin"} v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch >> >> 2023-12-21 <tel:20231221>T01:13:52.247 <tel:52247>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command([{prefix=config-key set, key=mgr/cephadm/host.node1}] v >> 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:52.260 <tel:52260>+0800 >> 7f88b96c1640 0 <tel:16400> log_channel(audit) log [INF] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> >> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88beecc640 10 <tel:64010> start_session entity_name=osd.0 >> global_id=14205 <tel:14205> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88beecc640 10 <tel:64010> cephx server osd.0: start_session >> server_challenge dba982c2bced072d >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.3 >> global_id=14379 <tel:14379> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88bf6cd640 10 <tel:64010> cephx server osd.3: start_session >> server_challenge e9db92f3f5af188c >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88beecc640 10 <tel:64010> start_session >> entity_name=mgr.node1.apderh global_id=14190 >> <tel:14190> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88beecc640 10 <tel:64010> cephx server mgr.node1.apderh: >> start_session server_challenge b9809292 <tel:9809292>ea2e62f6 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.5 >> global_id=24253 <tel:24253> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88bf6cd640 10 <tel:64010> cephx server osd.5: start_session >> server_challenge 1be9f89b4bd9cdbc >> >> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 >> 7f88b9ec2640 10 <tel:264010> start_session entity_name=osd.4 >> global_id=14406 <tel:14406> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 >> 7f88b9ec2640 10 <tel:264010> cephx server osd.4: start_session >> server_challenge cca4694d496c39a8 >> >> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 >> 7f88beecc640 10 <tel:64010> cephx server osd.0: handle_request >> get_auth_session_key for osd.0 >> >> >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io > > -- > -- > > Andrea Martra > Senior Manager > +39 393 9048451 > > Open Technologies Sas > Via Michele Buniva, 26 - 10064 Pinerolo (TO) > Web:http://www.oscct.it > > ** Open Source Cloud Computing Technologies ** > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Joshua Blanch 24 Apr '26

24 Apr '26

@Redo I think it might have something to do with how we are serializing the certs. https://github.com/ceph/ceph/blob/3dc6a60ce03606649480fd6341c0f0841d28c101/… it might be related to FIPS mode being stricter and requiring PKCS8 instead of TraditionalOpenSSL class PrivateFormat(utils.Enum): PKCS8 = "PKCS8" TraditionalOpenSSL = "TraditionalOpenSSL" Raw = "Raw" OpenSSH = "OpenSSH" https://cryptography.io/en/3.4.5/hazmat/primitives/asymmetric/serialization… On Fri, 24 Apr 2026 at 01:34, Redouane Kachach via ceph-users < ceph-users(a)ceph.io> wrote: > Hi Steve, > > Can you plz share the spec you used to configure the mgmt-gateway and the > active mgr logs? > > Best, > Redo. > > On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < > ceph-users(a)ceph.io> wrote: > > > I have a pre-production cluster running on Ubuntu 22.04 hosts using > > cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, > which > > went completely smoothly. We are a government facility so we are required > > to enable FIPS mode in the OS. I believe this last fact may be the cause > of > > the following: > > > > I wanted to enable mgmt-gateway so I made a service definition file to > > enable it and applied it. It appeared to work, but then “ceph health > > detail” showed: > > > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > > unsupported > > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > > routines] unsupported > > Module 'cephadm' has failed: [digital envelope routines] unsupported > > > > The mgmt-gateway service was started on all 3 nodes however. > > > > $ ceph orch ps --service_name mgmt-gateway > > NAME HOST PORTS STATUS REFRESHED AGE MEM > > USE MEM LIM VERSION IMAGE ID CONTAINER ID > > mgmt-gateway.alfred alfred running (4m) 3m ago 4m > > 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada > > mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m > > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (4m) 3m ago 4m > > 4452k - 1.26.1 c2348906c6ff 329705c23aad > > > > I then attempted to remove the service definition with ceph orch rm, but > > it only succeeded in removing it on 1 of the 3 nodes. The process is > still > > running on the other 2, and the warning is still present. “ceph mgr fail > && > > ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it > > comes right back, and the process is still present on 2 nodes (as shown > by > > “ceph orch ps”). > > > > $ ceph mgr fail && ceph orch rm mgmt-gateway > > Removed service mgmt-gateway > > > > $ ceph orch ps --service_name mgmt-gateway > > NAME HOST PORTS STATUS REFRESHED AGE MEM > > USE MEM LIM VERSION IMAGE ID CONTAINER ID > > mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h > > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (15h) 15h ago 15h > > 4448k - 1.26.1 c2348906c6ff 329705c23aad > > > > $ ceph health detail > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > > unsupported > > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > > routines] unsupported > > Module 'cephadm' has failed: [digital envelope routines] unsupported > > > > Despite the 2 processes running, the configured service IP does not > > respond to a ping or requests from curl. > > > > Since we were getting ready to move this cluster into production I > decided > > to look for help here rather than proceed experimenting further and risk > > mucking it up worse. How can I (a) diagnose what’s actually going wrong; > > (b) recover the cephadm module to get back to normal with no warnings; > and > > less importantly, (c) is there a way to get mgmt-gateway working in this > > environment? A little googling and AI querying suggests it may be an > issue > > with certs/ciphers so maybe I can provide my own? (Yes I know I could set > > up my own proxy; I was just trying to use the built-in tools where > > possible.) > > > > Thanks in advance. > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')])
by Konstantin Shalygin 24 Apr '26

24 Apr '26

Hi, Just guessing - it's Lets Encrypt certs? Or may be something InternalCA when certificate issues by golang application? k Sent from my iPhone > On 24 Apr 2026, at 15:54, Flemming Frandsen via ceph-users <ceph-users(a)ceph.io> wrote: > > I have a tentacle 20.2.1 cluster managed by cephadm, where I have > configured a certificate for the rgw using something like: > > service_type: ingress > service_id: rgw.default > service_name: ingress.rgw.default > placement: > count: 3 > label: rgw > spec: > backend_service: rgw.default > first_virtual_router_id: 50 > frontend_port: 443 > monitor_port: 8080 > virtual_ip: 10.0.40.3/24 > networks: > - 10.0.40.0/24 > ssl_cert: | > -----BEGIN CERTIFICATE----- > *snip* > -----END CERTIFICATE----- > -----BEGIN CERTIFICATE----- > *snip* > -----END CERTIFICATE----- > -----BEGIN RSA PRIVATE KEY----- > *snip* > -----END RSA PRIVATE KEY----- > > This initially worked perfectly, but I have just rotated the certificate by > generating the yaml with a fresh private key and associated certificate > then installing it with: > ceph orch apply -i new.yaml > > Clients accessing the rgw ingress get the correct certificate and work > perfectly, both before and after the key rotation. > > > I have a script that does this for me daily, and until the certificate was > rotated, everything was fine, but then I got this error: > > # ceph orch certmgr cert check > > Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > valid (error: Invalid cert/key pair: [('x509 certificate routines', '', > 'key values mismatch')]) > > After messing with the certificate a bunch I now get a completely > different, but also worrying error: > > # ceph orch certmgr cert check > - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > valid (error: Invalid private key: [('DECODER routines', '', > 'unsupported')]) > > The documentation offers no help: > https://docs.ceph.com/en/tentacle/cephadm/certmgr/ > > The error seems to come from this code: > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > So the error is an amputated exception from somewhere in openssl but the > actual exception is eaten by the code, so there is no way for me to debug > the situation: > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > I have tried installing the old certificate and key, that only leads the > the Invalid private key: [('DECODER routines', '', 'unsupported')] error. > > I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does > nothing. > > I have even tried copying the key and cert pem files into the mgr container > and verified them using the openssl installed in the container. > > The certificates are generated by cert-manager, running in k8s and a > cronjob that periodically pushes the new key/cert to the ceph cluster via > ssh. > > Has anyone seen these types of errors before? > > -- > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')])
by Redouane Kachach 24 Apr '26

24 Apr '26

Hi Flemming, certmgr at the moment doesn't support fullchain PEM certificates: https://tracker.ceph.com/issues/75710 Can you plz paste the spec you are currently using that is leading to the issue). Right now the format of cert/key has to be something like: service_type: rgw service_id: myrgw spec: ssl: true ssl_key: | -----BEGIN PRIVATE KEY----- MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQCM58IqqXGTCJof ..... QsJ69XOQV9Os37zG2/hELQ1ycBs2Jw== -----END PRIVATE KEY----- ssl_cert: | -----BEGIN CERTIFICATE----- MIIF4TCCA8mgAwIBAgIUI1F5+FeKTeF5J+xYLdkEDP9J0OMwDQYJKoZIhvcNAQEL .... fhTw47C8Dry+2bOaYQPe6qvoFQ+x -----END CERTIFICATE----- Best, Redo. On Fri, Apr 24, 2026 at 2:52 PM Flemming Frandsen via ceph-users < ceph-users(a)ceph.io> wrote: > I have a tentacle 20.2.1 cluster managed by cephadm, where I have > configured a certificate for the rgw using something like: > > service_type: ingress > service_id: rgw.default > service_name: ingress.rgw.default > placement: > count: 3 > label: rgw > spec: > backend_service: rgw.default > first_virtual_router_id: 50 > frontend_port: 443 > monitor_port: 8080 > virtual_ip: 10.0.40.3/24 > networks: > - 10.0.40.0/24 > ssl_cert: | > -----BEGIN CERTIFICATE----- > *snip* > -----END CERTIFICATE----- > -----BEGIN CERTIFICATE----- > *snip* > -----END CERTIFICATE----- > -----BEGIN RSA PRIVATE KEY----- > *snip* > -----END RSA PRIVATE KEY----- > > This initially worked perfectly, but I have just rotated the certificate by > generating the yaml with a fresh private key and associated certificate > then installing it with: > ceph orch apply -i new.yaml > > Clients accessing the rgw ingress get the correct certificate and work > perfectly, both before and after the key rotation. > > > I have a script that does this for me daily, and until the certificate was > rotated, everything was fine, but then I got this error: > > # ceph orch certmgr cert check > > Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > valid (error: Invalid cert/key pair: [('x509 certificate routines', '', > 'key values mismatch')]) > > After messing with the certificate a bunch I now get a completely > different, but also worrying error: > > # ceph orch certmgr cert check > - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > valid (error: Invalid private key: [('DECODER routines', '', > 'unsupported')]) > > The documentation offers no help: > https://docs.ceph.com/en/tentacle/cephadm/certmgr/ > > The error seems to come from this code: > > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > So the error is an amputated exception from somewhere in openssl but the > actual exception is eaten by the code, so there is no way for me to debug > the situation: > > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > I have tried installing the old certificate and key, that only leads the > the Invalid private key: [('DECODER routines', '', 'unsupported')] error. > > I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does > nothing. > > I have even tried copying the key and cert pem files into the mgr container > and verified them using the openssl installed in the container. > > The certificates are generated by cert-manager, running in k8s and a > cronjob that periodically pushes the new key/cert to the ceph cluster via > ssh. > > Has anyone seen these types of errors before? > > -- > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')])
by Flemming Frandsen 24 Apr '26

24 Apr '26

I have a tentacle 20.2.1 cluster managed by cephadm, where I have configured a certificate for the rgw using something like: service_type: ingress service_id: rgw.default service_name: ingress.rgw.default placement: count: 3 label: rgw spec: backend_service: rgw.default first_virtual_router_id: 50 frontend_port: 443 monitor_port: 8080 virtual_ip: 10.0.40.3/24 networks: - 10.0.40.0/24 ssl_cert: | -----BEGIN CERTIFICATE----- *snip* -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- *snip* -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- *snip* -----END RSA PRIVATE KEY----- This initially worked perfectly, but I have just rotated the certificate by generating the yaml with a fresh private key and associated certificate then installing it with: ceph orch apply -i new.yaml Clients accessing the rgw ingress get the correct certificate and work perfectly, both before and after the key rotation. I have a script that does this for me daily, and until the certificate was rotated, everything was fine, but then I got this error: # ceph orch certmgr cert check Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')]) After messing with the certificate a bunch I now get a completely different, but also worrying error: # ceph orch certmgr cert check - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid private key: [('DECODER routines', '', 'unsupported')]) The documentation offers no help: https://docs.ceph.com/en/tentacle/cephadm/certmgr/ The error seems to come from this code: https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… So the error is an amputated exception from somewhere in openssl but the actual exception is eaten by the code, so there is no way for me to debug the situation: https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… I have tried installing the old certificate and key, that only leads the the Invalid private key: [('DECODER routines', '', 'unsupported')] error. I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does nothing. I have even tried copying the key and cert pem files into the mgr container and verified them using the openssl installed in the container. The certificates are generated by cert-manager, running in k8s and a cronjob that periodically pushes the new key/cert to the ceph cluster via ssh. Has anyone seen these types of errors before? -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

1 0