April 2025 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

Announcing go-ceph v0.32.0
by Sven Anderson 12 Aug '25

12 Aug '25

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.32.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1

Squid 19.2.1 dashboard javascript error
by Chris Palmer 09 Aug '25

09 Aug '25

Firstly thank you so much for the 19.2.1 release. Initial testing suggests that the blockers that we had in 19.2.0 have all been resolved, so we are proceeding with further testing. We have noticed one small problem in 19.2.1 that was not present in 19.2.0 though. We use the older-style dashboard (mgr/dashboard/FEATURE_TOGGLE_DASHBOARD false). The problem happens on the Dashboard screen when health changes to WARN. If you click on WARN you get a small empty dropdown instead of the list of warnings. A javascript error is logged, and using browser inspection there is the additional bit of info that it happens in polyfill: 2025-02-07T15:59:00.970+0000 7f1d63877640 0 [dashboard ERROR frontend.error] (https://<redacted>:8443/#/dashboard): NG0901 Error: NG0901 at d.find (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3342365) at le.ngDoCheck (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3173112) at Qe (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225586) at bt (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225341) at cs (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225051) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259043) at jf (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3266563) at S1 (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259790) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259801) at fg (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3267248) Also, after this happens, no dropdowns work again until the page is forcibly refreshed. Environment is RPM install on Centos 9 Stream. I've created issue [0]. Thanks, Chris [0] https://tracker.ceph.com/issues/69867 <https://tracker.ceph.com/issues/69867?next_issue_id=69865&prev_issue_id=90>

4 9

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

Re: "BUG: soft lockup" with MDS
by Janek Bevendorff 10 Jul '25

10 Jul '25

Yes, they are older Dell PowerEdges. I have to check whether there's newer firmware, but we've been running Ceph for years without these problems. I checked the logs on the host on which I had a lockup just an hour ago, but there's nothing besides the expected hardreset messages. There are two older watchdog messages, but they are from March: -------------------------------------------------------------------------------- SeqNumber = 2089 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:16:03 Message = The watchdog timer expired. RawEventData = 0x03,0x00,0x02,0x33,0xFB,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- SeqNumber = 2088 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:06:41 Message = The watchdog timer expired. RawEventData = 0x02,0x00,0x02,0x01,0xF9,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- I grepped the logs of another host where it happened, but couldn't find any watchtdog messages there. I believe it's also unlikely that suddenly all MDS hosts (we have five active, five hot standbys, and one cold standby) start having hardware issues. I also ran a memtest on one of the hosts last week and couldn't find anything there either. On 16/04/2025 15:14, Anthony D'Atri wrote: > Curious, are your systems Dells? If so you might see some improvement from running DSU to update all the firmware. It might also be illuminating to run `racadm lclog view` > >> On Apr 16, 2025, at 8:32 AM, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: >> >> Hi, >> >> Since the latest Reef update I have the problem that some of my hosts suddenly go into a state where all CPUs are stuck in kernel mode causing all daemons on that host to become unresponsive. When I connect to the IPMI console, I see a lot of messages like: >> >> watchdog: BUG: soft lockup - CPU#8 stuck for 47s! [cron:868840] >> >> (it's basically a list of all processes running on the machine). >> >> Usually, this resolves itself after several minutes, but sometimes I have to hardreset the host. When this happens, all daemons are marked as down and I cannot interact with the host at all. I don't know what causes this but, I think it happens primarily on the hosts where my MDS run and it seems to be triggered by events such as cluster rebalances, MDS restarts, or just randomly. >> >> I found a few reports about similar issues on the bug tracker and mailing list, but they are all very unspecific, unanswered, or more than 6 years old. >> >> Is there any way I can debug this? I upgraded to Squid already, but that didn't solve the problem. I also had massive issues with this during the upgrade. Particularly at the end when the MDS were upgraded, I had constant struggles with it. I had to set the noout flag and then literally sit next to it to resume the upgrade every few minutes until it finally went through, because random MDS hosts went intermittently dark all the time. >> >> All hosts run Ubuntu 22.04 with kernel 6.8.0. >> >> Any ideas? Thanks! >> Janek >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 13

Adding OSD nodes
by Sinan Polat 21 Jun '25

21 Jun '25

Hello, I am currently managing a Ceph cluster that consists of 3 racks, each with 4 OSD nodes. Each node contains 24 OSDs. I plan to add three new nodes, one to each rack, to help alleviate the high OSD utilization. The current highest OSD utilization is 85%. I am concerned about the possibility of any OSD reaching the osd_full_ratio threshold during the rebalancing process. This would cause the cluster to enter a read-only state, which I want to avoid at all costs. I am planning to execute the following commands: ceph orch host add new-node-1 ceph orch host add new-node-2 ceph orch host add new-node-3 ceph osd crush move new-node-1 rack=rack-1 ceph osd crush move new-node-2 rack=rack-2 ceph osd crush move new-node-3 rack=rack-3 ceph config set osd osd_max_backfills 1 ceph config set osd osd_recovery_max_active 1 ceph config set osd osd_recovery_sleep 0.1 ceph orch apply osd --all-available-devices Before proceeding, I would like to ask if the above steps are safe to execute in a cluster with such high utilization. My main concern is whether the rebalancing could cause any OSD to exceed the osd_full_ratio and result in unexpected failures. Any insights or advice on how to safely add these nodes without impacting cluster stability would be greatly appreciated. Thanks!

7 7

4k IOPS: miserable performance in All-SSD cluster
by Martin Gerhard Loschwitz 18 Jun '25

18 Jun '25

Folks, I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! Best regards Martin

8 13

Repo name bug?
by Alex 15 May '25

15 May '25

Good morning everyone. Does the preflight playbook have a bug? https://github.com/ceph/cephadm-ansible/blob/devel/cephadm-preflight.yml Line 82: paths: "{{ ['noarch', '$basearch'] if ceph_origin == 'community' else ['$basearch'] }}" The yum repo file then gets named ceph_stable_$basearch. Shouldn't it be basearch without the $ ? - Alex

3 10

write latency increase after upgrade from pacific to quincy
by Nima AbolhassanBeigi 15 May '25

15 May '25

Hi dear ceph community We have encountered an issue with our ceph cluster after upgrading from v16.2.13 to v17.2.7. The issue is that the write latency on OSDs has increased significantly and doesn't seem to plummet back down. The average write latency has almost doubled, and this has happened since we upgraded the OSDs. If anybody could help figure this out. Kind regards Nima

2 4

failing to enable disk failure prediction
by Jens Galsgaard 10 May '25

10 May '25

Hello, I’ve a cluster built with cephadm running on Debian 12/Bookworm. Ceph 18.2.5. I want to enable disk failure prediction and run this command: ceph mgr module enable diskprediction_local Then the cluster goes into ERROR state and the logs shows: 2025-04-14T08:55:37.073118+0200 mgr.host01.eqvsde [ERR] Unhandled exception from module 'diskprediction_local' while running on mgr.host01.eqvsde: No module named 'sklearn.svm.classes' 2025-04-14T08:55:38.388289+0200 mon.host03 [ERR] Health check failed: Module 'diskprediction_local' has failed: No module named 'sklearn.svm.classes' (MGR_MODULE_ERROR) How to add sklearn to the container as it is obviously missing? Venlig hilsen - Mit freundlichen Grüßen - Kind Regards, Jens Galsgaard Gitservice.dk v/ Jens Galsgaard Sparresmindevej 7 4780 Stege Danmark Tlf: +45 46551100 Mob: +45 28864340 CVR/VAT: DK30128540

2 5

Update of MDS (non-cephadm cluster)
by Massimo Sgaravatto 07 May '25

07 May '25

I have done the update of the ceph software several times, but this is the first time I have to do an update (from quincy to reef) considering also ceph-fs I am not using cephadm I have 3 MDSs (2 active and one in standby) I'm looking at point 5 of: https://ceph.io/en/news/blog/2023/v18-2-0-reef-released/ If I have got it right It says to temporarily set max_mds to 1 and when you have only one active: 1 - turn off the MDS service on the 2 inactive MDS nodes 2- update and restart MDS on the active instance 3- restart (after updating the software: this is not written in the doc but it seems obvious to me) MDS on the other 2 instances 4 put max_mds back to 2 Did I understand correctly? But doesn't this mean to have a short period of time when the cluster is in error state (in 2, during the restart) and therefore with possible problems on the cephfs clients ? Thanks, Massimo

9 24

Grafana vulnerability - cephadm deployment
by Michael Worsham 06 May '25

06 May '25

With cephadm reef (v18.2.4), is there a way to upgrade the Grafana that is incorporated into the docker build? Currently it's showing Grafana v9.4.7 (quay.io/ceph/ceph-grafana:9.4.7) and that is actually an open vulnerability per CVE-2023-1387 (Grafana 9.1.0 < 9.2.17, 9.3.x < 9.3.13, 9.4.x < 9.4.9 Information Disclosure Vulnerability). Grafana shows that it's fixed with version 9.4.9 and above (https://grafana.com/security/security-advisories/cve-2023-1387/) -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

7 14

Heads up -- 18.2.6 regression
by Dan van der Ster 06 May '25

06 May '25

Hi all, Just a quick heads up -- 18.2.6 has a bluefs regression [1] which was not present in 18.2.4. Please avoid upgrading until further notice. Regards, Dan [1] Introduced in https://tracker.ceph.com/issues/65356 and fixed in https://tracker.ceph.com/issues/69764 -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO https://clyso.com | dan.vanderster(a)clyso.com

9 18

Doubled numbers of PGs from 8192 to 16384 - backfill bottlenecked
by Torkil Svensgaard 01 May '25

01 May '25

Hi 19.2.2 with WPQ, osd-max-backfills and osd-recovery-max-active are set to 1. In order to get our PG sizes better aligned we doubled the number of PGs on the pool with the largest PG size. The pool is HDD with DB/WAL on SATA SSD and HDD sizes between 2TB and 20TB and PG size was ~140GB before the doubling. All went well but looking at this: " cluster: id: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-flash1,ceph-flash2,ceph-flash3,ceph-e3s2,ceph-e3s3 (age 2d) mgr: ceph-flash2.utlhuz(active, since 4d), standbys: ceph-flash3.ciudre, ceph-flash1.erhakb mds: 1/1 daemons up, 2 standby osd: 576 osds: 576 up (since 2h), 576 in (since 3d); 8767 remapped pgs data: volumes: 1/1 healthy pools: 18 pools, 25249 pgs objects: 683.85M objects, 1.6 PiB usage: 2.7 PiB used, 1.9 PiB / 4.5 PiB avail pgs: 842769842/3951610673 objects misplaced (21.327%) 16481 active+clean 8762 active+remapped+backfill_wait 6 active+remapped+backfilling io: client: 374 MiB/s rd, 14 MiB/s wr, 2.86k op/s rd, 410 op/s wr recovery: 153 MiB/s, 38 objects/s " The balancer was running and seemingly making very small changes: " [root@lazy ~]# ceph balancer status { "active": true, "last_optimize_duration": "0:00:01.012679", "last_optimize_started": "Mon Apr 28 10:01:24 2025", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Optimization plan created successfully", "plans": [] } " I tried the pgremapper and that gave a nice throughput in the beginning but now it seems to be about back to scratch: " cluster: id: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-flash1,ceph-flash2,ceph-flash3,ceph-e3s2,ceph-e3s3 (age 7h) mgr: ceph-flash2.utlhuz(active, since 5d), standbys: ceph-flash3.ciudre, ceph-flash1.erhakb mds: 1/1 daemons up, 2 standby osd: 577 osds: 577 up (since 5h), 577 in (since 31h); 7392 remapped pgs data: volumes: 1/1 healthy pools: 18 pools, 25249 pgs objects: 683.96M objects, 1.6 PiB usage: 2.7 PiB used, 1.9 PiB / 4.5 PiB avail pgs: 459889823/3952312686 objects misplaced (11.636%) 17857 active+clean 7384 active+remapped+backfill_wait 8 active+remapped+backfilling io: client: 70 MiB/s rd, 15 MiB/s wr, 689 op/s rd, 378 op/s wr recovery: 294 MiB/s, 75 objects/s " Balancer is still running and making small changes. This is going to take a while, any tips on how to escape the apparent bottleneck? Is having many PGs misplaced actually counter productive, so it would be better to throttle via target_max_misplaced_ratio? I was thinking it was better to let the balancer balance all it could, as that would make all the moves available and decrease the risk of bottlenecking. Thanks. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

6 13

Understanding CRUSH rule changes to existing pools
by Conrad Hoffmann 30 Apr '25

30 Apr '25

Hello, I am trying to get a better understanding of how Ceph handles changes to CRUSH rules, such as changing the failure domain. I performed this (maybe somewhat academic, sorry) excercise, and would love to verify my conclusions (or get a good explanation of my observations): Starting point: - 2 Nodes - 6 OSDs each - osd_crush_chooseleaf_type = 0 (OSD) - pool size = 3 - pool min size = 2 As expected, the default CRUSH rule would distribute each PG across three random OSDs, without regard for the host they are on. I then added the following CRUSH rule (initially unused): rule replicated_three_over_two { id 1 type replicated step take default step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } Tests with `crushtool` gave me confidence that it worked as I expected: unlike the default rule, this one always made sure at least one copy was on a different host. When I applied this to rule to an existing pool, for a moment I got a HEALTH_WARN "Reduced data availability: 6 pgs peering" and the PGs being shown as inactive (cluster then proceeded to heal itself). This sort of surprised me. I have read threads like [1], where it seems even larger remappings go down without a hitch. I changed back to the default rule, this time with `osd set norebalance` set. But the outcome was the same (brief reduced availability, then healed). Having read more about peering, I now assume that a very brief interruption due to peering is likely unavoidable? But, more importantly, the length of the interruption probably does not scale with the size of the pool (I don't have much data in there right now). So changing the rule for a big pool would still mean only a brief interruption, is that correct? Trying to wrap my head around what exactly the lower-level implications of changing the CRUSH rule could be, I remembered some forum thread I had read, but initially dismissed as somewhat incoherent. Someone claimed that one should set the pool's min_size to 1 when changing rules. For science, I gave this a try. Interestingly enough, the warning about reduced data availability didn't happen (even though there were again inactive/peering PGs). I arrived at the following mental model of what happens on a CRUSH rule update: the lead OSD remains steady, but the placement of the two replicas is re-calculated. In case they need to be moved, peering needs to happen, briefly blocking. But, in case of min_size = 1, the lead OSD, I don't know, says YOLO and accepts writes without having finished peering? If so, why would the PG be considered inactive, though? And, out of curiosity: besides the obvious possibility of corrupting your only copy of an object, are there any implications to setting min_size to 1 before changing a rule? Like, could a write in the right moment cause the peering process to permanently fail or something? [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/UUXT4EBPZVO… Thanks in advance for any enlightenment :) Conrad

3 2

18.2.2: Upgrade not starting (ceph orch upgrade)
by Michel Jouvin 30 Apr '25

30 Apr '25

Hi, I'm trying to upgrade a (cephadm) cluster from 18.2.2 to 18.2.6, using 'ceph orch upgrade'. When I enter the command 'ceph orch upgrade start --ceph-version 18.2.6', I receive a message saying that the upgrade has been initiated, with a similar message in the logs but nothing happens after this. 'ceph orch upgrade status' says: ------- [root@ijc-mon1 ~]# ceph orch upgrade status { "target_image": "quay.io/ceph/ceph:v18.2.6", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false } ------- The first time I entered the command, the cluster status was HEALTH_WARN because of 2 stray daemons (caused by destroyed OSDs, rm --replace). I set mgr/cephadm/warn_on_stray_daemons to false to ignore these 2 daemons, the cluster is now HEALTH_OK but it doesn't help. Following a Red Hat KB entry, I tried to failover the mgr, stopped an restarted the upgrade but without any improvement. I have not seen anything in the logs, except that there is an INF entry every 10s about the destroyed OSD saying: ------ 2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14079 : cephadm [INF] osd.253 now down 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14081 : cephadm [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14082 : cephadm [INF] Zapping devices for osd.253 on dig-osd4 ----- The message seems to be only for one of the 2 destroyed OSDs since I restarted the mgr. May this be the cause for the stucked upgrade? What can I do for fixing this? Thanks in advance for any hint. Best regards, Michel

4 22

manual squid update procedure
by Christopher Durham 30 Apr '25

30 Apr '25

Hi, The non-cephadm update procedure to update reef to squid (for rpm-based clusters) here: https://docs.ceph.com/en/latest/releases/squid/#upgrading-non-cephadm-clust… suggests that your monitors/mds/radosgw/osds are all on separate servers. While perhaps theyshould be that is not possible at current. If for example, I have a mon and an mds on a single server, I really don't want to updatejust a mon as the mds and mon may use common libraries. As such, I am thinking that to do an update in this scenario I do the following set noout set max_mds to for my filesystem, and note the remaining active mds disable standby_replay Then, for each of the servers running an mds (but not the remaining active mds): 1. stop all ceph daemons (mon,mds,radosgw,osd) and do the update, reboot and/or restart the mon and mds 2. Proceed to all the other mds servers except the last one that is on reef3. Finish all other remaining mon updates as in #1 4. Then when the last server with a reef mds is left stop it and let a standby squid mds take over and then do the update and either restart all the daemons or reboot 5. if there are other osd-only nodes, update them and restart the osds Then, afterwards, allow standby replay, reset max_mds to what it was before, unset noout, require squid for osds. Does this make sense? Am I forgetting something? I know setting max_mds 1 is important but want to be sure I have not forgotten anything. Thanks. -Chris

2 1

Memory usage
by Dominique Ramaekers 29 Apr '25

29 Apr '25

Hi, Housekeeping... I was cleaning up my snapshots and was flattening clones... Suddenly I ran out of memory on my nodes! 4 node cluster with each 10 ssd osd's with total storage size 25TiB. Each node has about 45 GiB of free (available) memory in normal operation. After flattening several images and removing dozens of snapshots, the free memory probably was already lower than the usual 45GiB. I was flattening an image of 75GiB and 2 out of 4 nodes ran out of memory. One node even automatically killed processes randomly to free up memory... After regaining control over the system, I put that node in maintenance and rebooted this node. After reboot the free mem was like 70Gib. Over night all of the nodes were back on the usual 45GiB. Today I checked the free mem of each node: 45GiB free. So I flattened again an image of 75GiB. And yes, the free mem dropped from 45GiB to 5GiB really fast! Is there a way to avoid this behavior of the cluster? Greetings, Dominique.

5 5

v18.2.6 Reef released
by Yuri Weinstein 29 Apr '25

29 Apr '25

We're happy to announce the sixth backport (hotfix) release in the Reef series. https://ceph.io/en/news/blog/2025/v18-2-6-reef-released/ Notable Changes --------------- * ceph-volume: A bug related to cryptsetup version handling has been fixed. Related tracker: https://tracker.ceph.com/issues/66393 * RADOS: A bug related to IPv6 support is now fixed. Related tracker: https://tracker.ceph.com/issues/67517 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.2.6.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: ff498e17d264a1a4d588c361cbce9cc65afa2327

3 2

Re: write latency increase after upgrade from pacific to quincy
by Nima AbolhassanBeigi 29 Apr '25

29 Apr '25

Hi Enrico We were aware of the change on the OSD scheduler before upgrading and changed it back to wpq. We put off the change of this option for another time. However, to make sure this wasn't the root cause of the problem, we set the norecover flag on the cluster, and I suspect that if the recovery process were the reason behind this issue, it would have been mitigated. I think the rocksdb compaction_on_deletion option is what caused the problem, but as, for the time being, we can't put our theory to the test, I'm not quite sure it is what caused it. The reason we suspect so is because I checked the OSD logs and used a Python script that parses the compaction logs and gives useful information about all the compaction operations that happened on the OSD in that log file. I compared the extracted data from logs between two OSDs, one had the option set to true and the other set to false. The number of compactions on the second OSD was almost half of what it was on the first one, and as we had the experience of rocksdb compactions affecting the latency of our OSDs, especially the SSD ones, we came to this conclusion. The compaction data ( https://docs.google.com/document/d/1qXDAHvJnfnOPN4ZevdGbgHmsMse3UpL0gZfSvVi… ) *Do you think this is possible?* And to answer your question about the kind of workload, it is quite diverse. We serve both RBD and S3 clients. Our clients have different workloads, and I cannot describe their behaviour in a certain pattern. This might be a bad idea, and we probably should break the clusters down into several more specific-purpose clusters, but that's a problem for another day. If you have any specific questions in mind that I should answer, *don't hesitate to ask.* And by the increase in the number of compactions on our OSDs, after the option was set to true, we could say that the workload could have high delete requests, which I think is what causes the high number of tombstones. Regards, Nima On Tue, Apr 29, 2025 at 3:40 PM Nima AbolhassanBeigi < nima.abolhassanbeigi(a)gmail.com> wrote: > Hi Enrico > We were aware of the change on the OSD scheduler before upgrading and > changed it back to wpq. We put off the change of this option for another > time. However, to make sure this wasn't the root cause of the problem, we > set the norecover flag on the cluster, and I suspect that if the recovery > process were the reason behind this issue, it would have been mitigated. > > I think the rocksdb compaction_on_deletion option is what caused the > problem, but as, for the time being, we can't put our theory to the test, > I'm not quite sure it is what caused it. > The reason we suspect so is because I checked the OSD logs and used a > Python script that parses the compaction logs and gives useful information > about all the compaction operations that happened on the OSD in that log > file. > I compared the extracted data from logs between two OSDs, one had the > option set to true and the other set to false. The number of compactions on > the second OSD was almost half of what it was on the first one, and as we > had the experience of rocksdb compactions affecting the latency of our > OSDs, especially the SSD ones, we came to this conclusion. > The compaction data ( > https://docs.google.com/document/d/1qXDAHvJnfnOPN4ZevdGbgHmsMse3UpL0gZfSvVi… > ) > *Do you think this is possible?* > > And to answer your question about the kind of workload, it is quite > diverse. We serve both RBD and S3 clients. Our clients have different > workloads, and I cannot describe their behaviour in a certain pattern. This > might be a bad idea, and we probably should break the clusters down into > several more specific-purpose clusters, but that's a problem for another > day. > If you have any specific questions in mind that I should answer, *don't > hesitate to ask.* And by the increase in the number of compactions on our > OSDs, after the option was set to true, we could say that the workload > could have high delete requests, which I think is what causes the high > number of tombstones. > > Regards, > Nima > > On Tue, Apr 29, 2025 at 2:33 PM Enrico Bocchi <enrico.bocchi(a)cern.ch> > wrote: > >> Hello Nima, >> >> Unsure if you have found the root cause of the problem in the meantime> >> From the top of my head, if any useful: >> - Quincy changes the default scheduler from wpq to mclock >> - The default number of scrubs on each osd is increased from 1 to 3 >> - There's a new rocksdb compact_on_deletion option that triggers >> compaction more frequently according to number of tombstones over a >> sliding window. >> >> The latter, however, is very workload dependent. What type of workload >> does the cluster serve? >> >> Cheers, >> Enrico >> >> >> On 4/15/25 12:13, Nima AbolhassanBeigi wrote: >> > Hello again >> > We have done some investigating. Since our first message did not have >> any >> > information or context, let me add some new information. >> > >> > I looked through our dashboards and the write latency panels in the OSD >> > Overview dashboard. >> > The latency for write operations had increased significantly, but in >> > distinction, the physical write more or less stayed the same; however, >> the >> > write process operations and write prepare operations have increased >> > dramatically. >> > >> > In the end, the RocksDB tunings helped our situation and decreased the >> > latency more or less to the pre-upgrade amount. >> > We changed the default parameters to the following: >> > >> compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 >> > This seems to have done the trick, however, we are still worried that we >> > haven't found the root cause of this problem and only mitigated the >> > symptoms. >> > >> > Another probably important information is that the bluefs allocator for >> our >> > OSDs is not the default hybrid and is set to the bitmap option. In >> > addition, the bluestore allocation block score of these OSDs is, for >> most >> > of them, almost 0.9, which I know is terrible, but what I don't >> understand >> > is why this problem occurred post-upgrade. Could it be the terrible >> block >> > scores? >> > >> > Another interesting change was the memory usage of each OSD >> post-upgrade, >> > which dropped nearly in half, but the Node's memory cache/buffer usage >> > increased. >> > >> > I've put the Grafana panels screenshots in this Google Doc >> > < >> https://docs.google.com/document/d/1DSf4MJoze_BTSWAJWoYyQetA33P6fwpMavYXPdA… >> > >> > . >> > We started our upgrade process on March 6 and then again on March 7. We >> > tuned the RocksDB parameters on March 9th, as the screenshots show, >> which >> > drops the latency. >> > >> > We are still looking for the root cause, and I could use all the help I >> can >> > get to try to find it, and it is much appreciated. >> > >> > Regards >> > Nima >> > >> > On Mon, Apr 7, 2025 at 3:27 PM Nima AbolhassanBeigi < >> > nima.abolhassanbeigi(a)gmail.com> wrote: >> > >> >> Hi dear ceph community >> >> >> >> We have encountered an issue with our ceph cluster after upgrading from >> >> v16.2.13 to v17.2.7. >> >> The issue is that the write latency on OSDs has increased significantly >> >> and doesn't seem to plummet back down. >> >> The average write latency has almost doubled, and this has happened >> since >> >> we upgraded the OSDs. >> >> >> >> If anybody could help figure this out. >> >> >> >> Kind regards >> >> Nima >> >> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> -- >> Enrico Bocchi >> CERN European Laboratory for Particle Physics >> IT - Storage & Data Management - General Storage Services >> Mailbox: G20500 - Office: 31-2-010 >> 1211 Genève 23 >> Switzerland >> >>

1 0

crush rule: is it valid to use a non root element for the root parameter?
by Michel Jouvin 29 Apr '25

29 Apr '25

Hi, We have use case where we had like to restrict some pools to a subset of the OSDs located in a particular section of the crush map hierarchy (OSDs backed up by UPS/Diesel). We tried to define for these (replica 3) pools a specific crush rule with the root paramater defined to a specific row (which contains 3 OSD servers with #10 OSD each). At the beginning it worked but after some time (probably after doing a reweight on the OSDs in this row to reduce the number of PGs from other pools), a few PGs are active+clean+remapped and 1 is undersized. 'ceph osd pg dump|grep remapped' gives an output similar to the following one for each remapped PG: 20.1ae 648 0 0 648 0 374416846 0 0 438 1404 438 active+clean+remapped 2025-04-16T07:19:40.507778+0000 43117'1018433 48443:1131738 [70,58] 70 [70,58,45] 70 43117'1018433 2025-04-16T07:19:40.507443+0000 43117'1018433 2025-04-16T07:19:40.507443+0000 0 15 periodic scrub scheduled @ 2025-04-17T19:18:23.470846+0000 648 0 We can see that we currently have 3 replica but that Ceph would like to move to 2... (the undersized PG has currently only 2 replica for an unknown reason, probably the same). Is it wrong trying to do what we did, i.e. using a row for the crush rule root parameter? If not, where could we find more information about the cause? Thanks in advance for any help. Best regards, Michel --------------------- Crush rule used ----------------- { "rule_id": 2, "rule_name": "ha-replicated_ruleset", "type": 1, "steps": [ { "op": "take", "item": -22, "item_name": "row-01~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ------------------- Beginning of the CRUSH tree ------------------- ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 843.57141 root default -19 843.57141 datacenter bat.206 -21 283.81818 row row-01 -15 87.32867 host cephdevel-76079 1 hdd 7.27739 osd.1 up 0.50000 1.00000 2 hdd 7.27739 osd.2 up 0.50000 1.00000 14 hdd 7.27739 osd.14 up 0.50000 1.00000 39 hdd 7.27739 osd.39 up 0.50000 1.00000 40 hdd 7.27739 osd.40 up 0.50000 1.00000 41 hdd 7.27739 osd.41 up 0.50000 1.00000 42 hdd 7.27739 osd.42 up 0.50000 1.00000 43 hdd 7.27739 osd.43 up 0.50000 1.00000 44 hdd 7.27739 osd.44 up 0.50000 1.00000 45 hdd 7.27739 osd.45 up 0.50000 1.00000 46 hdd 7.27739 osd.46 up 0.50000 1.00000 47 hdd 7.27739 osd.47 up 0.50000 1.00000 -3 94.60606 host cephdevel-76154 49 hdd 7.27739 osd.49 up 0.50000 1.00000 50 hdd 7.27739 osd.50 up 0.50000 1.00000 51 hdd 7.27739 osd.51 up 0.50000 1.00000 66 hdd 7.27739 osd.66 up 0.50000 1.00000 67 hdd 7.27739 osd.67 up 0.50000 1.00000 68 hdd 7.27739 osd.68 up 0.50000 1.00000 69 hdd 7.27739 osd.69 up 0.50000 1.00000 70 hdd 7.27739 osd.70 up 0.50000 1.00000 71 hdd 7.27739 osd.71 up 0.50000 1.00000 72 hdd 7.27739 osd.72 up 0.50000 1.00000 73 hdd 7.27739 osd.73 up 0.50000 1.00000 74 hdd 7.27739 osd.74 up 0.50000 1.00000 75 hdd 7.27739 osd.75 up 0.50000 1.00000 -4 101.88345 host cephdevel-76204 48 hdd 7.27739 osd.48 up 0.50000 1.00000 52 hdd 7.27739 osd.52 up 0.50000 1.00000 53 hdd 7.27739 osd.53 up 0.50000 1.00000 54 hdd 7.27739 osd.54 up 0.50000 1.00000 56 hdd 7.27739 osd.56 up 0.50000 1.00000 57 hdd 7.27739 osd.57 up 0.50000 1.00000 58 hdd 7.27739 osd.58 up 0.50000 1.00000 59 hdd 7.27739 osd.59 up 0.50000 1.00000 60 hdd 7.27739 osd.60 up 0.50000 1.00000 61 hdd 7.27739 osd.61 up 0.50000 1.00000 62 hdd 7.27739 osd.62 up 0.50000 1.00000 63 hdd 7.27739 osd.63 up 0.50000 1.00000 64 hdd 7.27739 osd.64 up 0.50000 1.00000 65 hdd 7.27739 osd.65 up 0.50000 1.00000 -23 203.16110 row row-02 -13 87.32867 host cephdevel-76213 27 hdd 7.27739 osd.27 up 1.00000 1.00000 28 hdd 7.27739 osd.28 up 1.00000 1.00000 29 hdd 7.27739 osd.29 up 1.00000 1.00000 30 hdd 7.27739 osd.30 up 1.00000 1.00000 31 hdd 7.27739 osd.31 up 1.00000 1.00000 32 hdd 7.27739 osd.32 up 1.00000 1.00000 33 hdd 7.27739 osd.33 up 1.00000 1.00000 34 hdd 7.27739 osd.34 up 1.00000 1.00000 35 hdd 7.27739 osd.35 up 1.00000 1.00000 36 hdd 7.27739 osd.36 up 1.00000 1.00000 37 hdd 7.27739 osd.37 up 1.00000 1.00000 38 hdd 7.27739 osd.38 up 1.00000 1.00000 ......

3 5

18.2.6 upgrade OSDs fail to mount
by Marco Pizzolo 29 Apr '25

29 Apr '25

Hello Everyone, I'm upgrading from 18.2.4 to 18.2.6, and I have a 4-node cluster with 8 NVMe's per node. Each NVMe is split into 2 OSDs. The upgrade went through the mgr, mon, crash and began upgrading OSDs. The OSDs it was upgrading were not coming back online. I tried rebooting, and no luck. journalctl -xe shows the following: ░░ The unit docker-02cb79ef9a657cdaa26b781966aa6d2f1d5e54cdc9efa6c5ff1f0e98c3a866e4.scope has successfully entered the 'dead' state. Apr 29 06:24:09 prdhcistonode01 dockerd[2967]: time="2025-04-29T06:24:09.282073583-04:00" level=info msg="ignoring event" container=76c56ddd668015de0022bfa2527060e64a9513> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.282129114-04:00" level=info msg="shim disconnected" id=76c56ddd668015de0022bfa2527060e64a95137> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.282219664-04:00" level=warning msg="cleaning up after shim disconnected" id=76c56ddd668015de00> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.282242484-04:00" level=info msg="cleaning up dead shim" Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_p> Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 0 osd.3:0.OSDShard using op scheduler mclock_scheduler, cutoff=196 Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 1 bdev(0x56046b4c8000 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/cep> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.292047607-04:00" level=warning msg="cleanup warnings time=\"2025-04-29T06:24:09-04:00\" level=> Apr 29 06:24:09 prdhcistonode01 dockerd[2967]: time="2025-04-29T06:24:09.292163618-04:00" level=info msg="ignoring event" container=02cb79ef9a657cdaa26b781966aa6d2f1d5e54> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.292216428-04:00" level=info msg="shim disconnected" id=02cb79ef9a657cdaa26b781966aa6d2f1d5e54c> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.292277279-04:00" level=warning msg="cleaning up after shim disconnected" id=02cb79ef9a657cdaa2> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.292291949-04:00" level=info msg="cleaning up dead shim" Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 1 bdev(0x56046b4c8000 /var/lib/ceph/osd/ceph-3/block) open size 640122932428> Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 -1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes bluestore_cache_meta_> Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 1 bdev(0x56046b4c8000 /var/lib/ceph/osd/ceph-3/block) close Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.303385220-04:00" level=warning msg="cleanup warnings time=\"2025-04-29T06:24:09-04:00\" level=> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_p> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 0 osd.0:0.OSDShard using op scheduler mclock_scheduler, cutoff=196 Apr 29 06:24:09 prdhcistonode01 bash[23144]: debug 2025-04-29T10:24:09.307+0000 7f12f08c5740 -1 osd.15 0 OSD:init: unable to mount object store Apr 29 06:24:09 prdhcistonode01 bash[23144]: debug 2025-04-29T10:24:09.307+0000 7f12f08c5740 -1 ** ERROR: osd init failed: (22) Invalid argument Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 1 bdev(0x55d5e45f0000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/cep> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 1 bdev(0x55d5e45f0000 /var/lib/ceph/osd/ceph-0/block) open size 640122932428> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 -1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes bluestore_cache_meta_> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 1 bdev(0x55d5e45f0000 /var/lib/ceph/osd/ceph-0/block) close Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_p> Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 0 osd.8:0.OSDShard using op scheduler mclock_scheduler, cutoff=196 Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 1 bdev(0x555f40688000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/cep> Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 1 bdev(0x555f40688000 /var/lib/ceph/osd/ceph-8/block) open size 640122932428> Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 -1 bluestore(/var/lib/ceph/osd/ceph-8) _set_cache_sizes bluestore_cache_meta_> Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 1 bdev(0x555f40688000 /var/lib/ceph/osd/ceph-8/block) close Apr 29 06:24:09 prdhcistonode01 systemd[1]: ceph-fbc38f5c-a3a6-11ea-805c-3b954db9ce7a(a)osd.12.service: Main process exited, code=exited, status=1/FAILURE Any help you can offer would be greatly appreciated. This is running in docker: Client: Docker Engine - Community Version: 24.0.7 API version: 1.43 Go version: go1.20.10 Git commit: afdd53b Built: Thu Oct 26 09:08:01 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.7 API version: 1.43 (minimum version 1.12) Go version: go1.20.10 Git commit: 311b9ff Built: Thu Oct 26 09:08:01 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.25 GitCommit: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f runc: Version: 1.1.10 GitCommit: v1.1.10-0-g18a0cb0 docker-init: Version: 0.19.0 GitCommit: de40ad0 Thanks, Marco

2 5

While update minor version of Ceph cluster with ceph-ansible
by dhivagar selvam 29 Apr '25

29 Apr '25

Hi.. We are facing the below error while update minor version from 18.2.4 to 18.2.6 using rolling-update playbook. TASK [ceph-facts : Set_fact _monitor_addresses - ipv4] ************************************************************************************************************************************************************ task path: /root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml:2 fatal: [ceph-mont1]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mont2]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mont3]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mont4]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-osdt1]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-osdt2]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mdst1]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mdst2]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here

1 0

Subject: Ceph MGR k8sevents module - community feedback wanted
by John Mulligan 28 Apr '25

28 Apr '25

Some Background: https://tracker.ceph.com/issues/64213 In particular, the python cryptography module uses PyO3 and on some versions of python, Ceph, and a new enough cryptography module the mgr may fail to start certain modules with an error like: PyO3 modules may only be initialized once per interpreter process A few of us have been looking into the problem and have come up with a workaround for the Ceph dashboard. The dashboard makes occasional use of cryptography and other libraries that also use cryptography. However, this use is fairly easy to encapsulate and not called to frequently. The cephadm module uses a ssh library that more heavily uses cryptography and is the module that will have direct access. That leaves two other modules that make use of cryptography. The 'rook' module and 'k8sevents' module. Speaking with some of the other Ceph developers we think that the 'rook' module is being used actively on rook deployments to enable the dashboard when orchestrated with rook. However, we are unsure if anyone is using the 'k8sevents' module at all. --- To help us better understand the status of the k8sevents module we'd like to ask the community if anyone is successfully using the k8sevents module. This will help us understand if this module is something we might be able to ultimately remove. Note that only one module of either k8sevents and rook could run on systems with newer cryptography libraries - so if we kept both we'd have to figure out a way to make the two modules that use kubernetes libraries compatible - and that may be complicated. To summarize: please let us know if you are using the k8sevents module, that it works for you and accomplishes something useful. Feel free to also let us know if you're using rook and the 'rook' module with-or-without the k8smodule. Thanks!

1 0

Re: ceph mon out of quorom and cephfs degraded
by Eugen Block 28 Apr '25

28 Apr '25

Was there any issue in your network or anything? Something that would explain the MDS crash? I would recommend to scan syslog, dmesg etc. for anything suspicious. If you don't find anything, you might need to go through https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ to get your CephFS back up. Don't forget to create a backup of the journals. Read the instructions carefully, you might not need to do all the mentioned steps. I would start with (for both journals where applicable): cephfs-journal-tool journal export backup.bin cephfs-journal-tool event recover_dentries summary cephfs-journal-tool [--rank=<fs_name>:{mds-rank|all}] journal reset --yes-i-really-really-mean-it cephfs-table-tool all reset session But again, read the instructions carefully. This procedure is usually the last resort, I would first try to find out what happened causing this situation. Zitat von Amudhan P <amudhan83(a)gmail.com>: > I don't think I have a memory issue. > Sorry, for sending log as file Pastebin is not working due to size > limitation. > > Crash info > >> ceph crash info >> 2025-04-17T08:50:35.931485Z_0109f173-206c-471e-afac-c0d3e0aa2785 >> { >> "backtrace": [ >> "/lib64/libpthread.so.0(+0x12b20) [0x7fbd42bbdb20]", >> "/usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7fbd4c81da00]" >> ], >> "ceph_version": "16.2.6", >> "crash_id": >> "2025-04-17T08:50:35.931485Z_0109f173-206c-471e-afac-c0d3e0aa2785", >> "entity_name": "mds.mummasstrg.strg-node1.gchapr", >> "os_id": "centos", >> "os_name": "CentOS Linux", >> "os_version": "8", >> "os_version_id": "8", >> "process_name": "ceph-mds", >> "stack_sig": >> "5238fe1be0b82b479b10ee7d17b5ad3182cdc93ede581af63d627a472a4fcf9e", >> "timestamp": "2025-04-17T08:50:35.931485Z", >> "utsname_hostname": "strg-node1", >> "utsname_machine": "x86_64", >> "utsname_release": "5.10.0-8-amd64", >> "utsname_sysname": "Linux", >> "utsname_version": "#1 SMP Debian 5.10.46-5 (2021-09-23)" >> } >> > > On Thu, Apr 17, 2025 at 2:55 PM Eugen Block <eblock(a)nde.ag> wrote: > >> Oh right, I just noticed you had the status in your first message. Is >> there any chance the MDS is out of memory? Can you paste a complete >> (fresh) startup log on some pastebin or so? Also, a complete 'ceph >> crash info <crash>' could be useful. >> >> Zitat von Amudhan P <amudhan83(a)gmail.com>: >> >> > I have run the status and stat command below is the output. >> > >> > ceph -s >> > >> > cluster: >> > id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a >> > health: HEALTH_WARN >> > 2 failed cephadm daemon(s) >> > 1 filesystem is degraded >> > insufficient standby MDS daemons available >> > 7 daemons have recently crashed >> > >> > services: >> > mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 20h) >> > mgr: strg-node2.unyimy(active, since 20h), standbys: >> strg-node1.ivkfid >> > mds: 1/1 daemons up >> > osd: 32 osds: 32 up (since 20h), 32 in (since 10w) >> > >> > data: >> > volumes: 0/1 healthy, 1 recovering >> > pools: 3 pools, 321 pgs >> > objects: 15.49M objects, 54 TiB >> > usage: 109 TiB used, 66 TiB / 175 TiB avail >> > pgs: 317 active+clean >> > 4 active+clean+scrubbing+deep >> > >> > >> > ceph mds stat >> > mumstrg:1/1 {0=mumstrg.strg-node1.gchapr=up:replay(laggy or crashed)} >> > >> > ceph osd lspools >> > 1 device_health_metrics >> > 2 cephfs.mumstrg.meta >> > 3 cephfs.mumstrg.data >> > >> > >> > >> > On Thu, Apr 17, 2025 at 10:33 AM Eugen Block <eblock(a)nde.ag> wrote: >> > >> >> What’s your overall Ceph status? It says data pool 3 not found. >> >> >> >> Zitat von Amudhan P <amudhan83(a)gmail.com>: >> >> >> >> > There are few more logs in MDS. I have highlighted few log lines >> which I >> >> am >> >> > not sure what it is. >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -79> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command dump inode hook 0x560a2c354580 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -78> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command exit hook 0x560a2c354580 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -77> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command respawn hook 0x560a2c354580 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -76> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command heap hook 0x560a2c354580 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -75> >> >> > 2025-04-16T14:43:59.170+0000 7f74b5030700 1 >> >> mds.mumstrg.strg-node3.xhxbwx >> >> > Updating MDS map to version 127517 f >> >> > rom mon.2 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -74> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command cpu_profiler hook 0x560a2c35458 >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -73> >> >> > 2025-04-16T14:43:59.170+0000 7f74b302c700 5 >> >> > mds.beacon.mumstrg.strg-node3.xhxbwx Sending beacon up:boot seq 1 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -72> >> >> > 2025-04-16T14:43:59.170+0000 7f74b302c700 10 monclient: >> _send_mon_message >> >> > to mon.strg-node3 at v2:10.0.103.3:3300/ >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -71> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 1 >> >> mds.mumstrg.strg-node3.xhxbwx >> >> > Updating MDS map to version 127518 f >> >> > rom mon.2 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -70> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -69> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: >> _send_mon_message >> >> > to mon.strg-node3 at v2:10.0.103.3:3300/ >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -68> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue >> >> operator(): >> >> > data pool 3 not found in OSDMap >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -67> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 5 asok(0x560a2c44e000) >> >> > register_command objecter_requests hook 0x560a2c >> >> > 3544c0 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -66> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -65> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: >> _send_mon_message >> >> > to mon.strg-node3 at v2:10.0.103.3:3300/ >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -64> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 log_channel(cluster) >> >> > update_config to_monitors: true to_syslog: false >> >> > syslog_facility: daemon prio: info to_graylog: false graylog_host: >> >> > 127.0.0.1 graylog_port: 12201) >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -63> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue >> >> operator(): >> >> > data pool 3 not found in OSDMap >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -62> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.0 handle_osd_map >> epoch >> >> > 0, 0 new blocklist entries >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -61> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 >> handle_mds_map >> >> i >> >> > am now mds.0.127518 >> >> > >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -60> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 >> handle_mds_map >> >> >> state change up:boot --> up:replay >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -59> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 >> >> >> mds.beacon.mummasstrg.strg-node3.xhxbwx set_want_state: up:boot -> >> >> up:replay >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -58> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 >> replay_start >> >> >> *Apr 16 20:13:59 strg-node3 bash[59114]: debug -57> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 waiting >> for >> >> >> osdmap 45749 (which blocklists prior instance)* >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -56> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: >> >> _send_mon_message >> >> >> to mon.strg-node3 at v2:10.0.103.3:3300/0 >> >> >> *Apr 16 20:13:59 strg-node3 bash[59114]: debug -55> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue >> >> operator(): >> >> >> data pool 3 not found in OSDMap* >> >> >> >> >> > >> >> > >> >> > On Thu, Apr 17, 2025 at 7:06 AM Amudhan P <amudhan83(a)gmail.com> >> wrote: >> >> > >> >> >> Eugen, >> >> >> >> >> >> This is the output for the command >> >> >> cephfs-journal-tool --rank=mumstrg:all --journal=purge_queue journal >> >> >> inspect >> >> >> Overall journal integrity: OK >> >> >> cephfs-journal-tool --rank=mumstrg:all --journal=mdlog journal >> inspect >> >> >> Overall journal integrity: OK >> >> >> >> >> >> On Thu, Apr 17, 2025 at 2:59 AM Eugen Block <eblock(a)nde.ag> wrote: >> >> >> >> >> >>> I think either your mdlog or the purge_queue journal is corrupted: >> >> >>> >> >> >>> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: >> 2: >> >> >>> waiting for purge queue recovered >> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> >> >> >>> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: >> >> get_auth_request >> >> >>> con 0x562856a25400 auth_method 0 >> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> >> >> >>> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal >> >> >>> (Segmentation fault) ** >> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 >> >> >>> thread_name:md_log_replay >> >> >>> >> >> >>> Can you paste the output of this command? >> >> >>> >> >> >>> cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=purge_queue >> >> >>> journal inspect >> >> >>> cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=mdlog >> journal >> >> >>> inspect >> >> >>> >> >> >>> I expect one or more damaged entries. Check this thread for more >> >> details: >> >> >>> >> >> >>> https://www.spinics.net/lists/ceph-users/msg80124.html >> >> >>> >> >> >>> You should try to backup the journal, but in my case that wasn't >> >> >>> possible, so I had no other choice than resetting it. >> >> >>> >> >> >>> Zitat von Amudhan P <amudhan83(a)gmail.com>: >> >> >>> >> >> >>> > Hi, >> >> >>> > >> >> >>> > I am having 2 problems with my Ceph version 16.2.6 >> >> >>> > (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) >> deployed >> >> >>> thru >> >> >>> > cephadm. >> >> >>> > >> >> >>> > First issue :- >> >> >>> > 1 out 3 mon service went out of quorum . >> >> >>> > When restarted service it comes normal but after a few minutes in >> >> ceph >> >> >>> > watch log it reports slow ops and mon goes out of quorum. >> >> >>> > Node where this mon service failed had one weird thing that I >> could >> >> see >> >> >>> 40% >> >> >>> > of wait in the top command. But I don't see any error in dmesg or >> >> >>> anything >> >> >>> > related to drive IO error. >> >> >>> > Below are the logs that were printed in ceph watch command. >> >> >>> > >> >> >>> > 2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] >> MON_DOWN: >> >> 1/3 >> >> >>> > mons down, quorum strg-node2,strg-node3 >> >> >>> > 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] >> >> mon.strg-node1 >> >> >>> > (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down >> >> (out >> >> >>> of >> >> >>> > quorum) >> >> >>> > >> >> >>> > For now this is not appearing again. >> >> >>> > >> >> >>> > >> >> >>> > Second issue Cephfs degraded :- >> >> >>> > I have 2 MDS services running in 2 different nodes. Both are in a >> >> >>> stopped >> >> >>> > state. >> >> >>> > when running Ceph -s command >> >> >>> > >> >> >>> > cluster: >> >> >>> > id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a >> >> >>> > health: HEALTH_WARN >> >> >>> > 2 failed cephadm daemon(s) >> >> >>> > 1 filesystem is degraded >> >> >>> > insufficient standby MDS daemons available >> >> >>> > >> >> >>> > services: >> >> >>> > mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age >> 4h) >> >> >>> > mgr: strg-node2.unyimy(active, since 4h), standbys: >> >> >>> strg-node1.ivkfid >> >> >>> > mds: 1/1 daemons up >> >> >>> > osd: 32 osds: 32 up (since 4h), 32 in (since 10w) >> >> >>> > >> >> >>> > data: >> >> >>> > volumes: 0/1 healthy, 1 recovering >> >> >>> > pools: 3 pools, 321 pgs >> >> >>> > objects: 15.49M objects, 54 TiB >> >> >>> > usage: 109 TiB used, 66 TiB / 175 TiB avail >> >> >>> > pgs: 321 active+clean >> >> >>> > >> >> >>> > Volume shows recovering but there wasn't any progress till now >> even >> >> >>> manual >> >> >>> > start mds service fails again. In Ceph -s command under services >> it >> >> >>> shows >> >> >>> > mds up no any mds service is running. >> >> >>> > >> >> >>> > Below is a log snip from one of the mds service. >> >> >>> > >> >> >>> > >> >> >>> > -25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 >> >> >>> > mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, >> >> ex> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> >> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d0874700 1 >> mds.0.journaler.pq(ro) >> >> >>> probing >> >> >>> > for end of the log >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> >> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a17400 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> >> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a17c00 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > recover start >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > read_head >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for >> >> >>> journal >> >> >>> > 0x200 to recover... >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a25000 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 1 >> mds.0.journaler.pq(ro) >> >> >>> > _finish_probe_end write_pos = 13968309289 (hea> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue >> >> >>> operator(): >> >> >>> > open complete >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 1 >> mds.0.journaler.pq(ro) >> >> >>> > set_writeable >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43cf872700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > _finish_read_head loghead(trim 189741504921> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43cf872700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > probing for end of the log >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> >> >> >>> > 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a25c00 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf872700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > _finish_probe_end write_pos = 1897428915052> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal >> 0x200 >> >> >>> > recovered. >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered >> >> journal >> >> >>> > 0x200 in format 1 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 >> Booting: 1: >> >> >>> > loading/discovering base inodes >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating >> >> system >> >> >>> > inode with ino:0x100 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating >> >> system >> >> >>> > inode with ino:0x1 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a25800 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a5dc00 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> >> >> >>> > 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 >> Booting: 2: >> >> >>> > replaying mds log >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> >> >> >>> > 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 >> Booting: 2: >> >> >>> > waiting for purge queue recovered >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> >> >> >>> > 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a25400 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> >> >> >>> > 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal >> >> >>> > (Segmentation fault) ** >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 >> >> >>> > thread_name:md_log_replay >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 >> >> >>> > (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: 1: >> >> >>> /lib64/libpthread.so.0(+0x12b20) >> >> >>> > [0x7f43dd293b20] >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: 2: >> >> >>> > /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the >> >> executable, >> >> >>> or >> >> >>> > `objdump -rdS <executable>` is needed to interpret this. >> >> >>> > >> >> >>> > >> >> >>> > Not sure what caused the issue. I couldn't find any resources to >> fix >> >> >>> this >> >> >>> > issue. >> >> >>> > Need help from someone to bring the ceph cluster online. >> >> >>> > _______________________________________________ >> >> >>> > ceph-users mailing list -- ceph-users(a)ceph.io >> >> >>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >>> >> >> >>> >> >> >>> _______________________________________________ >> >> >>> ceph-users mailing list -- ceph-users(a)ceph.io >> >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>

2 6

Best Practices and real world experiences for Big Scale S3 Clusters
by Şenol Çolak 28 Apr '25

28 Apr '25

Hello everyone, We’re running a 30+ node cluster (+plan to add more) to provide S3 services to our consumers. Each node has: - 64 Cores, 300+ GB RAM (Monitors have less memory, we have 5 monitors separated from OSD nodes) - 14 NVMe (14TB) or 14 HDD (10TB) disks - Each OSD node also hosts RGW services(pods). - HP/Dell/Lenovo hardware - Around 6 PB Raw capacity. - roughly 3 CPU cores and 8GB RAM are allocated per OSD sizing. - We use Rook to deploy our cluster I know that some of the answers might be long, I also have my own experiences on some of them(which I included), but I would highly appreciate to see the state for Ceph S3 on 04/2025. and here we have several questions we’d love your input on: * Performance Counters - We want to enable performance counters to identify noisy buckets and diagnose issues when they occur. - Does enabling performance counters cause noticeable performance degradation? If so, by how much? - For now, we plan to keep them disabled.(Because we are not sure) * OS Tuning for Ceph Nodes - WriteCache is disabled for NVMe disks. - No multi-socket nodes (no NUMA configuration). - No other workloads running on Ceph nodes. - Plan is to keep the default installation unless strongly advised otherwise. - Is there anything else critical we should tune (kernel parameters, etc.)? * Tracking Over-utilization - How can we detect and identify users overloading the cluster (e.g., excessive object creation/deletion, heavy throughput)? * NVMe Configuration for Ceph - Beyond disabling cache, what are the best practices around discard/trimming operations for NVMe drives? * Buying New NVMe Disks - When we order disks, we request 2 batches per disk group (e.g., 100 disks from Batch1, 100 from Batch2). We are looking for more professional recommendations. Is it a good idea to have different batches or shall we combine two different vendor disk groups? - We always check the rebranding values for NVMe disks and try to group them with rebranding informations. * Tuning Ceph for Large Scale S3 - What default parameters should be adjusted for a large-scale S3 deployment? * Customer Limitations - In theory we don't need any limitations, but we don't want to see non-sharding objects, etc. , If we allow everything our customers will find ways to abuse this. What parameters will protect us from operational difficulties. - Are there standard limitations we should enforce, like maximum object size, maximum number of objects per bucket, etc. * Inventory Management and Netbox Integration - Is it a good practice to integrate Netbox into Ceph operations? - Like, should we register all disks in Netbox for easier daily maintenance, adding automation to the netbox and replacing the disks via inventory changes on netbox ? * Cluster Deployment Best Practices - Ansible, Rook, custom deployments, or manual host installations — what’s best for large clusters, might depend on the team and company structure but would be nice to see opinions? * Why Some Large Deployments Avoid Rook - I mostly see Rook deployments for Openstack HCI deployments only (maybe I am wrong on this) - Is it about operational complexity, or is Rook seen as unsuitable for very large clusters? * Monitoring Metrics - What are the most critical metrics for diagnosing S3 issues? * RocksDB Tuning and Archiving - How should we manage RocksDB in large-scale setups? - Should we periodically split/archive the DB? - How important is RocksDB latency monitoring? * OS-Level Monitoring - Besides standard exporter logs, what else should we monitor (e.g., network latency, OSD-to-OSD throughput)? - we are planning to create a separate exporter than the ceph_exporter to export some specific metrics (mostly disk and NW related) * Design Mistakes to Avoid - What are the common mistakes (e.g., settings that block easy upgrades) that we should be aware of when planning a large-scale S3 service? * Vendor Relations - Is there a resource to check disk types, endurance, wearout timelines, etc. When we order new HW the vendor doesn't provide us any data? - Maybe there is a group of Ceph users holding some internal information about different vendors and their HW, if not I would be happy to start this.. - What's the best approach when dealing with hardware vendors, getting the cheapest or getting the reliable is the right approach? * Scheduled Operations - Are there standard Ansible playbooks or procedures for recurring maintenance tasks? * Unsupported Ceph Versions - Some large deployments stick with Ceph 16.x (unsupported). - Why do they prefer being alone on the old version and backporting features themselves? this seems very dangerous (there has to be a valid reason why?) * Cluster Health Monitoring - Do large deployments rely only on HEALTH_OK? - Or do they combine metrics for a custom health status definition? * Security Best Practices - What's the best approach for securing the cluster? - How should we handle disk encryption, key rotation (frequency/method), and secure disposal? * Diversity of Pools - If our service is focused purely on S3, should we still consider using other pool types (RBD, CephFS)? * Disk Size Diversity - Is it problematic to have OSDs with different disk sizes? * Multisite Clusters - In a multisite setup, if replication is enabled for some pools, is the performance impact localized to those pools, or does it affect the entire cluster? * WAF or Smart Proxy for RGW nodes - Our idea is to have RGW on each of our OSD node, (Monitor and Mgr would be excluded on any operation), what would be the best idea besides having a LDB in front of these RGW services. How are the others designing RGW, is it safe enough to just use it directly? I would highly appreciate your answers/opinions on these topics. I also assume that most of these comments would come from real use cases and experiences. Thanks in advance, _ Senol Colak

2 1

April 28th CSC Meeting Notes
by Laura Flores 28 Apr '25

28 Apr '25

- [Adam King] PyO3 issue (https://tracker.ceph.com/issues/64213) - pecastro/John Mulligan/Ernesto have been working on this - https://github.com/ceph/ceph/pull/61737 - Can be fixed for cephadm and dashboard through that work - still an issue with rook/k8sevents modules - https://pad.ceph.com/p/pyo3-cryptography-imports-k8s - Is anyone actually using these? - Ongoing issue on newer versions that affects the mgr (patch ready to work around this) - Rook issue still WIP; seeking advice on discontinuing some of the affected modules - Yaarit will check on Telemetry and follow up next week - Suggestion to reach out to the user list - [cbodley] follow up on 'ceph api tests' with Ernesto (bump to next week) - Tentacle branch cut last week on Fri. - Can we build packages now? - https://jenkins.ceph.com/job/ceph-dev/58072/ - David can check that the branch is protected - Done - Tentacle milestone created: https://github.com/ceph/ceph/milestone/31 - Patrick needs reviews on https://github.com/ceph/ceph/pull/62089 - Need to check how upgrade tests are doing - v20.3.0 needs tagged on main to allow upgrades from tentacle -> main - Adam Kraitman: upgraded redmine and improved database - A bunch of people no longer get emails from Redmine, the last email is from Apr 24th -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Re: Squid Manager Daemon: balancer crashing orchestrator and dashboard
by Laimis Juzeliūnas 28 Apr '25

28 Apr '25

Hi all, Since there were a few people asking around and I have not seen any users confirming I’ve just wanted to put this fact in writing for anyone in the future: 19.2.2 version fixes all 19.2.0 balancer issues. Everything started working immediately once the mgr modules were upgraded. I’m sure 19.2.1 already does that, but it’s recommended to go to 19.2.2 directly. Huge thanks to everyone involved, thread closed. Best, Laimis J.

1 0

Stretched pool or not ?
by Soeren Malchow 28 Apr '25

28 Apr '25

Hi, i am working on the plan for a 3 datacenter setup using ceph (in proxmox nodes). Each datacenter has 3 physical nodes to start with and 100Gbit switches. I will also have 2 x 100 Gbit/s connectivity between the datacenters (each datacenter to each other). the physical nodes have 2 x 100Gbit/s for the public network and 2 x 100Gbit/s for the cluster network. About this setup i have 2 questions. 1. is it even necessary to evaluate a stretched cluster since the WAN connections are as fast as the local ones (including the latency, since it is only 25km) ? 2. If using a stretched pool across all 3 datacenters, what happens if one datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ? If i am on the wrong path, maybe someone has a link for me, where is can find information on this setup ? Cheers Soeren

3 7

Re: HEALTH_ERR: 1 MDSs report damaged metadata - damage_type=dentry
by Christophe DIARRA 28 Apr '25

28 Apr '25

Hello David, Thank you for the tip about the scrubbing. I have tried the commands found in the documentation but it seems to have no effect: [root@mon-01 ~]#*ceph tell mds.cfs_irods_test:0 scrub start / recursive,repair,force* 2025-04-17T12:07:20.958+0200 7fd4157fa640 0 client.86301 ms_handle_reset on v2:130.84.80.10:6800/3218663047 2025-04-17T12:07:20.979+0200 7fd4157fa640 0 client.86307 ms_handle_reset on v2:130.84.80.10:6800/3218663047 { "return_code": 0, "scrub_tag": "733b1c6d-a418-4c83-bc8e-b28b556e970c", "mode": "asynchronous" } [root@mon-01 ~]#*ceph tell mds.cfs_irods_test:0 scrub status* 2025-04-17T12:07:30.734+0200 7f26cdffb640 0 client.86319 ms_handle_reset on v2:130.84.80.10:6800/3218663047 2025-04-17T12:07:30.753+0200 7f26cdffb640 0 client.86325 ms_handle_reset on v2:130.84.80.10:6800/3218663047 { "status": "no active scrubs running", "scrubs": {} } [root@mon-01 ~]# ceph -s cluster: id: b87276e0-1d92-11ef-a9d6-507c6f66ae2e *health: HEALTH_ERR 1 MDSs report damaged metadata* services: mon: 3 daemons, quorum mon-01,mon-03,mon-02 (age 19h) mgr: mon-02.mqaubn(active, since 19h), standbys: mon-03.gvywio, mon-01.xhxqdi mds: 1/1 daemons up, 2 standby osd: 368 osds: 368 up (since 18h), 368 in (since 3w) data: volumes: 1/1 healthy pools: 10 pools, 4353 pgs objects: 1.25M objects, 3.9 TiB usage: 417 TiB used, 6.4 PiB / 6.8 PiB avail pgs: 4353 active+clean Did I miss something ? The server didn't crash. I don't understand what you are meaning by "there may be a design flaw in the infrastructure (insecure cache, for example)". How to know if we have a design problem ? What should we check ? Best regards, Christophe On 17/04/2025 11:07, David C. wrote: > Hello Christophe, > > Check the file system scrubbing procedure => > https://docs.ceph.com/en/latest/cephfs/scrub/ But this doesn't > guarantee data recovery. > > Was the cluster crashed? > Ceph should be able to handle it; there may be a design flaw in the > infrastructure (insecure cache, for example). > > David > > Le jeu. 17 avr. 2025 à 10:44, Christophe DIARRA > <christophe.diarra(a)idris.fr> a écrit : > > Hello, > > After an electrical maintenance I restarted our ceph cluster but it > remains in an unhealthy state: HEALTH_ERR 1 MDSs report damaged > metadata. > > How to repair this damaged metadata ? > > To bring down the cephfs cluster I unmounted the fs from the client > first and then did: ceph fs set cfs_irods_test down true > > To bring up the cephfs cluster I did: ceph fs set cfs_irods_test > down false > > Fortunately the cfs_irods_test fs is almost empty and is a fs for > tests.The ceph cluster is not in production yet. > > Following is the current status: > > [root@mon-01 ~]# ceph health detail > HEALTH_ERR 1 MDSs report damaged metadata > *[ERR] MDS_DAMAGE: 1 MDSs report damaged metadata > mds.cfs_irods_test.mon-03.vlmeuz(mds.0): Metadata damage > detected* > > [root@mon-01 ~]# ceph -s > cluster: > id: b87276e0-1d92-11ef-a9d6-507c6f66ae2e > health: HEALTH_ERR > 1 MDSs report damaged metadata > > services: > mon: 3 daemons, quorum mon-01,mon-03,mon-02 (age 17h) > mgr: mon-02.mqaubn(active, since 17h), standbys: mon-03.gvywio, > mon-01.xhxqdi > mds: 1/1 daemons up, 2 standby > osd: 368 osds: 368 up (since 17h), 368 in (since 3w) > > data: > volumes: 1/1 healthy > pools: 10 pools, 4353 pgs > objects: 1.25M objects, 3.9 TiB > usage: 417 TiB used, 6.4 PiB / 6.8 PiB avail > pgs: 4353 active+clean > > > [root@mon-01 ~]# ceph fs ls > name: cfs_irods_test, metadata pool: cfs_irods_md_test, data pools: > [cfs_irods_def_test cfs_irods_data_test ] > > [root@mon-01 ~]# ceph mds stat > cfs_irods_test:1 {0=cfs_irods_test.mon-03.vlmeuz=up:active} 2 > up:standby > > [root@mon-01 ~]# ceph fs status > cfs_irods_test - 0 clients > ============== > RANK STATE MDS ACTIVITY DNS > INOS DIRS CAPS > 0 active cfs_irods_test.mon-03.vlmeuz Reqs: 0 /s 12 15 > 14 0 > POOL TYPE USED AVAIL > cfs_irods_md_test metadata 11.4M 34.4T > cfs_irods_def_test data 0 34.4T > cfs_irods_data_test data 0 4542T > STANDBY MDS > cfs_irods_test.mon-01.hitdem > cfs_irods_test.mon-02.awuygq > MDS version: ceph version 18.2.2 > (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) > [root@mon-01 ~]# > > [root@mon-01 ~]# ceph tell mds.cfs_irods_test:0 damage ls > 2025-04-17T10:23:31.849+0200 7f4b87fff640 0 client.86181 > ms_handle_reset on v2:130.84.80.10:6800/3218663047 > <http://130.84.80.10:6800/3218663047> > 2025-04-17T10:23:31.866+0200 7f4b87fff640 0 client.86187 > ms_handle_reset on v2:130.84.80.10:6800/3218663047 > <http://130.84.80.10:6800/3218663047> > [ > { > *"damage_type": "dentry",* > "id": 241447932, > "ino": 1, > "frag": "*", > "dname": "testdir2", > "snap_id": "head", > "path": "/testdir2" > }, > { > *"damage_type": "dentry"*, > "id": 2273238993, > "ino": 1, > "frag": "*", > "dname": "testdir1", > "snap_id": "head", > "path": "/testdir1" > } > ] > [root@mon-01 ~]# > > Any help will be appreciated, > > Thanks, > > Christophe > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

5 25

Missing Grafana dash boards
by Nicola Mori 28 Apr '25

28 Apr '25

Dear Ceph users, my Cephadm cluster (19.2.1) is currenlty lacking all the Grafana dashboards so I cannot see plots of any metrics. For example, trying to access the "Performance Details" tab of my filesystem on the web dashboard I get the following message: Grafana Dashboard doesn't exist. Please refer to documentation on how to add dashboards to Grafana. The linked instructions reports how to enable the monitoring stack, which was already deployed on my system. So I first removed it: ceph orch rm prometheus ceph orch rm node-exporter ceph orch rm ceph-exporter ceph mgr module disable prometheus and the re-deployed it: ceph orch apply node-exporter ceph orch apply alertmanager ceph orch apply prometheus ceph orch apply grafana ceph mgr module enable prometheus but still I get no Grafana dashboards. How can I fix this? Thank you, Nicola

2 1

Putting an OSD host with NFS daemons in maintenance
by Devin A. Bougie 28 Apr '25

28 Apr '25

Hello, When trying to put one of our OSD hosts in maintenance, "ceph orch host maintenance enter” displays: WARNING: Removing NFS daemons can cause clients to lose connectivity. We have a single Cephfs filesystem in our cluster. Five hosts are “admin” hosts that run cephadm, mds, mgr, mon, etc. The rests of our hosts are OSDs with the spinning disks that make up our Cephfs data pool. Our current setup follows the "HIGH-AVAILABILITY NFS” documentation, which gives us an Ingress.nfs.cephfs service with the haproxy and keepalived daemons and a nfs.cephfs service for the actual nfs daemons. This service was deployed using: ceph nfs cluster create cephfs "label:_admin" --ingress --virtual_ip virtual_ip And then we manually updated the nfs.cephfs service this created to place the nfs daemons on our OSD nodes. This gives us the following: ——— service_type: ingress service_id: nfs.cephfs service_name: ingress.nfs.cephfs placement: label: _admin spec: backend_service: nfs.cephfs first_virtual_router_id: 50 frontend_port: 2049 monitor_port: 9049 virtual_ip: virtual_ip/prefix service_type: nfs service_id: cephfs service_name: nfs.cephfs placement: label: osd spec: port: 12049 ——— How can we safely put one of our osd nodes in maintenance? Or is there a better way to organize our daemons or setup and manage our NFS service that would avoid this issue? Many thanks, Devin

2 1

rbd commands don't return to prompt
by Dominique Ramaekers 25 Apr '25

25 Apr '25

Hi, Weird thing happened with my ceph. I've got nice nightly scripts (bash scripts) for making backups, snapshots, cleaning up etc... Starting from my last upgrade to ceph v19.2.2 my scripts hang during execution. The rbd map and rbd unmap commands doesn't return to the prompt. So my script invoces a command like "cephadm shell -- rbd --pool libvirt-pool map --read-only --image CmsrvXCH2-SWAP@snap_4" but my script doesn't continue because the rbd map command never quits. I've also tried it myself. After killing my script a few times, I saw several images being mapped. So let's unmap them. This happened... root@hvs001:/# rbd unmap /dev/rbd0 ^C root@hvs001:/# rbd unmap /dev/rbd1 ^C The devices aren't mounted so the unmapping isn't blocked. Strangely enough after activating the unmap command, the device is removed from the /dev but I still have to do a ^C to return to the shell... Do I have an issue with my ceph cluster? Has anybody experienced something similar? Should I report a bug? Greetings, Dominique.

2 3

Pull from local registry fails
by Sake Ceph 25 Apr '25

25 Apr '25

We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine. I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials. When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image". But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image harbor.url/project/ceph/ceph:v18.2.6-20250417 pull Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [ "harbor.url/project/ceph/ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551" ] } I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password So what's wrong? Currently using 18.2.4.

3 7

help：unsubscribe
by 黄浩然 25 Apr '25

25 Apr '25

| | 黄浩然 | | huanghaoran_1989(a)163.com | ---- Replied Message ---- | From | <ceph-users-request(a)ceph.io> | | Date | 04/25/2025 17:02 | | To | <ceph-users(a)ceph.io> | | Subject | ceph-users Digest, Vol 130, Issue 155 | Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics: 1. Re: Patching Ceph cluster (Sake Ceph) 2. Re: Patching Ceph cluster (Sake Ceph) ---------------------------------------------------------------------- Date: Fri, 25 Apr 2025 10:59:52 +0200 (CEST) From: Sake Ceph <ceph(a)paulusma.eu> Subject: [ceph-users] Re: Patching Ceph cluster To: Michael Worsham <mworsham(a)datadimensions.com>, "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> Message-ID: <683056728.264684.1745571592421(a)webmail.strato.com> Content-Type: text/plain; charset=UTF-8 The tiebreaker is for a stretch cluster, which we deployed. It's only used to assign the host to a group. The Playbook is indeed written for RHEL, because that's the OS we use. It can be improved a lot, but it's a start for someone else. I know I still need to share this on GitHub, but to busy at the moment at work and at home. Op 25-04-2025 06:05 CEST schreef Michael Worsham <mworsham(a)datadimensions.com>: I've been reading over the playbook code, and it's nicely written. I know it's primarily RHEL focused, but I think it could be modified for Ubuntu/Debian platforms as well. A couple of questions though... In the test example hosts file, what is the tiebreaker? I know there isn't a role in the roles folder, but do you have an example of one, just so we know what it does? Thanks. -- Michael Get Outlook for Android (https://aka.ms/AAb9ysg) ------------------------------ From: Sake Ceph <ceph(a)paulusma.eu> Sent: Friday, June 14, 2024 4:28:34 AM To: Michael Worsham <mworsham(a)datadimensions.com>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Re: Patching Ceph cluster This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. I needed to do some cleaning before I could share this :) Maybe you or someone else can use it. Kind regards, Sake Op 14-06-2024 03:53 CEST schreef Michael Worsham <mworsham(a)datadimensions.com>: I'd love to see what your playbook(s) looks like for doing this. -- Michael ________________________________ From: Sake Ceph <ceph(a)paulusma.eu> Sent: Thursday, June 13, 2024 4:05 PM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Patching Ceph cluster This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. Yeah we fully automated this with Ansible. In short we do the following. 1. Check if cluster is healthy before continuing (via REST-API) only health_ok is good 2. Disable scrub and deep-scrub 3. Update all applications on all the hosts in the cluster 4. For every host, one by one, do the following: 4a. Check if applications got updated 4b. Check via reboot-hint if a reboot is necessary 4c. If applications got updated or reboot is necessary, do the following : 4c1. Put host in maintenance 4c2. Reboot host if necessary 4c3. Check and wait via 'ceph orch host ls' if status of the host is maintance and nothing else 4c4. Get host out of maintenance 4d. Check if cluster is healthy before continuing (via Rest-API) only warning about scrub and deep-scrub is allowed, but no pg's should be degraded 5. Enable scrub and deep-scrub when all hosts are done 6. Check if cluster is healthy (via Rest-API) only health_ok is good 7. Done For upgrade the OS we have something similar, but exiting maintenance mode is broken (with 17.2.7) :( I need to check the tracker for similar issues and if I can't find anything, I will create a ticket. Kind regards, Sake Op 12-06-2024 19:02 CEST schreef Daniel Brown <daniel.h.brown(a)thermify.cloud>: I have two ansible roles, one for enter, one for exit. There’s likely better ways to do this — and I’ll not be surprised if someone here lets me know. They’re using orch commands via the cephadm shell. I’m using Ansible for other configuration management in my environment, as well, including setting up clients of the ceph cluster. Below excerpts from main.yml in the “tasks” for the enter/exit roles. The host I’m running ansible from is one of my CEPH servers - I’ve limited which process run there though so it’s in the cluster but not equal to the others. ————— Enter ————— - name: Ceph Maintenance Mode Enter shell: cmd: ' cephadm shell ceph orch host maintenance enter {{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }} --force --yes-i-really-mean-it ‘ become: True ————— Exit ————— - name: Ceph Maintenance Mode Exit shell: cmd: 'cephadm shell ceph orch host maintenance exit {{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }} ‘ become: True connection: local - name: Wait for Ceph to be available ansible.builtin.wait_for: delay: 60 host: '{{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }}’ port: 9100 connection: local On Jun 12, 2024, at 11:28 AM, Michael Worsham <mworsham(a)datadimensions.com> wrote: Interesting. How do you set this "maintenance mode"? If you have a series of documented steps that you have to do and could provide as an example, that would be beneficial for my efforts. We are in the process of standing up both a dev-test environment consisting of 3 Ceph servers (strictly for testing purposes) and a new production environment consisting of 20+ Ceph servers. We are using Ubuntu 22.04. -- Michael From: Daniel Brown <daniel.h.brown(a)thermify.cloud> Sent: Wednesday, June 12, 2024 9:18 AM To: Anthony D'Atri <anthony.datri(a)gmail.com> Cc: Michael Worsham <mworsham(a)datadimensions.com>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Patching Ceph cluster This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. There’s also a Maintenance mode that you can set for each server, as you’re doing updates, so that the cluster doesn’t try to move data from affected OSD’s, while the server being updated is offline or down. I’ve worked some on automating this with Ansible, but have found my process (and/or my cluster) still requires some manual intervention while it’s running to get things done cleanly. On Jun 12, 2024, at 8:49 AM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: Do you mean patching the OS? If so, easy -- one node at a time, then after it comes back up, wait until all PGs are active+clean and the mon quorum is complete before proceeding. On Jun 12, 2024, at 07:56, Michael Worsham <mworsham(a)datadimensions.com> wrote: What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. ------------------------------ Date: Fri, 25 Apr 2025 11:02:28 +0200 (CEST) From: Sake Ceph <ceph(a)paulusma.eu> Subject: [ceph-users] Re: Patching Ceph cluster To: Lukasz Borek <lukasz(a)borek.org.pl> Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> Message-ID: <796303375.265011.1745571748485(a)webmail.strato.com> Content-Type: text/plain; charset=UTF-8 Nope, it was really broken in 17.2.7. When RHEL 10 comes available, I will look into this part again :) Op 25-04-2025 07:22 CEST schreef Lukasz Borek <lukasz(a)borek.org.pl>: For upgrade the OS we have something similar, but exiting maintenance mode is broken (with 17.2.7) :( I need to check the tracker for similar issues and if I can't find anything, I will create a ticket For 18.2.2 first maint exit command threw an exception for some reason. In my patching script I execute commands in a loop and the 2nd shoot usually works. exit maint 1/3 Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1809, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 183, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 119, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 108, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 778, in _host_maintenance_exit raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 237, in raise_if_exception e = pickle.loads(c.serialized_exception) TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr' exit maint 2/3 Ceph cluster f3e63d9e-2f4c-11ef-87a2-0f1170f55ed5 on cephbackup-osd1 has exited maintenance mode exit maint 3/3 Error EINVAL: Host cephbackup-osd1 is not in maintenance mode Fri Apr 25 07:17:58 CEST 2025 cluster state is HEALTH_WARN Fri Apr 25 07:18:02 CEST 2025 cluster state is HEALTH_WARN [...] On Thu, 13 Jun 2024 at 22:07, Sake Ceph <ceph(a)paulusma.eu> wrote: For upgrade the OS we have something similar, but exiting maintenance mode is broken (with 17.2.7) :( I need to check the tracker for similar issues and if I can't find anything, I will create a ticket. Kind regards, Sake Op 12-06-2024 19:02 CEST schreef Daniel Brown <daniel.h.brown(a)thermify.cloud>: I have two ansible roles, one for enter, one for exit. There’s likely better ways to do this — and I’ll not be surprised if someone here lets me know. They’re using orch commands via the cephadm shell. I’m using Ansible for other configuration management in my environment, as well, including setting up clients of the ceph cluster. Below excerpts from main.yml in the “tasks” for the enter/exit roles. The host I’m running ansible from is one of my CEPH servers - I’ve limited which process run there though so it’s in the cluster but not equal to the others. ————— Enter ————— - name: Ceph Maintenance Mode Enter shell: cmd: ' cephadm shell ceph orch host maintenance enter {{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }} --force --yes-i-really-mean-it ‘ become: True ————— Exit ————— - name: Ceph Maintenance Mode Exit shell: cmd: 'cephadm shell ceph orch host maintenance exit {{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }} ‘ become: True connection: local - name: Wait for Ceph to be available ansible.builtin.wait_for: delay: 60 host: '{{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }}’ port: 9100 connection: local On Jun 12, 2024, at 11:28 AM, Michael Worsham <mworsham(a)datadimensions.com> wrote: Interesting. How do you set this "maintenance mode"? If you have a series of documented steps that you have to do and could provide as an example, that would be beneficial for my efforts. We are in the process of standing up both a dev-test environment consisting of 3 Ceph servers (strictly for testing purposes) and a new production environment consisting of 20+ Ceph servers. We are using Ubuntu 22.04. -- Michael From: Daniel Brown <daniel.h.brown(a)thermify.cloud> Sent: Wednesday, June 12, 2024 9:18 AM To: Anthony D'Atri <anthony.datri(a)gmail.com> Cc: Michael Worsham <mworsham(a)datadimensions.com>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Patching Ceph cluster This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. There’s also a Maintenance mode that you can set for each server, as you’re doing updates, so that the cluster doesn’t try to move data from affected OSD’s, while the server being updated is offline or down. I’ve worked some on automating this with Ansible, but have found my process (and/or my cluster) still requires some manual intervention while it’s running to get things done cleanly. On Jun 12, 2024, at 8:49 AM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: Do you mean patching the OS? If so, easy -- one node at a time, then after it comes back up, wait until all PGs are active+clean and the mon quorum is complete before proceeding. On Jun 12, 2024, at 07:56, Michael Worsham <mworsham(a)datadimensions.com> wrote: What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Łukasz Borek lukasz(a)borek.org.pl ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 130, Issue 155 ********************************************

1 0

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

FS S5850 - 7 nodes lots and continuous "slow OSD heartbeats"
by Steven Vacaroaia 25 Apr '25

25 Apr '25

Hi, We are in the process of using for production a 7 host reef ( 18.2.2) cluster and we noticed lots and continuous "slow OSD heartbeats" although there is no clients, no data and the nodes are using 2 x 25GB for Frontend and 2 x 100GB for backend The logical culprits is the network we have 2 x S5850 with MLAG and LACP I was wondering if any of you are familiar with FS and can provide some guidance as to how to properly configured ( or troubleshoot) Note We are running in a air gapped environment so cannot provide outputs/config but here are some pictures Many thanks Steven [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png]

2 2

Increase amount of OSDs on nodes
by Florian Schwab 24 Apr '25

24 Apr '25

Hi everyone, I’m currently facing the following challenge were I’d like to hear what the community thinks on how to solve it. We have a ceph cluster (only using RGW) with which we had some performance issues when initially setting it up and instead of using all 60 HDDs installed in each server we only use half (30) of the HDDs - they exist but are set to “out”. Using only half of the OSDs per server solved our performance issues - not the part of my question just a bit of backstory for this weird setup. We have now optimised things and are looking into getting those OSDs back “in”. However we have quite some client load on the system and we want to keep the any necessary downtime (for users) low. The HDDs are only used for the buckets.data pool (using erasure coding) as the other pools necessary for RGW are on separate NVMe drives. To achieve this I was looking into using pgremapper and upmap-remapped. My expectation was to be able to remap all PGs to stay on the “old” OSDs (the ones that are currently “in”) and have zero PGs on the “new” OSDs (the ones that are currently “out”). If that part is done I thought somehow I can double the amount of PGs to cover the now doubled capacity for the pool. What I tried so far (with some variations): - Set norecover and nobackfill flags - Set “new” OSDs to “in” - Use pgremapper and/or upmap-remapped to create upmap entries so the acting OSDs of a PGs are unchanged - Increase the amount of PGs - Map new PGs to “new” OSDs - (Maybe remove the upmap entries for the PGs one at a time over an extend period to allow for automatic balancing again) I’m trying this on a smaller test setup with 6 servers and 8 OSDs per server (4 in / 4 out) so 48 OSDs in total. For testing I filled the “in” OSDs to around 50% so the overall used capacity is also 50% - when bringing the other 24 OSDs "in" this is of course means only 25% of the overall capacity is used. The following information is based on this test setup. The pgremapper/upmap-remapped part is where I’m currently stuck as I always end up with around 2% of objects being misplaced. What I was also able to manage is that if I after setting all OSDs to “in”, remap the PGs, increase the PG count and remap again I end up around 3% of misplaced objects however the additional/new OSDs then will have zero PGs again. Is it somehow possible to add the OSDs and double the PG count without causing any (or only minimal) backfill? If possible I also want to bring in the “new” OSDs in batches so we can check the performance after each batch to avoid having issues in the final state with all OSDs set to “in”. Thanks in advance for any input. Cheers, Florian

2 1

Inquiry About IO Ordering in Ceph Octopus Version
by 段世博 24 Apr '25

24 Apr '25

We are currently using the Ceph Octopus version and have some questions regarding a specific commit in librbd ( https://github.com/ceph/ceph/commit/081d28ae7ca46fd1f40034cc558def77a95a9294 ). Could you please clarify why Ceph implements ordering for overlapping IO? Our understanding is that overlapping IO in block storage can typically be parallelized. We are curious about which components of Ceph depend on this feature. Additionally, we noticed that this feature was removed in the subsequent Pacific version. Could you provide some insight into the reasons behind this decision? Thank you for your assistance. Best regards, shibo

2 1

Removing a zonegroup from multi-site cluster
by Boris 24 Apr '25

24 Apr '25

Hi, I wanted to remove a zonegroup from a multisite cluster. I deleted it from the master with radosgw-admin zonegroup delete --rgw-zonegroup=zonegroup radosgw-admin period update --commit After that, I had to restart all RGW daemons, because the metadata sync stopped. The zonegroup is gone on the master, but it is still listed in all other sites. I checked the current period on all sites and they are all the same, but the deleted zonegroup is still in the radosgw-admin period get on all zonegroups that are not the master. How do I proceed? Best wishes Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

ceph deployment best practice
by gagan tiwari 23 Apr '25

23 Apr '25

Hi Guys, We have an HPC environment which currently has a single master host that stores the entire data around 100T and exports that data via NFS to the clients. We are using OpenZFS on the single master host. But now we need to store much more data around 500T and we are facing scalability issues with ZFS as it can't be scaled out. So, we are considering replacing OpenZFS with Ceph. I have gone the online docs on https://docs.ceph.com and I have come up with the following ceph deployment plan :- We would start deploying Ceph with 4 hosts ( HP Proliant servers ) each running RockyLinux 9. One of the hosts called ceph-adm will be smaller one and will have following hardware :- 2x4T SSD with raid 1 to install OS on. 8 Core with 3600MHz freq. 64G RAM We are planning to run all Ceph daemons except OSD daemon like monitor , metadata ,etc on this host. We will have 3 hosts to run OSD which will store actual data. Each OSD host will have following hardware 2x4T SSD with raid 1 to install OS on. 22X8T SSG to store data ( OSDs ) ( without partition ). We will use entire disk without partitions Each OSD host will have 128G RAM ( No swap space ) Each OSD host will have 16 cores. All 4 hosts will connect to each via 10G nic. The 500T data will be accessed by the clients. We need to have read performance as fast as possible. We can't afford data loss and downtime. So, we want to have a Ceph deployment which serves our purpose. So, please advise me if the plan that I have designed will serve our purpose. Or is there a better way , please advise that. Thanks, Gagan We have a HP storage server with 12 SDD of 5T each and have set-up hardware RAID6 on these disks. HP storage server has 64G RAM and 18 cores. So, please advise how I should go about setting up Ceph on it to have best read performance. We need fastest read performance. Thanks, Gagan

7 22

Cluster recovery: DC power failure killed OSD node BlueStore block.DB devices
by Paul Browne 23 Apr '25

23 Apr '25

Hi ceph-users, We recently suffered a total power failure at our main DC; fortunately, our production Ceph cluster emerged unscathed but a smaller Ceph cluster came back with the majority of its dedicated SSD BlueStore block.DB devices dead (but its HDD OSD devices unharmed). This cluster underpinned a small OpenStack cloud, so it would be preferable to recover it rather than writing it off. In terms of deployment tooling, this ailing Ceph cluster is a fairly standard Red Hat Ceph Storage 7 (so Quincy) cephadm deployed cluster, with the main wrinkle about it being that both the DB and HDD OSD devices make use of the cephadm supported LVM->LUKS layering above the BlueStore devices. The dead BlueStore block.DB devices are of course blocking the surviving HDD OSD daemons (in cephadm deployed containers) from coming up cleanly and so the Ceph cluster status is currently very degraded (attached status for the ugly picture) I've kicked around some ideas of recovering the dead DB devices and restarting down+out OSDs by; * Manually partitioning replacement SSDs into new DB device partitions+LVs * Installing the same LUKS keys on them retrieved from the Ceph config DB, matching up against which OSD is on which OSD host. * Manually changing over device links for OSDs to their DB device with "ceph-bluestore-tool bluefs-bdev-new-db" or similar * Try restarting OSDs with updated links to new LVM->LUKS->block.DB devices This approach seems highly messy and subject to needing to extract a lot of information error-free from dumps of ceph volume lvm, list in order to exactly match extant OSD UUIDs to newly created DB devicemapper devices. Is there going to be some smarter/better/faster way to non-destructively recover these intact HDD OSDs which have links to dead block.DB devices, using native cephadm tooling rather than getting so low-level as all the above? Many thanks for any advice, ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

5 9

Help with HA NFS
by Devin A. Bougie 23 Apr '25

23 Apr '25

Hello, We’ve found that if we lose one of the nfs.cephfs service daemons in our cephadm 19.2.2 cluster, all NFS traffic is blocked until either: - the down nfs.cephfs daemon is restarted - or we reconfigure the placement of the nfs.cephs service to not use the affected host. After this, the ingress.nfs.cephfs service is automatically reconfigured and everything resumes Our current setup follows the "HIGH-AVAILABILITY NFS” documentation, which gives us an ingress.nfs.cephfs service with the haproxy and keepalived daemons and an nfs.cephfs service for the actual nfs daemons. This service was deployed using: ceph nfs cluster create cephfs "label:_admin" --ingress --virtual_ip virtual_ip And then we updated the ingress.nfs.cephfs service to only deploy a single service (which in this case, results in two daemons on a single host). This gives us the following: ——— [root@cephman1 ~]# ceph orch ls --service_name=ingress.nfs.cephfs --export service_type: ingress service_id: nfs.cephfs service_name: ingress.nfs.cephfs placement: count: 1 label: _admin spec: backend_service: nfs.cephfs first_virtual_router_id: 50 frontend_port: 2049 monitor_port: 9049 virtual_ip: 128.84.45.48/22 [root@cephman1 ~]# ceph orch ls --service_name=nfs.cephfs --export service_type: nfs service_id: cephfs service_name: nfs.cephfs placement: label: _admin spec: port: 12049 ——— Can anyone show us the config for a true “HA” nfs service where they can lose any single host without impacting access to the NFS export from clients? I would expect to be able to lose the host running the ingress.nfs.cephfs service, and have it automatically restarted on a different host. Likewise, I would expect to be able to lose an nfs.cephs daemon without impacting access to the export. Or should we be taking a completely different approach and move our NFS service out of Ceph and into our pacemaker / corosync cluster? Sorry if this sounds redundant to questions I’ve previously asked, but we’ve reconfigured things a little and it feels like we’re getting closer with each attempt? Many thanks, Devin

5 5

NVMeOF GW with Squid
by Michel Raabe 23 Apr '25

23 Apr '25

Hi all, does anyone have a successfully - with connected clients/traffic - running nvmeof gw with squid? https://docs.ceph.com/en/squid/rbd/nvmeof-overview/ Thanks! Michel -- Michel Raabe Cloud Solution Architect Mail: raabe(a)b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / https://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

1 0

Re: Cephadm upgrade from 16.2.15 -> 17.2.0
by Jeremy Hansen 22 Apr '25

22 Apr '25

Snipped some of the irrelevant logs to keep message size down. ceph config-key get mgr/cephadm/upgrade_state {"target_name": "quay.io/ceph/ceph:v17.2.0", "progress_id": "e7e1a809-558d-43a7-842a-c6229fdc57af", "target_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "target_digests": ["quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667"], "target_version": "17.2.0", "fs_original_max_mds": null, "fs_original_allow_standby_replay": null, "error": null, "paused": false, "daemon_types": null, "hosts": null, "services": null, "total_count": null, "remaining_count": null} What should I do next? Thank you! -jeremy > On Sunday, Apr 06, 2025 at 1:38 AM, Eugen Block <eblock(a)nde.ag (mailto:eblock@nde.ag)> wrote: > Can you check if you have this config-key? > > ceph config-key get mgr/cephadm/upgrade_state > > If you reset the MGRs, it might be necessary to clear this key, > otherwise you might end up in some inconsistency. Just to be sure. > > Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: > > > Thanks. I’m trying to be extra careful since this cluster is > > actually in use. I’ll wait for your feedback. > > > > -jeremy > > > > > On Saturday, Apr 05, 2025 at 3:39 PM, Eugen Block <eblock(a)nde.ag > > > (mailto:eblock@nde.ag)> wrote: > > > No, that's not necessary, just edit the unit.run file for the MGRs to > > > use a different image. See Frédéric's instructions: > > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/32APKOXKRA… > > > > > > But I'm not entirely sure if you need to clear some config-keys first > > > in order to reset the upgrade state. If I have time, I'll try to check > > > tomorrow, or on Monday. > > > > > > Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: > > > > > > > Would I follow this process to downgrade? > > > > > > > > > > > https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying… > > > > > > > > Thank you > > > > > > > > > On Saturday, Apr 05, 2025 at 2:04 PM, Jeremy Hansen > > > > > <jeremy(a)skidrow.la (mailto:jeremy@skidrow.la)> wrote: > > > > > ceph -s claims things are healthy: > > > > > > > > > > ceph -s > > > > > cluster: > > > > > id: 95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1 > > > > > health: HEALTH_OK > > > > > > > > > > services: > > > > > mon: 3 daemons, quorum cn01,cn03,cn02 (age 20h) > > > > > mgr: cn03.negzvb(active, since 26m), standbys: cn01.tjmtph, > > > > > cn02.ceph.xyz.corp.ggixgj > > > > > mds: 1/1 daemons up, 2 standby > > > > > osd: 15 osds: 15 up (since 19h), 15 in (since 14M) > > > > > > > > > > data: > > > > > volumes: 1/1 healthy > > > > > pools: 6 pools, 610 pgs > > > > > objects: 284.59k objects, 1.1 TiB > > > > > usage: 3.3 TiB used, 106 TiB / 109 TiB avail > > > > > pgs: 610 active+clean > > > > > > > > > > io: > > > > > client: 255 B/s rd, 1.2 MiB/s wr, 10 op/s rd, 16 op/s wr > > > > > > > > > > > > > > > > > > > > — > > > > > How do I downgrade if the orch is down? > > > > > > > > > > Thank you > > > > > -jeremy > > > > > > > > > > > > > > > > > > > > > On Saturday, Apr 05, 2025 at 1:56 PM, Eugen Block <eblock(a)nde.ag > > > > > (mailto:eblock@nde.ag)> wrote: > > > > > > It would help if you only pasted the relevant parts. Anyway, these two > > > > > > sections stand out: > > > > > > > > > > > > ---snip--- > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO root] > > > > > > Some PGs (1.000000) are unknown; try again later > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed to > > > > > > construct class in 'cephadm' > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Traceback > > > > > > (most recent call last): > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > self.upgrade = CephadmUpgrade(self) > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > self.upgrade_state: Optional[UpgradeState] = > > > > > > UpgradeState.from_json(json.loads(t)) > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > return cls(**c) > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > TypeError: __init__() got an unexpected keyword argument > > > 'daemon_types' > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() > > > > > > Failed to run module in active mode ('cephadm') > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed to > > > > > > construct class in 'snap_schedule' > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Traceback > > > > > > (most recent call last): > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, > > > in __init__ > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > self.client = SnapSchedClient(self) > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > > > > > > 158, in __init__ > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > with self.get_schedule_db(fs_name) as conn_mgr: > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > > > > > > 192, in get_schedule_db > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > db.executescript(dump) > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > sqlite3.OperationalError: table schedules already exists > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() > > > > > > Failed to run module in active mode ('snap_schedule') > > > > > > ---snip--- > > > > > > > > > > > > Your cluster seems to be in an error state (ceph -s) because of an > > > > > > unknown PG. It's recommended to have a healthy cluster before > > > > > > attemping an upgrade. It's possible that these errors come from the > > > > > > not upgraded MGR, I'm not sure. > > > > > > > > > > > > Since the upgrade was only successful for two MGRs, I am thinking > > > > > > about downgrading both MGRs back to 16.2.15, then retry an upgrade to > > > > > > a newer version, either 17.2.8 or 18.2.4. I haven't checked the > > > > > > snap_schedule error yet, though. Maybe someone else knows > > > that already. > > > > > >

2 15

April 21st CSC Meeting Notes
by Adam King 21 Apr '25

21 Apr '25

[Matt] required github checks (and make check) instability (and long run times) greatly hurt developer productivity - thanks for renewed attention to this from several folks - can we break up make check into component wise portions - can we take ceph api test out of the CI checks - it also runs in teuthology - more stable there - should discuss with Ernesto [Venky] Next squid/reef point release? - Yuri to open 18.2.7 and 19.2.3 thread - PTLs can review and decide what they need in these releases - Reef EOL is August 2025 - should plan for final point release around then - T release may complicate planning [Yehuda] cephx changes update - planned to be completed by the end of May - Should not block initial T release - will want to backport this [Mike] Infra update: Shaman updated / chacra is next - Are there any other infra projects that need updating? - nothing suggested

1 0

Dashboard lies to me
by Tim Holloway 21 Apr '25

21 Apr '25

Although I've had this problem since at least Pacific, I'm still seeing it on Reef. After much pain and suffering (covered elsewhere), I got my Prometheus services deployed as intended, Ceph health OK, green across the board. However, over the weekend, the dreaded "CephMgrPrometheusModuleInactive" alert has returned to the Dashboard. "The mgr/prometheus module at dell02.mousetech.com:9283 is unreachable." It's a blatant lie. I still get "Ceph HEALTH_OK". All monitor status command show everything running. Checking ports on the host says it's listening. More to the point, I can send my desktop browser to http://dell02.mousetech.com:9283 and get a page that will allow me to see the metrics. So everyone can see it but the Dashboard! I did have some issues when the other prometheus host couldn't resolve the hostname, but I fixed that for all ceph hosts and it was green for days. Now the error is back. Restarting Prometheus didn't help. How is the Dashboard hallucinating this??? Tim

3 7

Upgrade from 18.2.4 to 19.x or even 20
by e3gh75＠gmail.com 21 Apr '25

21 Apr '25

Hello, I have dumb question but hopefully simple question. In the cephadm documentation it states you can upgrade from a point release to another point release without problem, 15.2.2 to 15.2.3. What about jumping up a version or two, like from 18.2.4 to 19.2.2? What have others experience been? Any advice? Thank you for your time.

5 4

Question about compatibility of librdb and backend cluster
by 段世博 21 Apr '25

21 Apr '25

Hi everyone, I would like to ask about the compatibility of librdb versions. If the client version (librbd) used is 17.2.8, but the backend cluster is 16.2.15 or 15.2.17, or even lower versions, can it be accessed normally? Any reply will be very much appreciated.

2 1

Re: Preflight Playbook
by Frédéric Nass 18 Apr '25

18 Apr '25

Do you mean RHCS 5 to 6? The recommended and documented upgrade path is actually RHCS 5 to 7 after upgrading RHCS 5 nodes to RHEL 9. Did you upgrade cephadm and cephadm-ansible packages before running the preflight playbook? Regards, Frédéric. ________________________________ De : Alex <mr.alexey(a)gmail.com> Envoyé : mercredi 16 avril 2025 20:03 À : Frédéric Nass Cc: ceph-users Objet : Re: [ceph-users] Preflight Playbook I was upgrading from Ceph 5 to 6 on RHEL 9. Looking at the dnf logs it seems that i was running crun 1.8.7 and now on 1.16.1 Thanks.

2 8

ceph mon out of quorom and cephfs degraded
by Amudhan P 17 Apr '25

17 Apr '25

Hi, I am having 2 problems with my Ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) deployed thru cephadm. First issue :- 1 out 3 mon service went out of quorum . When restarted service it comes normal but after a few minutes in ceph watch log it reports slow ops and mon goes out of quorum. Node where this mon service failed had one weird thing that I could see 40% of wait in the top command. But I don't see any error in dmesg or anything related to drive IO error. Below are the logs that were printed in ceph watch command. 2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum strg-node2,strg-node3 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] mon.strg-node1 (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down (out of quorum) For now this is not appearing again. Second issue Cephfs degraded :- I have 2 MDS services running in 2 different nodes. Both are in a stopped state. when running Ceph -s command cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 4h) mgr: strg-node2.unyimy(active, since 4h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 4h), 32 in (since 10w) data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 321 active+clean Volume shows recovering but there wasn't any progress till now even manual start mds service fails again. In Ceph -s command under services it shows mds up no any mds service is running. Below is a log snip from one of the mds service. -25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, ex> Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a17400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a17c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) recover start Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) read_head Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for journal 0x200 to recover... Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a25000 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13968309289 (hea> Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue operator(): open complete Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) set_writeable Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 189741504921> Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> 2025-04-16T09:59:30.098+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1897428915052> Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal 0x200 recovered. Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered journal 0x200 in format 1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 Booting: 1: loading/discovering base inodes Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x100 Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a25800 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a5dc00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: replaying mds log Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) Apr 16 15:29:30 strg-node4 bash[7566]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43dd293b20] Apr 16 15:29:30 strg-node4 bash[7566]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Not sure what caused the issue. I couldn't find any resources to fix this issue. Need help from someone to bring the ceph cluster online.

2 6

HEALTH_ERR: 1 MDSs report damaged metadata - damage_type=dentry
by Christophe DIARRA 17 Apr '25

17 Apr '25

Hello, After an electrical maintenance I restarted our ceph cluster but it remains in an unhealthy state: HEALTH_ERR 1 MDSs report damaged metadata. How to repair this damaged metadata ? To bring down the cephfs cluster I unmounted the fs from the client first and then did: ceph fs set cfs_irods_test down true To bring up the cephfs cluster I did: ceph fs set cfs_irods_test down false Fortunately the cfs_irods_test fs is almost empty and is a fs for tests.The ceph cluster is not in production yet. Following is the current status: [root@mon-01 ~]# ceph health detail HEALTH_ERR 1 MDSs report damaged metadata *[ERR] MDS_DAMAGE: 1 MDSs report damaged metadata mds.cfs_irods_test.mon-03.vlmeuz(mds.0): Metadata damage detected* [root@mon-01 ~]# ceph -s cluster: id: b87276e0-1d92-11ef-a9d6-507c6f66ae2e health: HEALTH_ERR 1 MDSs report damaged metadata services: mon: 3 daemons, quorum mon-01,mon-03,mon-02 (age 17h) mgr: mon-02.mqaubn(active, since 17h), standbys: mon-03.gvywio, mon-01.xhxqdi mds: 1/1 daemons up, 2 standby osd: 368 osds: 368 up (since 17h), 368 in (since 3w) data: volumes: 1/1 healthy pools: 10 pools, 4353 pgs objects: 1.25M objects, 3.9 TiB usage: 417 TiB used, 6.4 PiB / 6.8 PiB avail pgs: 4353 active+clean [root@mon-01 ~]# ceph fs ls name: cfs_irods_test, metadata pool: cfs_irods_md_test, data pools: [cfs_irods_def_test cfs_irods_data_test ] [root@mon-01 ~]# ceph mds stat cfs_irods_test:1 {0=cfs_irods_test.mon-03.vlmeuz=up:active} 2 up:standby [root@mon-01 ~]# ceph fs status cfs_irods_test - 0 clients ============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cfs_irods_test.mon-03.vlmeuz Reqs: 0 /s 12 15 14 0 POOL TYPE USED AVAIL cfs_irods_md_test metadata 11.4M 34.4T cfs_irods_def_test data 0 34.4T cfs_irods_data_test data 0 4542T STANDBY MDS cfs_irods_test.mon-01.hitdem cfs_irods_test.mon-02.awuygq MDS version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) [root@mon-01 ~]# [root@mon-01 ~]# ceph tell mds.cfs_irods_test:0 damage ls 2025-04-17T10:23:31.849+0200 7f4b87fff640 0 client.86181 ms_handle_reset on v2:130.84.80.10:6800/3218663047 2025-04-17T10:23:31.866+0200 7f4b87fff640 0 client.86187 ms_handle_reset on v2:130.84.80.10:6800/3218663047 [ { *"damage_type": "dentry",* "id": 241447932, "ino": 1, "frag": "*", "dname": "testdir2", "snap_id": "head", "path": "/testdir2" }, { *"damage_type": "dentry"*, "id": 2273238993, "ino": 1, "frag": "*", "dname": "testdir1", "snap_id": "head", "path": "/testdir1" } ] [root@mon-01 ~]# Any help will be appreciated, Thanks, Christophe

1 0

CephFS Snapshot Mirroring
by Vladimir Cvetkovic 16 Apr '25

16 Apr '25

Hi everyone, We have two remote clusters and are trying to set up snapshot mirroring. Our directories are in a syncing state but never seem to actually sync the snapshots. Snapshot mirroring seems way to slow for large file systems - does anyone have a working setup? Thanks!

4 5

reef 18.2.6 hotfix QE validation status
by Yuri Weinstein 16 Apr '25

16 Apr '25

Details of this release are summarized here: https://tracker.ceph.com/issues/70938#note-1 Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke - same as in 18.2.5 rados - Radek, Laura approved? orch - Adam King, Guillaume approved? This release has two PRs: https://github.com/ceph/ceph/pull/62791 https://github.com/ceph/ceph/pull/62814 and has limited testing (only three suits above)

3 2

Preflight Playbook
by Alex 16 Apr '25

16 Apr '25

Good morning. Should the Preflight playbook maybe update `crun` as well? I was updating the Ceph version yesterday and it wouldn't work until I updated `crun`. Thanks.

2 2

Ceph crash service deployed with cephadm: unable to find a keyring
by Daniel Vogelbacher 16 Apr '25

16 Apr '25

Hello, recently I checked my server logs for error events and got some hits for the ceph crash service, deployed with cephadm. When restarted, crash service logs to journald: auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory AuthRegistry(0x7f21a0068da0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory AuthRegistry(0x7f21a861bff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 13] RADOS permission denied (error connecting to the cluster) The image is mounted with: -v /var/lib/ceph/7644057a-00f6-11f0-9a0c-eac00fed9338/crash.virt-master3/keyring:/etc/ceph/ceph.client.crash.virt-master3.keyring so I assume there should be a key available, but crash daemon searches for a admin keyring instead of "client.crash.virt-master3". Any advice how to fix this error? I see these error logs on all machines where crash service is deployed. I've tried a redeployment without any effect. # ceph -v ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable) # systemctl restart ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service # journalctl -u ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service -ef Output: Apr 12 18:36:24 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1402272]: *** Interrupted with signal 15 *** Apr 12 18:36:24 virt-master3 podman[1408122]: 2025-04-12 18:36:24.992684396 +0200 CEST m=+0.043782773 container died 6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_REF=squid, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, io.buildah.version=1.33.7, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20250124, OSD_FLAVOR=default) Apr 12 18:36:25 virt-master3 podman[1408122]: 2025-04-12 18:36:25.016797163 +0200 CEST m=+0.067895534 container remove 6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20250124, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, io.buildah.version=1.33.7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default, org.label-schema.license=GPLv2, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>) Apr 12 18:36:25 virt-master3 bash[1408104]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3 Apr 12 18:36:25 virt-master3 systemd[1]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service: Deactivated successfully. Apr 12 18:36:25 virt-master3 systemd[1]: Stopped ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338. Apr 12 18:36:25 virt-master3 systemd[1]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service: Consumed 1.540s CPU time. Apr 12 18:36:25 virt-master3 systemd[1]: Starting ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338... Apr 12 18:36:25 virt-master3 podman[1408259]: Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.498991532 +0200 CEST m=+0.071935136 container create bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.build-date=20250124, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>) Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.462816456 +0200 CEST m=+0.035760071 image pull quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.589479015 +0200 CEST m=+0.162422619 container init bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.build-date=20250124, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=squid, io.buildah.version=1.33.7) Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.595018205 +0200 CEST m=+0.167961840 container start bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20250124, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.vendor=CentOS, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) Apr 12 18:36:25 virt-master3 bash[1408259]: bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 Apr 12 18:36:25 virt-master3 systemd[1]: Started ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338. Apr 12 18:36:25 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: INFO:ceph-crash:pinging cluster to exercise our key Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 AuthRegistry(0x7f21a0068da0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 AuthRegistry(0x7f21a861bff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a5b91640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a6392640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: [errno 13] RADOS permission denied (error connecting to the cluster) Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s -- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher

3 5

v18.2.5 Reef released
by Yuri Weinstein 16 Apr '25

16 Apr '25

We're happy to announce the 5th point release in the Reef series. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/ Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface. * RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired. Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1

5 9

Reef 18.2.5: upgraded OSDs are shown as crashed
by Eugen Block 16 Apr '25

16 Apr '25

Hi, I'm currently upgrading the first customer cluster to latest Reef (18.2.5). I haven't seen that in my lab clusters yet, but it might be due to insufficient crash configuration (or the shutdown_timeout is not triggered). I found this backport for 18.2.5: https://tracker.ceph.com/issues/63527 Am I interpreting correctly that the assert is triggered because the OSDs were shut down on 18.2.4, but this won't happen anymore on 18.2.5? Thanks! Eugen ---snip--- root@storage01:~# ceph crash info 2025-04-16T08:51:38.757577Z_bd34dfdd-183a-4274-8a40-a6cb3311ca56 { "archived": "2025-04-16 08:52:50.374393", "assert_condition": "end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/osd/OSD.cc", "assert_func": "int OSD::shutdown()", "assert_line": 4585, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/osd/OSD.cc: In function 'int OSD::shutdown()' thread 7f313de0a640 time 2025-04-16T08:51:38.751362+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/osd/OSD.cc: 4585: FAILED ceph_assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)\n", "assert_thread_name": "signal_handler", "backtrace": [ "/lib64/libc.so.6(+0x3e6f0) [0x7f31412d16f0]", "/lib64/libc.so.6(+0x8b94c) [0x7f314131e94c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56015bef62b2]", "/usr/bin/ceph-osd(+0x3c9413) [0x56015bef6413]", "/usr/bin/ceph-osd(+0x3591d7) [0x56015be861d7]", "/usr/bin/ceph-osd(+0xa1da57) [0x56015c54aa57]", "/lib64/libc.so.6(+0x89c02) [0x7f314131cc02]", "/lib64/libc.so.6(+0x10ec40) [0x7f31413a1c40]" ], "ceph_version": "18.2.4", "crash_id": "2025-04-16T08:51:38.757577Z_bd34dfdd-183a-4274-8a40-a6cb3311ca56", "entity_name": "osd.162", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-osd", "stack_sig": "434a67dd4e3c173992edcf6eb5dc6014d029ca333c4904eced28762dc1959db1", "timestamp": "2025-04-16T08:51:38.757577Z", "utsname_hostname": "storage04", "utsname_machine": "x86_64", "utsname_release": "5.15.0-130-generic", "utsname_sysname": "Linux", "utsname_version": "#140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024" } ---snip---

2 2

"BUG: soft lockup" with MDS
by Janek Bevendorff 16 Apr '25

16 Apr '25

Hi, Since the latest Reef update I have the problem that some of my hosts suddenly go into a state where all CPUs are stuck in kernel mode causing all daemons on that host to become unresponsive. When I connect to the IPMI console, I see a lot of messages like: watchdog: BUG: soft lockup - CPU#8 stuck for 47s! [cron:868840] (it's basically a list of all processes running on the machine). Usually, this resolves itself after several minutes, but sometimes I have to hardreset the host. When this happens, all daemons are marked as down and I cannot interact with the host at all. I don't know what causes this but, I think it happens primarily on the hosts where my MDS run and it seems to be triggered by events such as cluster rebalances, MDS restarts, or just randomly. I found a few reports about similar issues on the bug tracker and mailing list, but they are all very unspecific, unanswered, or more than 6 years old. Is there any way I can debug this? I upgraded to Squid already, but that didn't solve the problem. I also had massive issues with this during the upgrade. Particularly at the end when the MDS were upgraded, I had constant struggles with it. I had to set the noout flag and then literally sit next to it to resume the upgrade every few minutes until it finally went through, because random MDS hosts went intermittently dark all the time. All hosts run Ubuntu 22.04 with kernel 6.8.0. Any ideas? Thanks! Janek

1 0

Announcing go-ceph v0.33.0
by John Mulligan 15 Apr '25

15 Apr '25

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.33.0 Changes include additions to the rbd, rgw, and cephfs packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project.

1 0

All OSD_UNREACHABLE in IPv6 cluster - roleback or wait?
by Björn Lässig 15 Apr '25

15 Apr '25

Hi, after upgrading from 18.2.4 to 18.2.5, I have the result that the health status of all Ceph clusters that do not use legacyIP is always HEALTH_ERR for no useful reason. Do we get a new stable Release soon (including https://github.com/ceph/ceph/pull/61323 ) or should i roleback to 18.2.4? have a nice day Björn Lässig PS: attached is some output of the errors -- cluster: id: <myclusterUUID> health: HEALTH_ERR 44 osds(s) are not reachable services: mon: 5 daemons, quorum marvin06,marvin08,cephmon1,cephmon3,cephmon2 (age 2d) mgr: cephmgr2.zvtgjh(active, since 2d), standbys: cephmgr1.sxadnq mds: 2/2 daemons up, 2 standby osd: 44 osds: 44 up (since 9h), 44 in (since 6M) data: volumes: 2/2 healthy pools: 7 pools, 1953 pgs objects: 6.51M objects, 17 TiB usage: 50 TiB used, 174 TiB / 224 TiB avail pgs: 1951 active+clean 2 active+clean+scrubbing+deep [root@ceph:~] # ceph health detail [ERR] OSD_UNREACHABLE: 44 osds(s) are not reachable osd.0's public address is not in '2001:db8:0:2::/64' subnet [… every osd of this cluster …] osd.43's public address is not in '2001:db8:0:2::/64' subnet but this is all false [root@marvin09:~] 1 # ss -tlpne | grep ceph-<myclusterUUID>@osd.43.service | sed 's/users.*//' LISTEN 0 512 [2001:db8:0:3:1d::3f]:6812 [::]:* LISTEN 0 512 [2001:db8:0:3:1d::3f]:6813 [::]:* LISTEN 0 512 [2001:db8:0:3:1d::3f]:6814 [::]:* LISTEN 0 512 [2001:db8:0:3:1d::3f]:6815 [::]:* LISTEN 0 512 [2001:db8:0:2:1d::3f]:6818 [::]:* LISTEN 0 512 [2001:db8:0:2:1d::3f]:6819 [::]:* LISTEN 0 512 [2001:db8:0:2:1d::3f]:6816 [::]:* LISTEN 0 512 [2001:db8:0:2:1d::3f]:6817 [::]:* [root@marvin09:~] # ceph config get osd.43 | grep public_network global advanced public_network 2001:db8:0:2::/64 * global advanced public_network_interface rbdfenet *

2 3

Urgent help: I accidentally nuked all my Monitor
by Jonas Schwab 15 Apr '25

15 Apr '25

Hello everyone, I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup. I am very grateful for all help! Best regards, Jonas Schwab

4 18

Upcoming Ceph Days - CFP and Join Us
by Dan van der Ster 15 Apr '25

15 Apr '25

Dear Ceph Community, Just a quick note to highlight two upcoming Ceph Days events: * Seattle, May 15: https://ceph.io/en/community/events/2025/ceph-day-seattle/ * London, June 4: https://ceph.io/en/community/events/2025/ceph-days-london/ Hope to see you there! -- Dan

3 5

Unbalance between index pool osds in rgw
by Jaemin Joo 15 Apr '25

15 Apr '25

Hi all, I am testing rgw cluster which separated between index pool osd and data pool osd. After uploading a lot of objects, I found that index pool osd usage is unbalanced. I just know that index pool use rocksdb. rocksdb has object metadata, bucket metadata, multipart, versioning data for index pool. I thought that most of usage is object metadata. so I checked balance state of object metadata between osds of index pool. it's balanced well. (I know it though OMAP stat). Which part can make unbalance between index pool disks? (you can check osd.27, osd.29 which are bigger than other osds) ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 24 ssd 3.49309 1.00000 3.5 TiB 1.8 TiB 399 MiB 1.2 TiB 581 GiB 1.7 TiB 51.72 0.85 71 up 25 ssd 3.49309 1.00000 3.5 TiB 2.2 TiB 179 MiB 1.4 TiB 836 GiB 1.3 TiB 62.22 1.02 71 up 26 ssd 3.49309 1.00000 3.5 TiB 2.0 TiB 180 MiB 1.2 TiB 752 GiB 1.5 TiB 56.55 0.93 72 up *27 ssd 3.49309 1.00000 3.5 TiB 2.7 TiB 399 MiB 1.3 TiB 1.4 TiB 815 GiB 77.22 1.27 70 up* 28 ssd 3.49309 1.00000 3.5 TiB 1.5 TiB 179 MiB 1.2 TiB 321 GiB 2.0 TiB 43.57 0.71 72 up *29 ssd 3.49309 1.00000 3.5 TiB 2.8 TiB 179 MiB 1.4 TiB 1.4 TiB 748 GiB 79.08 1.30 73 up* 30 ssd 3.49309 1.00000 3.5 TiB 1.7 TiB 179 MiB 1.4 TiB 342 GiB 1.8 TiB 49.37 0.81 75 up 31 ssd 3.49309 1.00000 3.5 TiB 2.5 TiB 179 MiB 1.4 TiB 1.2 TiB 969 GiB 72.90 1.20 69 up 32 ssd 3.49309 1.00000 3.5 TiB 2.4 TiB 179 MiB 1.3 TiB 1.1 TiB 1.1 TiB 67.46 1.11 66 up 33 ssd 3.49309 1.00000 3.5 TiB 2.1 TiB 179 MiB 1.2 TiB 1015 GiB 1.3 TiB 61.40 1.01 68 up .... omission .... TOTAL 63 TiB 38 TiB 3.8 GiB 23 TiB 16 TiB 25 TiB 60.96 MIN/MAX VAR: 0.68/1.30 STDDEV: 9.98

2 1

MDS crashing with "Assertion `px != 0' failed"
by Simon Campion 15 Apr '25

15 Apr '25

We're trying to determine the root cause of a CephFS outage. We have three MDS ranks with active-standby. During the outage, several MDSs crashed. The timeline of the crashes was: 2025-04-13T14:19:45 mds.r-cephfs-hdd-f on node06.internal 2025-04-13T14:38:35 mds.r-cephfs-hdd-a on node02.internal 2025-04-13T14:38:37 mds.r-cephfs-hdd-b on node07.internal 2025-04-13T14:38:38 mds.r-cephfs-hdd-d on node05.internal 2025-04-13T14:48:52 mds.r-cephfs-hdd-e on node08.internal 2025-04-13T14:54:12 mds.r-cephfs-hdd-f on node06.internal At around 3pm, the file system recovered by itself. For all six crashes, the MDS logged this as the reason of the crash: ceph-mds: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.1/rpm/el9/BUILD/ceph-19.2.1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr<T>::operator->() const [with T = MDRequestImpl]: Assertion `px != 0' failed. For the earliest crash, here's the `ceph crash info` output: bash-5.1$ ceph crash info 2025-04-13T14:19:45.645607Z_7e6475e0-9a22-4e3f-a282-0ab02a7c972c { "backtrace": [ "/lib64/libc.so.6(+0x3e930) [0x7f4990b08930]", "/lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc]", "raise()", "abort()", "/lib64/libc.so.6(+0x2875b) [0x7f4990af275b]", "/lib64/libc.so.6(+0x375c6) [0x7f4990b015c6]", "ceph-mds(+0x1c2829) [0x559da9485829]", "ceph-mds(+0x3191a7) [0x559da95dc1a7]", "(MDSContext::complete(int)+0x5c) [0x559da971117c]", "(Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d]", "/lib64/libc.so.6(+0x8a292) [0x7f4990b54292]", "/lib64/libc.so.6(+0x10f300) [0x7f4990bd9300]" ], "ceph_version": "19.2.1", "crash_id": "2025-04-13T14:19:45.645607Z_7e6475e0-9a22-4e3f-a282-0ab02a7c972c", "entity_name": "mds.r-cephfs-hdd-f", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mds", "stack_sig": "8975f8e99bd02b53c8d37ce7cc9e85dc5d4898104a0949d0829819a753123f18", "timestamp": "2025-04-13T14:19:45.645607Z", "utsname_hostname": "node06.internal", "utsname_machine": "x86_64", "utsname_release": "6.6.83-flatcar", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Mon Mar 17 16:07:40 -00 2025" } And here's an excerpt of the logs, starting with the aforementioned assertion failure: ceph-mds: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.1/rpm/el9/BUILD/ceph-19.2.1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr<T>::operator->() const [with T = MDRequestImpl]: Assertion `px != 0' failed. *** Caught signal (Aborted) ** in thread 7f4984f30640 thread_name: ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable) 1: /lib64/libc.so.6(+0x3e930) [0x7f4990b08930] 2: /lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc] 3: raise() 4: abort() 5: /lib64/libc.so.6(+0x2875b) [0x7f4990af275b] 6: /lib64/libc.so.6(+0x375c6) [0x7f4990b015c6] 7: ceph-mds(+0x1c2829) [0x559da9485829] 8: ceph-mds(+0x3191a7) [0x559da95dc1a7] 9: (MDSContext::complete(int)+0x5c) [0x559da971117c] 10: (Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d] 11: /lib64/libc.so.6(+0x8a292) [0x7f4990b54292] 12: /lib64/libc.so.6(+0x10f300) [0x7f4990bd9300] debug 2025-04-13T14:19:45.645+0000 7f4984f30640 -1 *** Caught signal (Aborted) ** in thread 7f4984f30640 thread_name: ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable) 1: /lib64/libc.so.6(+0x3e930) [0x7f4990b08930] 2: /lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc] 3: raise() 4: abort() 5: /lib64/libc.so.6(+0x2875b) [0x7f4990af275b] 6: /lib64/libc.so.6(+0x375c6) [0x7f4990b015c6] 7: ceph-mds(+0x1c2829) [0x559da9485829] 8: ceph-mds(+0x3191a7) [0x559da95dc1a7] 9: (MDSContext::complete(int)+0x5c) [0x559da971117c] 10: (Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d] 11: /lib64/libc.so.6(+0x8a292) [0x7f4990b54292] 12: /lib64/libc.so.6(+0x10f300) [0x7f4990bd9300] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- debug -9999> 2025-04-13T14:17:22.696+0000 7f4986f34640 5 mds.2.log trim already expired LogSegment(79533796/0x1a5ea43a405 events=117) debug -9998> 2025-04-13T14:17:22.696+0000 7f4986f34640 5 mds.2.log trim already expired LogSegment(79533913/0x1a5ea8350c6 events=75) debug -9997> 2025-04-13T14:17:22.696+0000 7f4986f34640 5 mds.2.log trim already expired LogSegment(79533988/0x1a5eac5c3df events=95) ... Do you have ideas what could cause these crashes or how we could troubleshoot further? We're happy to provide more information if that'd help. Simon

2 1

constant increase in osdmap epoch
by Frank Schilder 15 Apr '25

15 Apr '25

Hi all, we observe a problem that has been reported before, but I can't find a resolution. This is related to an earlier thread "failed to load OSD map for epoch 2898146, got 0 bytes" (https://www.spinics.net/lists/ceph-users/msg84485.html) We run an octopus latest cluster and observe a constant increase in osdmap epoch every few seconds. There is no change in the contents between two successive epochs: # diff map.3075085.txt map.3075086.txt 1c1 < epoch 3075085 --- > epoch 3075086 4c4 < modified 2024-11-18T15:38:45.512100+0100 --- > modified 2024-11-18T15:38:47.858092+0100 This is exactly what others reported too, for example, "steady increasing of osd map epoch since octopus" (https://www.spinics.net/lists/ceph-users/msg69443.html) Its a real problem since it dramatically shortens the time window an OSD can be down before its latest OSD map is purged from the cluster. This, in turn, leads to serious follow-up problems with OSD restart as reported in the thread I'm referring to at the beginning. Related to that I also see the mgrs increasing the pgmap version constantly every 2 seconds. However, I believe this is intentional. I don't see this redundant pgp_num_actual setting by the mgrs reported here: https://tracker.ceph.com/issues/51433 . I can't find a resolution anywhere. Any help would be very much appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 7

Who creates /etc/lograte.d/cephadm in 19.2.2?
by Robert Sander 15 Apr '25

15 Apr '25

Hi, I have a cephadm managed 19.2.2 cluster where logrotate is unable to run because of a configuration conflict. The package ceph-common is installed for all the command line tools and provides /etc/logrotate.d/ceph-common to rotate all logfiles in /var/log. Some process now regularily creates /etc/logrotate.d/cephadm to rotate only /var/log/cephadm.log. And this creates a conflict for logrotate which refuses to run in the end. I tried to remove /etc/logrotate.d/cephadm with rm just to see it reappear after a few minutes. The package cephadm is not installed. I have not seen an entry in the MGR or MON log mentioning the creation of this file. Which process is doing this? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 2

Squid: Ceph mgr shows OSD_Unreachable after setting new host address
by Mevludin Blazevic 14 Apr '25

14 Apr '25

Hi all, after updating host addr via "ceph orch host set-addr x y", the alert OSD_UNREACHABLE appears, althouth the OSDs are accessible. The issue persists after I delete OSDs and add them again after zapping the disc. How to resolve this? Best, Mejdi

1 0

Ceph Day Silicon Valley 2025 - Call for Proposals Now Open!
by Neha Ojha 14 Apr '25

14 Apr '25

Dear Ceph Community, We are very excited to announce that Ceph Day is returning to Silicon Valley after seven years! On March 25, 2025, we will be hosting Ceph Day at IBM Almaden Research Center in San Jose, CA. We are now accepting talk proposals! For more details, visit https://ceph.io/en/community/events/2025/ceph-days-almaden/. We look forward to bringing the Ceph community together once again! Cheers, Neha

2 2

UI for Object Gateway S3 ?
by Anthony Fecarotta 14 Apr '25

14 Apr '25

Hi everyone, Thanks to a member of this wonderful mailing list, I successfully, and easily deployed the RADOS Gateway S3 API on each of the nodes in one of our five node Proxmox clusters. I can see managing this at scale from the terminal being a little tricky, but more so inconvenient. Are there any options (closed or open-source) that offer a way GUI or TUI to manage the S3 Buckets? I was able to find this GitHub Gist ( https://gist.github.com/e02c6081b515881aeb97b26016d0c658.git) that lets Minio act as a GUI for "Ceph backed S3," but it hasn't been updated in nine months and it seems to be intended strictly for Kubernetes. Any of your recommendations are appreciated. I started building a TUI with GO, to solve this problem, but that will take some time to finish 🙃 Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony(a)linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>

3 5

Re: Cannot reinstate ceph fs mirror because i destroyed the ceph fs mirror peer/ target server
by Eugen Block 14 Apr '25

14 Apr '25

Please don't drop the list from your responses. Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that. Zitat von Jan Zeinstra <jan(a)delectat.nl>: > Thanks for the response, > I got: > ceph config-key ls |grep "peer/prodfs" > "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", > "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5", > > Since there is no mirroring at all going on, I saw no harm in deleting them > both. > ceph config-key rm > "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" > key deleted > ceph config-key rm > "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" > key deleted > > And sure enough: > ceph config-key ls |grep "peer/prodfs" > resulted in nothing found > > However: > ceph fs snapshot mirror peer_list proofs > gave > {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": > "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}} > > Also after redeploying the daemon. > > Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing. > > Do you have more suggestions, where to look ? > > (Mirroring is still stuck in trying to find the now nonexistent remote > cluster: > cephadm logs --name cephfs-mirror.s1mon.lvlkwp > > Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp > for d0ea284a-8a16-11ee-9232-5934f0f00ec2. > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 > (ceph:ceph) > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 > (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process > cephfs-mirror, pid 2 > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty > --pid-file > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register > cephfs-mirror.23174196 metadata > {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 > (e7ad5345525c7aa95470c26863873b581076945d) reef > (stable),ceph_version_short=18.2.4,container_hostna> > Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils > connect: error connecting to bk-site: (2) No such file or directory > Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: > error connecting to remote cluster: (2) No such file or directory > Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS > SRV with service name: ceph-mon > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors > to contact > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: > (2) No such file or directory > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: > error connecting to remote cluster: (2) No such file or directory > ) > > Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock(a)nde.ag>: > >> Hi, >> >> I would expect that you have a similar config-key entry: >> >> ceph config-key ls |grep "peer/cephfs" >> "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56", >> >> Maybe removing that peer would already suffice? >> >> >> Zitat von Jan Zeinstra <jan(a)delectat.nl>: >> >> > Hi, >> > This is my first post to the forum and I don't know if it's appropriate, >> > but I'd like to express my gratitude to all people working hard on ceph >> > because I think it's a fantastic piece of software. >> > >> > The problem I'm having is caused by me; we had a well working ceph fs >> > mirror solution; let's call it source cluster A, and target cluster B. >> > Source cluster A is a modest cluster consisting of 6 instances, 3 OSD >> > instances, and 3 mon instances. The OSD instances all have 3 disks >> (HDD's) >> > and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B >> is >> > a single node system having 3 OSD daemons and 3 HDD's. Both clusters run >> > ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both >> > systems are installed using cephadm. >> > I have destroyed cluster B, and have built it from the ground up (I made >> a >> > mistake in PG sizing in the original cluster) >> > Now i find i cannot create/ reinstate the mirroring between 2 ceph fs >> > filesystems, and i suspect there is a peer left behind in the filesystem >> of >> > the source, pointing to the now non-existent target cluster. >> > When i do 'ceph fs snapshot mirror peer_list prodfs', i get: >> > '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": >> > "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' >> > When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs >> > f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to >> > remove peeraccess denied: does your client key have mgr caps? See >> > http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication >> ', >> > but the logging of the daemon points to the more likely reason of >> failure: >> > ---- >> > Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp >> > for d0ea284a-8a16-11ee-9232-5934f0f00ec2. >> > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 >> > (ceph:ceph) >> > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 >> > (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process >> > cephfs-mirror, pid 2 >> > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty >> > --pid-file >> > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register >> > cephfs-mirror.22849497 metadata >> > {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 >> > (e7ad5345525c7a> >> > Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >> > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> > remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0] >> > Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >> > 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server >> > allowed_methods [2] but i only support [2,1] >> > Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >> > 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to >> bk-site: >> > (13) Permission denied >> > Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils >> connect: >> > error connecting to bk-site: (13) Permission denied >> > Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >> > 7f57d81e0640 -1 >> > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> > error connecting to remote cl> >> > Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >> > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> > error connecting to remote cluster: (13) Permission denied >> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > ---- >> > Is there any chance I can get the mirroring daemon to forget about the >> > cluster I lost ? >> > >> > Best regards, Jan Zeinstra >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

2 7

OSDs ignore memory limit
by Jonas Schwab 14 Apr '25

14 Apr '25

Hello everyone, I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits? See e.g. the following: $ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d Best regards, Jonas -- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat(a)listserv.dfn.de Email: jonas.schwab(a)uni-wuerzburg.de Tel: +49 931 31-84460

7 9

nodes with high density of OSDs
by Alex from North 13 Apr '25

13 Apr '25

Hello everybody! I have a 4 nodes with 112 OSDs each and 18.2.4. OSD consist of db on SSD and data on HDD For some reason, when I reboot node, not all OSDs get up because some VG or LV are not active. To make it alive again I manually do vgchange -ay $VG_NAME or lvchange -ay $LV_NAME. I suspect it is linked to high amount of vg/lv but cannot find an answer. Maybe you can gimme a hint how to struggle it over?

9 22

FS not mount after update to quincy
by Iban Cabrillo 12 Apr '25

12 Apr '25

Hi guys Good morning, Since I performed the update to Quincy, I've noticed a problem that wasn't present with Octopus. Currently, our Ceph cluster exports a filesystem to certain nodes, which we use as a backup repository. The machines that mount this FS are currently running Ubuntu 24 with Ceph Squid as the client version. zeus22:~ # ls -la /cephvmsfs/ total 225986576 drwxrwxrwx 13 root root 17 Apr 4 13:10 . drwxr-xr-x 1 root root 286 Mar 19 13:27 .. -rw-r--r-- 1 root root 124998647808 Apr 4 13:18 arcceal9.img drwxrwxrwx 2 nobody nogroup 2 Jul 12 2018 backup drwxr-xr-x 2 nobody nogroup 1 Oct 18 2017 Default -rw-r--r-- 1 root root 214cat /etc74836480 Mar 26 18:11 ns1.img drwxr-xr-x 2 root root 1 Aug 29 2024 OnlyOffice Before the update, these nodes mounted the FS correctly (even cluster in octopus and clients in squid), and the nodes that haven't been restarted are still accessing it. One of these machines has been reinstalled, and using the same configuration as the nodes that are still mounting this FS, it is unable to mount, giving errors such as: `mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized` 10.10.3.1:3300,10.10.3.2:3300,10.10.3.3:3300:/ /cephvmsfs ceph name=cephvmsfs,secretfile=/etc/ceph/cephvmsfs.secret,noatime,mds_namespace=cephvmsfs,_netdev 0 0 If I change the port to use 6789 (v1) mount error 110 = Connection timed out ceph cluster is healty and msd are up cephmon01:~ # ceph -s cluster: id: 6f5a65a7-yyy-zzzz-xxxx-428608941dd1 health: HEALTH_OK services: mon: 3 daemons, quorum cephmon01,cephmon03,cephmon02 (age 2d) mgr: cephmon02(active, since 7d), standbys: cephmon01, cephmon03 mds: 1/1 daemons up, 1 standby osd: 231 osds: 231 up (since 7d), 231 in (since 9d) rgw: 2 daemons active (2 hosts, 1 zones) Cephmons are available from clients in both ports: zeus:~ # telnet cephmon02 6789 Trying 10.10.3.2... Connected to cephmon02. Escape character is '^]'. ceph v027�� Ҭ zeus01:~ # telnet cephmon02 3300 Trying 10.10.3.2... Connected to cephmon02. Escape character is '^]'. ceph v2 Any advise is welcomed, regards I -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Exter… ==========================================================================================

3 4

Cephadm flooding /var/log/ceph/cephadm.log
by Alex 11 Apr '25

11 Apr '25

Hi everyone. My company has paid Ceph support. The tech is saying: "...cephadm package of ceph 5 is having bug. So It generate debug logs even its set for "info" logs..." I have two clusters running, one Ceph 5 and the other Ceph 6 (Quincy). Both of them are sending "DEBUG ---------" messages to cephadm.log. I checked everywhere I can think of and all the logs are set to INFO. Is there really a bug? I can't find any mention of one anywhere on the internet, besides, it seems that if there really is a bug it hasn't been fixed in version 6 either. I tried setting " mgr/cephadm/log_to_cluster_level" and "mgr/cephadm/log_level" to WARN and noting changed. /etc/ceph/ceph.conf has no mention of log level. Any help would be appreciated, Thanks!

5 27

Experience with 100G Ceph in Proxmox
by Giovanna Ratini 11 Apr '25

11 Apr '25

Hello everyone, We are running Ceph in Proxmox with a 10G network. Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link <https://forum.proxmox.com/threads/slow-performance-on-ceph-per-vm.151223/#p…> For this reason, we are considering purchasing a 100G switch for our servers. This raises some questions: Should I still use separate networks for VMs and Ceph with 100G? I have read that running Ceph on bridged connections is not recommended. Does anyone have experience with 100G Ceph in Proxmox? Is upgrading to 100G a good idea, or will I have 60G sitting idle? Thanks in advance! Gio

13 32

v19.2.2 Squid released
by Yuri Weinstein 11 Apr '25

11 Apr '25

We're happy to announce the 2nd backport release in the Squid series. https://ceph.io/en/news/blog/2025/v19-2-2-squid-released/ Notable Changes --------------- - This hotfix release resolves an RGW data loss bug when CopyObject is used to copy an object onto itself. S3 clients typically do this when they want to change the metadata of an existing object. Due to a regression caused by an earlier fix for https://tracker.ceph.com/issues/66286, any tail objects associated with such objects are erroneously marked for garbage collection. RGW deployments on Squid are encouraged to upgrade as soon as possible to minimize the damage. The experimental rgw-gap-list tool can help to identify damaged objects. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.2.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 0eceb0defba60152a8182f7bd87d164b639885b8

4 3

Cannot reinstate ceph fs mirror because i destroyed the ceph fs mirror peer/ target server
by Jan Zeinstra 11 Apr '25

11 Apr '25

Hi, This is my first post to the forum and I don't know if it's appropriate, but I'd like to express my gratitude to all people working hard on ceph because I think it's a fantastic piece of software. The problem I'm having is caused by me; we had a well working ceph fs mirror solution; let's call it source cluster A, and target cluster B. Source cluster A is a modest cluster consisting of 6 instances, 3 OSD instances, and 3 mon instances. The OSD instances all have 3 disks (HDD's) and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B is a single node system having 3 OSD daemons and 3 HDD's. Both clusters run ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both systems are installed using cephadm. I have destroyed cluster B, and have built it from the ground up (I made a mistake in PG sizing in the original cluster) Now i find i cannot create/ reinstate the mirroring between 2 ceph fs filesystems, and i suspect there is a peer left behind in the filesystem of the source, pointing to the now non-existent target cluster. When i do 'ceph fs snapshot mirror peer_list prodfs', i get: '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to remove peeraccess denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication', but the logging of the daemon points to the more likely reason of failure: ---- Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 (ceph:ceph) Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty --pid-file Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register cephfs-mirror.22849497 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7a> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cl> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (13) Permission denied Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 ---- Is there any chance I can get the mirroring daemon to forget about the cluster I lost ? Best regards, Jan Zeinstra

2 1

endless remapping after increasing number of PG in a pool
by Michel Jouvin 11 Apr '25

11 Apr '25

Hi, We are observing a new strange behaviour on our production cluster : we increased the number of PG (from 256 to 2048) in a (EC) pool after a warning that there was a very high number of objects per pool (the pool has 52M objects). Background: this happens in the cluster that had a strange problem last week, discussed in the thread "Production cluster in bad shape after several OSD crashes". The PG increase was done after the cluster returned to a normal state. The increase of the number of PG resulted in 20% misplaced objects and ~160 PG remapped (over 256). As there is no much user activity on this cluster (except on this pool) these days, we decided to set the mclock profile to high_recovery_ops. We also disabled autoscaler on this pool (it was enabled and it is not clear why we add the warning with autoscaler enabled). The pool was created with --bulk. The remapping went steadily during 2-3 days (as much as we can tell) but when reaching the end (between 0.5% and 1% of misplaced objects, ~10 PG remapped) it readds remapped PG (that can be seen with 'ceph pg dump_stuck'), all belonging to the pool affected by the increase. This already happened 3-4 times and it is very unclear why. There was no specific problem reported on the cluster that may explain this (no OSD down). I was wondering if the balancer may be responsible for this but I don't have the feeling it is the case: first the balancer doesn't report about doing something (but I may miss the history), second the balancer would probably affect PG from different pools. (there are 50 ppols in the cluster). There are 2 warnings that may or may not be related: - All mon have their data size > 15 GB (mon_data_size_warn). Currently 16 GB. It happened progressively during the remapping on all 3 mon, I guess it is due to the operation in progress and is harmless. Do you confirm? - Since yesterday we have 4 PG that have not been deep scrubbed in time, belonging to different pools. Again, I tend to correlate this to the remapping in progress putting too much load (or other constraints), as there are a lot of deep scrubs per day. The current age distribution of deep scrubs is: 4 "2025-03-19 21 "2025-03-20 46 "2025-03-21 35 "2025-03-22 81 "2025-03-23 597 "2025-03-24 1446 "2025-03-25 2234 "2025-03-26 2256 "2025-03-27 1625 "2025-03-28 1980 "2025-03-29 2993 "2025-03-30 3871 "2025-03-31 1113 "2025-04-01 Should we worry about the situation? If yes, what would you advise to look at or do? To clear the problem last week, we had to restart all OSD but we didn't restart mon. Do they play a role in deciding the remapping plan? Is restarting them something that may help? As usual, thanks in advance for any help/hint. Best regards, Michel

2 4

Image Live-Migration does not respond to OpenStack Glance images
by Yuta Kambe (Fujitsu) 10 Apr '25

10 Apr '25

Hi everyone. I am trying Image Live-Migration but it is not working well and I would like some advice. https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ I use Ceph as a backend for OpenStack Glance. I tried to migrate the Pool of Ceph used in Glance to the new Pool. Source Pool: - images_meta : metadata pool, Replication - images_data : data pool, Erasure Code Target Pool: - images_meta: metadata pool, Replication (Same as source Pool) - images_data_hdd: data pool, Erasure Code The following command I executed, but did not return a response. rbd migration prepare images_meta/image_name images_meta/image_name --data-pool images_data_hdd I checked the logs in /var/log/messages and /var/log/ceph, but no useful information was available. I would like some advice on this. - Are there any other logs I should check? - Is there a case where the rbd migration command cannot be executed? The following is supplemental information. - ceph version 17.2.8 - The migration of the OpenStack Nova image was successful with the same Pool configuration and command. - I don't know if it is related, but there is a snapshot in the image of Glance, and unprotect of the snapshot is also unresponsive. rbd snap unprotect images_meta/image_name@snap

2 1

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

NIH Datasets
by Linas Vepstas 09 Apr '25

09 Apr '25

OK what you will read below might sound insane but I am obliged to ask. There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere. I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know. While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list. OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment. Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope? Am I misunderstanding something about IPFS that merits taking a second look at it? Is there any other way of getting scalable reliable "decentralized" internet-wide storage? I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work? -- Linas -- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us.

9 17

Diskprediction_local mgr module removal - Call for feedback
by Yaarit Hatuka 09 Apr '25

09 Apr '25

Hi everyone, On today's Ceph Steering Committee call we discussed the idea of removing the diskprediction_local mgr module, as the current prediction model is obsolete and not maintained. We would like to gather feedback from the community about the usage of this module, and find out if anyone is interested in maintaining it. Thanks, Yaarit

7 12

RBD Block alignment 16k for Databases
by filip Mutterer 09 Apr '25

09 Apr '25

Hi! Is Block alignment for Databases a thing with RBD? I heard something with 16K alignment ... If yes how can be checked what alignment a RBD got. And yes I need to run my Database in Ceph as I have no other storage with redundandency. Hints for optimizing a single Block device for a Database are very welcome. Greetings filip

2 1

Use cephfs when cephfs-data-scan cleanup is not finished
by Jiří Župka 09 Apr '25

09 Apr '25

Dear conference, I would like to ask if is secure use cephfs when cephfs-data-scan cleanup is not finished yet? Best regards, Jiri

2 1

File immutability on CephFS ?
by Fabien Sirjean 09 Apr '25

09 Apr '25

Dear all, If I am to believe this issue [1], it seems that it is still not possible to make files immutable (chattr +i) in cephfs. Do you have any update on this matter ? Thanks a lot for all the good work! Cheers, Fabien [1] : https://tracker.ceph.com/issues/10679

2 1

squid 19.2.2 hotfix QE validation status
by Yuri Weinstein 08 Apr '25

08 Apr '25

This is a hotfix release to address only one issue: https://github.com/ceph/ceph/pull/62711 and has limited testing. Details of this release are summarized here: https://tracker.ceph.com/issues/70822#note-1 Release Notes - TBD LRC upgrade - excluded from this release Gibba upgrade - excluded from this release smoke - Adam E, Casey approved? rgw - Adam E, Casey approved?

2 1

irc/slack/discord bridge failing
by Alvaro Soto 08 Apr '25

08 Apr '25

Hi all, Our IRC/Slack/Discord bridge started failing since ~2 days ago. Slack deprecated the use of legacy bots[1], so at the moment, IRC/Discord is the only bridge working. [1] ~~~ time="2025-04-08T18:09:30Z" level=error msg="Could not retrieve channels: slack.SlackErrorResponse{Err:"legacy_custom_bots_deprecated", ResponseMetadata:slack.ResponseMetadata{Cursor:"", Messages:[]string(nil), Warnings:[]string(nil)}}" prefix=slack ~~~ I'll keep you posted. Cheers! -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 0

Re: Ceph squid fresh install
by quaglio＠bol.com.br 08 Apr '25

08 Apr '25

1 0

Re: Ceph squid fresh install
by quaglio＠bol.com.br 08 Apr '25

08 Apr '25

1 0

Re: Ceph squid fresh install
by quaglio＠bol.com.br 08 Apr '25

08 Apr '25

2 3

Data Loss Bug Identified on Squid
by Adam Emerson 07 Apr '25

07 Apr '25

https://tracker.ceph.com/issues/70746 tracks a data loss bug on squid when CopyObject is used to copy an object onto itself. S3 clients typically do this when they want to change the metadata of an existing object. Due to a regression caused by an earlier fix for https://tracker.ceph.com/issues/66286, any tail objects associated with such objects are erroneously marked for garbage collection. This regression was introduced with the initial 19.2.0 squid release and is still present in 19.2.1. There is no workaround to prevent clients from triggering this bug, so we're preparing 19.2.2 as a hotfix release to resolve this as soon as possible.

1 0

ceph.log using seconds since epoch instead of date/time stamp
by Stillwell, Bryan 07 Apr '25

07 Apr '25

Does anyone know why log entries in ceph.log and ceph.audit.log would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but ceph-mon.*.log and ceph-osd.*.log use date/time stamps? Example: # ceph.log 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok # ceph-mon.log 2025-02-21T00:00:01.814-0500 7fe33378e700 0 mon.XXXXXX@0(leader) e2 handle_command mon_command({"prefix": "osd dump"} v 0) v1 This doesn't seem to be the case on my home cluster running Squid 19.2.1/Ubuntu 24.04. I found this Proxmox forum post that seems to imply that could be a distro setting (same Ceph version, but bookworm is affected, and bullseye is not): https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.… Thanks, Bryan

3 4

Re: Cephadm upgrade from 16.2.15 -> 17.2.0
by Eugen Block 07 Apr '25

07 Apr '25

It would help if you only pasted the relevant parts. Anyway, these two sections stand out: ---snip--- Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again later Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed to construct class in 'cephadm' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade = CephadmUpgrade(self) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: return cls(**c) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: TypeError: __init__() got an unexpected keyword argument 'daemon_types' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('cephadm') Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed to construct class in 'snap_schedule' Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.client = SnapSchedClient(self) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 158, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: with self.get_schedule_db(fs_name) as conn_mgr: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 192, in get_schedule_db Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: db.executescript(dump) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: sqlite3.OperationalError: table schedules already exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('snap_schedule') ---snip--- Your cluster seems to be in an error state (ceph -s) because of an unknown PG. It's recommended to have a healthy cluster before attemping an upgrade. It's possible that these errors come from the not upgraded MGR, I'm not sure. Since the upgrade was only successful for two MGRs, I am thinking about downgrading both MGRs back to 16.2.15, then retry an upgrade to a newer version, either 17.2.8 or 18.2.4. I haven't checked the snap_schedule error yet, though. Maybe someone else knows that already. Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: > This is a small three node cluster. Here’s logs from all three nodes > during the failover. The current manager was cn02 and it failed over > to cn03. > > cn01 > —— > Apr 05 20:33:35 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:35.823+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v821: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.170+0000 7f2aaec00700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.823+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v822: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:33:39 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:39] "GET /metrics > HTTP/1.1" 200 267241 "" "Prometheus/2.33.4" > Apr 05 20:33:39 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:39.825+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v823: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 97 KiB/s wr, 12 op/s > Apr 05 20:33:41 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:41.826+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v824: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 85 KiB/s wr, 10 op/s > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.170+0000 7f2aaec00700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:42] "GET /metrics > HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics > HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:43.827+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v825: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 511 B/s rd, 133 KiB/s wr, 18 op/s > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 -1 mgr > handle_mgr_map I was active but no longer am > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn e: > '/usr/bin/ceph-mgr' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 0: > '/usr/bin/ceph-mgr' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 1: '-n' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 2: > 'mgr.cn01.tjmtph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 3: '-f' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 4: > '--setuser' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 5: > 'ceph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 6: > '--setgroup' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 7: > 'ceph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 8: > '--default-log-to-file=false' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 9: > '--default-log-to-stderr=true' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 10: > '--default-log-stderr-prefix=debug ' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn > respawning with exe /usr/bin/ceph-mgr > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn > exe_path /proc/self/exe > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ignoring --setuser ceph since I am not root > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ignoring --setgroup ceph since I am not root > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 ceph version > 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), > process ceph-mgr, pid 2 > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 pidfile_write: > ignore empty --pid-file > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.477+0000 7fb814447000 1 mgr[py] Loading > python module 'alerts' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.612+0000 7fb814447000 -1 mgr[py] Module > alerts has missing NOTIFY_TYPES member > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.612+0000 7fb814447000 1 mgr[py] Loading > python module 'balancer' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.726+0000 7fb814447000 -1 mgr[py] Module > balancer has missing NOTIFY_TYPES member > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.726+0000 7fb814447000 1 mgr[py] Loading > python module 'cephadm' > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.417+0000 7fb814447000 1 mgr[py] Loading > python module 'crash' > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.540+0000 7fb814447000 -1 mgr[py] Module > crash has missing NOTIFY_TYPES member > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.540+0000 7fb814447000 1 mgr[py] Loading > python module 'dashboard' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.210+0000 7fb814447000 1 mgr[py] Loading > python module 'devicehealth' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.331+0000 7fb814447000 -1 mgr[py] Module > devicehealth has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.331+0000 7fb814447000 1 mgr[py] Loading > python module 'diskprediction_local' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.653+0000 7fb814447000 -1 mgr[py] Module > diskprediction_local has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.654+0000 7fb814447000 1 mgr[py] Loading > python module 'influx' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.769+0000 7fb814447000 -1 mgr[py] Module > influx has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.769+0000 7fb814447000 1 mgr[py] Loading > python module 'insights' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.884+0000 7fb814447000 1 mgr[py] Loading > python module 'iostat' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.082+0000 7fb814447000 -1 mgr[py] Module > iostat has missing NOTIFY_TYPES member > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.082+0000 7fb814447000 1 mgr[py] Loading > python module 'k8sevents' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.679+0000 7fb814447000 1 mgr[py] Loading > python module 'localpool' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.796+0000 7fb814447000 1 mgr[py] Loading > python module 'mds_autoscaler' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.983+0000 7fb814447000 1 mgr[py] Loading > python module 'mirroring' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.118+0000 7fb814447000 1 mgr[py] Loading > python module 'nfs' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.315+0000 7fb814447000 -1 mgr[py] Module > nfs has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.315+0000 7fb814447000 1 mgr[py] Loading > python module 'orchestrator' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.607+0000 7fb814447000 -1 mgr[py] Module > orchestrator has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.607+0000 7fb814447000 1 mgr[py] Loading > python module 'osd_perf_query' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.723+0000 7fb814447000 -1 mgr[py] Module > osd_perf_query has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.723+0000 7fb814447000 1 mgr[py] Loading > python module 'osd_support' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.837+0000 7fb814447000 -1 mgr[py] Module > osd_support has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.837+0000 7fb814447000 1 mgr[py] Loading > python module 'pg_autoscaler' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.970+0000 7fb814447000 -1 mgr[py] Module > pg_autoscaler has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.970+0000 7fb814447000 1 mgr[py] Loading > python module 'progress' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.082+0000 7fb814447000 -1 mgr[py] Module > progress has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.082+0000 7fb814447000 1 mgr[py] Loading > python module 'prometheus' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.617+0000 7fb814447000 -1 mgr[py] Module > prometheus has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.617+0000 7fb814447000 1 mgr[py] Loading > python module 'rbd_support' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.761+0000 7fb814447000 -1 mgr[py] Module > rbd_support has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.761+0000 7fb814447000 1 mgr[py] Loading > python module 'restful' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.023+0000 7fb814447000 1 mgr[py] Loading > python module 'rook' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.788+0000 7fb814447000 -1 mgr[py] Module > rook has missing NOTIFY_TYPES member > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.788+0000 7fb814447000 1 mgr[py] Loading > python module 'selftest' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.912+0000 7fb814447000 -1 mgr[py] Module > selftest has missing NOTIFY_TYPES member > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.912+0000 7fb814447000 1 mgr[py] Loading > python module 'snap_schedule' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.038+0000 7fb814447000 -1 mgr[py] Module > snap_schedule has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.038+0000 7fb814447000 1 mgr[py] Loading > python module 'stats' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.155+0000 7fb814447000 1 mgr[py] Loading > python module 'status' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.280+0000 7fb814447000 -1 mgr[py] Module > status has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.280+0000 7fb814447000 1 mgr[py] Loading > python module 'telegraf' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.398+0000 7fb814447000 -1 mgr[py] Module > telegraf has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.398+0000 7fb814447000 1 mgr[py] Loading > python module 'telemetry' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.594+0000 7fb814447000 -1 mgr[py] Module > telemetry has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.594+0000 7fb814447000 1 mgr[py] Loading > python module 'test_orchestrator' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.771+0000 7fb814447000 -1 mgr[py] Module > test_orchestrator has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.771+0000 7fb814447000 1 mgr[py] Loading > python module 'volumes' > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.225+0000 7fb814447000 -1 mgr[py] Module > volumes has missing NOTIFY_TYPES member > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.225+0000 7fb814447000 1 mgr[py] Loading > python module 'zabbix' > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.342+0000 7fb814447000 -1 mgr[py] Module > zabbix has missing NOTIFY_TYPES member > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.344+0000 7fb807c00700 0 > ms_deliver_dispatch: unhandled message 0x555c78fe1080 mon_map magic: > 0 v1 from mon.0 v2:192.168.47.71:3300/0 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.344+0000 7fb808600700 0 [dashboard DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load > Constructed class from module: dashboard > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 0 [prometheus DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load > Constructed class from module: prometheus > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb6abe00700 0 [dashboard INFO > root] server: ssl=yes host=:: port=8443 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > root] server_addr: :: server_port: 9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > root] Starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Bus STARTING > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTING > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO > root] Configured CherryPy, starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO > root] Starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > CherryPy Checker: > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > The Application mounted at '' has an empty config. > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Serving on http://:::9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Serving on > http://:::9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Bus STARTED > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTED > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.451+0000 7fb6b4a00700 0 [prometheus INFO > root] Engine started. > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.489+0000 7fb6abe00700 0 [dashboard INFO > root] Engine started... > Apr 05 20:33:59 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:59] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:59 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:59.603+0000 7fb76c600700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:59] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:02 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:02] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:02 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:02.476+0000 7fb763a00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:03 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:03.061+0000 7fb75ae00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:09 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:09] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:09 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:09.602+0000 7fb752200700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:09] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:12 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:12] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:12 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:12.476+0000 7fb749600700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:13 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:13.060+0000 7fb740a00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.493+0000 7fb714e00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.494+0000 7fb714e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52368] [GET] [303] [0.003s] [96.0B] > [0031fbdd-947f-4c6c-943d-55cf256edfba] /api/summary > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.512+0000 7fb70c200700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.513+0000 7fb70c200700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52369] [GET] [303] [0.001s] [96.0B] > [76c9a6ab-aae0-4ef9-815e-dbd0ac8f8b49] /api/feature_toggles > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.527+0000 7fb703600700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.528+0000 7fb703600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52370] [GET] [303] [0.001s] [96.0B] > [db017da3-e6f6-45fc-9d99-7b9f3b3859f2] /api/health/minimal > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.541+0000 7fb6faa00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.542+0000 7fb6faa00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52371] [GET] [303] [0.001s] [96.0B] > [564ba322-b99c-49de-a3a3-e0ed422bfae7] /api/prometheus > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.616+0000 7fb6f1e00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.617+0000 7fb6f1e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52372] [GET] [303] [0.001s] [96.0B] > [1f349e52-9a10-4fda-b13f-621785208ea9] /api/prometheus/rules > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52368] [GET] [303] [0.001s] [96.0B] > [30d749d7-c972-44f0-88c7-b70078e2385c] /api/prometheus/notifications > Apr 05 20:34:19 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:19] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:19 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:19.601+0000 7fb737e00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:19] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > > cn02 > —— > Apr 05 20:33:40 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:40 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:40.701+0000 7ff710400700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:43 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:43 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:43.287+0000 7ff707800700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.370+0000 7ff7e7c00700 1 mgr handle_mgr_map > Activating! > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.371+0000 7ff7e7c00700 1 mgr handle_mgr_map I am now > activating > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root] Stopping > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root] Stopped > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root] Stopping > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root] Stopped > engine > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > STOPPING > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.383+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > STOPPING > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.394+0000 7ff68d000700 0 [dashboard INFO root] Engine > stopped. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut > down > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.395+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut > down > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > STOPPED > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > STOPPED > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO root] Engine > stopped. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts INFO root] > Init > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 1 mgr load Constructed class from module: > alerts > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 0 [balancer DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 1 mgr load Constructed class from module: > balancer > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff74d800700 0 [alerts INFO root] > Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.401+0000 7ff67aa00700 0 [cephadm DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.402+0000 7ff710400700 0 [balancer INFO root] > Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Optimize plan > auto_2025-04-05_20:33:45 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Mode upmap, max misplaced > 0.050000 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.409+0000 7ff710400700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again > later > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.551+0000 7ff67aa00700 1 mgr load Constructed class from module: > cephadm > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 0 [crash DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 1 mgr load Constructed class from module: > crash > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 1 mgr load Constructed class from module: > dashboard > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 1 mgr load Constructed class from module: > devicehealth > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO access_control] Loading user roles DB > version=2 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO sso] Loading SSO DB > version=1 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO root] server: ssl=yes host=:: > port=8443 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.557+0000 7ff674600700 0 [devicehealth INFO root] > Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.558+0000 7ff67aa00700 0 [insights DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.559+0000 7ff67aa00700 1 mgr load Constructed class from module: > insights > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.561+0000 7ff674600700 0 [devicehealth INFO root] Check > health > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.566+0000 7ff671e00700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 0 [iostat DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 1 mgr load Constructed class from module: > iostat > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.572+0000 7ff67aa00700 0 [mirroring DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.579+0000 7ff67aa00700 1 mgr load Constructed class from module: > mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 0 [nfs DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 1 mgr load Constructed class from module: > nfs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 1 mgr load Constructed class from module: > orchestrator > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 1 mgr load Constructed class from module: > pg_autoscaler > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.625+0000 7ff662a00700 0 [pg_autoscaler INFO root] > _maybe_adjust > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 0 [progress DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 1 mgr load Constructed class from module: > progress > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.72' (ECDSA) to the list of known > hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.73' (ECDSA) to the list of known > hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 1 mgr load Constructed class from module: > prometheus > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff661600700 0 [progress INFO root] > Loading... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] server_addr: :: server_port: > 9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] Cache > enabled > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff65ee00700 0 [prometheus INFO root] starting metric collection > thread > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.640+0000 7ff67aa00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7ff7443993c8>, <progress.module.GhostEvent object at 0x7ff744399400>, <progress.module.GhostEvent object at 0x7ff744399438>, <progress.module.GhostEvent object at 0x7ff744399470>, <progress.module.GhostEvent object at 0x7ff7443994a8>, <progress.module.GhostEvent object at 0x7ff7443994e0>, <progress.module.GhostEvent object at 0x7ff744399518>, <progress.module.GhostEvent object at 0x7ff744399550>, <progress.module.GhostEvent object at 0x7ff744399588>, <progress.module.GhostEvent object at 0x7ff7443995c0>, <progress.module.GhostEvent object at 0x7ff7443995f8>, <progress.module.GhostEvent object at 0x7ff744399630>, <progress.module.GhostEvent object at 0x7ff744399668>, <progress.module.GhostEvent object at 0x7ff7443996a0>, <progress.module.GhostEvent object at 0x7ff7443996d8>, <progress.module.GhostEvent object at 0x7ff744399710>, <progress.module.GhostEvent object at 0x7ff744399748>, <progress.module.GhostEvent object at 0x7ff744399780>, <progress.module.GhostEvent object at 0x7ff7443997b8>, <progress.module.GhostEvent object at 0x7ff7443997f0>, <progress.module.GhostEvent object at 0x7ff744399828>, <progress.module.GhostEvent object at 0x7ff744399860>, <progress.module.GhostEvent object at 0x7ff744399898>, <progress.module.GhostEvent object at 0x7ff7443998d0>, <progress.module.GhostEvent object at 0x7ff744399908>, <progress.module.GhostEvent object at 0x7ff744399940>, <progress.module.GhostEvent object at 0x7ff744399978>, <progress.module.GhostEvent object at 0x7ff7443999b0>, <progress.module.GhostEvent object at 0x7ff7443999e8>, <progress.module.GhostEvent object at 0x7ff744399a20>, <progress.module.GhostEvent object at 0x7ff744399a58>, <progress.module.GhostEvent object at 0x7ff744399a90>, <progress.module.GhostEvent object at 0x7ff744399ac8>, <progress.module.GhostEvent object at 0x7ff744399b00>, <progress.module.GhostEvent object at 0x7ff744399b38>, <progress.module.GhostEvent object at 0x7ff744399b70>, <progress.module.GhostEvent object at 0x7ff744399ba8>, <progress.module.GhostEvent object at 0x7ff744399be0>, <progress.module.GhostEvent object at 0x7ff744399c18>, <progress.module.GhostEvent object at 0x7ff744399c50>, <progress.module.GhostEvent object at 0x7ff744399c88>, <progress.module.GhostEvent object at 0x7ff744399cc0>, <progress.module.GhostEvent object at 0x7ff744399cf8>, <progress.module.GhostEvent object at 0x7ff744399d30>, <progress.module.GhostEvent object at 0x7ff744399d68>, <progress.module.GhostEvent object at 0x7ff744399da0>, <progress.module.GhostEvent object at 0x7ff744399dd8>, <progress.module.GhostEvent object at 0x7ff744399e10>, <progress.module.GhostEvent object at 0x7ff744399e48>, <progress.module.GhostEvent object at 0x7ff744399e80>] historic > events > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded OSDMap, > ready. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Processing OSDMap change > 27185..27185 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff676400700 0 [dashboard INFO root] Configured CherryPy, starting > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff660200700 0 [prometheus INFO root] Starting > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > STARTING > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.647+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > STARTING > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] recovery thread > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] starting > setup > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff67aa00700 1 mgr load Constructed class from module: > rbd_support > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.71' (ECDSA) to the list of known > hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 0 [restful DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 1 mgr load Constructed class from module: > restful > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.660+0000 7ff651c00700 0 [restful INFO root] server_addr: :: server_port: > 8003 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.661+0000 7ff654e00700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: > load_schedules > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.697+0000 7ff651c00700 0 [restful WARNING root] server not running: no certificate > configured > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.705+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules: csxyz, > start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.706+0000 7ff67aa00700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.707+0000 7ff64f400700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.708+0000 7ff64ea00700 0 [rbd_support INFO root] PerfHandler: > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.709+0000 7ff654e00700 0 [rbd_support INFO root] load_task_task: csxyz, > start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.718+0000 7ff649a00700 0 [rbd_support INFO root] TaskHandler: > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.719+0000 7ff654e00700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: > load_schedules > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.721+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules: csxyz, > start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff649000700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff654e00700 0 [rbd_support INFO root] setup > complete > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.726+0000 7ff67aa00700 1 mgr load Constructed class from module: > snap_schedule > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 0 [status DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 1 mgr load Constructed class from module: > status > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.728+0000 7ff67aa00700 0 [telemetry DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 1 mgr load Constructed class from module: > telemetry > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 0 [volumes DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume > 'cssnfsxyz' > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume > 'cssnfsxyz' > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 1 mgr load Constructed class from module: > volumes > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy > Checker: > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty > config. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Serving on > http://:::9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Serving on > http://:::9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > STARTED > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > STARTED > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO root] Engine > started. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.785+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS -> > /api/cephfs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.787+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi -> > /ui-api/cephfs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi -> > /ui-api/iscsi > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi -> > /api/iscsi > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget -> > /api/iscsi/target > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster -> > /api/nfs-ganesha/cluster > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports -> > /api/nfs-ganesha/export > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi -> > /ui-api/nfs-ganesha > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd -> > /api/block/image > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdStatus -> > /ui-api/block/rbd > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot -> > /api/block/image/{image_spec}/snap > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.796+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash -> > /api/block/image/trash > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace -> > /api/block/pool/{pool_name}/namespace > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool -> > /api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.799+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi -> > /ui-api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.800+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RBDPool -> > /api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator -> > /ui-api/orchestrator > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service -> > /api/service > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring -> > /api/block/mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary -> > /api/block/mirroring/summary > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode -> > /api/block/mirroring/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap -> > /api/block/mirroring/pool/{pool_name}/bootstrap > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer -> > /api/block/mirroring/pool/{pool_name}/peer > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.804+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringStatus -> > /ui-api/block/mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw -> > /ui-api/rgw > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon -> > /api/rgw/daemon > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite -> > /api/rgw/site > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket -> > /api/rgw/bucket > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.807+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser -> > /api/rgw/user > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.810+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth -> > /api/auth > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster -> > /api/cluster > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration -> > /api/cluster_conf > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.813+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule -> > /api/crush_rule > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi -> > /ui-api/crush_rule > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon -> > /api/daemon > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs -> > /docs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile -> > /api/erasure_code_profile > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.816+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi -> > /ui-api/erasure_code_profile > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging -> > /ui-api/logging > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana -> > /api/grafana > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host -> > /api/host > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.819+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi -> > /ui-api/host > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health -> > /api/health > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController -> > / > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.822+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController -> > /ui-api/langs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LoginController -> > /ui-api/login > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs -> > /api/logs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules -> > /api/mgr/module > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor -> > /api/monitor > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd -> > /api/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.828+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdUi -> > /ui-api/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.830+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController -> > /api/osd/flags > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter -> > /api/perf_counters/mds > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter -> > /api/perf_counters/mon > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter -> > /api/perf_counters/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter -> > /api/perf_counters/rgw > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter -> > /api/perf_counters/rbd-mirror > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter -> > /api/perf_counters/mgr > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter -> > /api/perf_counters/tcmu-runner > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters -> > /api/perf_counters > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver -> > /api/prometheus_receiver > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus -> > /api/prometheus > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications -> > /api/prometheus/notifications > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role -> > /api/role > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope -> > /ui-api/scope > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2 -> > /auth/saml2 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings -> > /api/settings > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings -> > /ui-api/standard_settings > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary -> > /api/summary > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task -> > /api/task > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry -> > /api/telemetry > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.842+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User -> > /api/user > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy -> > /api/user > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword -> > /api/user/{username} > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint -> > /api/feature_toggles > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.844+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay -> > /ui-api/motd > Apr 05 20:33:46 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.050+0000 7ff676400700 0 [dashboard INFO dashboard.module] Engine > started. > Apr 05 20:33:46 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.378+0000 7ff67a000700 0 log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB > avail > Apr 05 20:33:47 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:47.379+0000 7ff678c00700 0 log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB > avail > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104 "" > "Prometheus/2.33.4" > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.568+0000 7ff65b200700 0 [prometheus INFO cherrypy.access.140699978276536] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104 "" > "Prometheus/2.33.4" > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 [cephadm INFO cephadm.serve] Updating > cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 log_channel(cephadm) log [INF] : Updating > cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 [cephadm INFO cephadm.serve] Updating > cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve] Updating > cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 log_channel(cephadm) log [INF] : Updating > cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] : Updating > cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 [cephadm INFO cephadm.serve] Updating > cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 log_channel(cephadm) log [INF] : Updating > cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 [cephadm INFO cephadm.serve] Updating > cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 log_channel(cephadm) log [INF] : Updating > cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve] Updating > cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] : Updating > cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.812+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 824b8fde-5a37-4436-af45-a7d8255e2c66 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 621cb660-d584-45e4-8033-4a476e804bba does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 007a4dd2-46d3-4655-9c93-c02f97e555cb does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 6b0fbff2-881e-4203-9149-f0977bc13eab does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 8a1a90b8-1f2a-48c8-924f-6e74032b6319 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.815+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3c322d26-9edb-4d26-b4bc-97c330f5a1d8 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev f3e5a517-d643-4fa5-b127-57716264e4e3 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3fb020c7-92e8-4eb1-a884-2d04e5bca889 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.840+0000 7ff6a7400700 0 [progress WARNING root] complete: ev bf9e0705-075e-4134-b77f-aa64fe92112a does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade myself > (mgr.cn02.ceph.xyz.corp.ggixgj) > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade myself > (mgr.cn02.ceph.xyz.corp.ggixgj) > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [progress INFO root] update: starting ev e7e1a809-558d-43a7-842a-c6229fdc57af (Upgrade to > 17.2.0) > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 [cephadm INFO cephadm.services.cephadmservice] Failing over to other > MGR > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Failing over to other > MGR > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 -1 mgr handle_mgr_map I was active but no longer > am > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn e: > '/usr/bin/ceph-mgr' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 0: > '/usr/bin/ceph-mgr' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 1: > '-n' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 2: > 'mgr.cn02.ceph.xyz.corp.ggixgj' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 3: > '-f' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 4: > '--setuser' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 5: > 'ceph' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 6: > '--setgroup' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 7: > 'ceph' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 8: > '--default-log-to-file=false' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 9: > '--default-log-to-stderr=true' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 10: '--default-log-stderr-prefix=debug > ' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn respawning with exe > /usr/bin/ceph-mgr > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn exe_path > /proc/self/exe > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setuser ceph since I am not > root > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setgroup ceph since I am not > root > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable), process ceph-mgr, pid > 2 > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 pidfile_write: ignore empty > --pid-file > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.976+0000 7f5a66492680 1 mgr[py] Loading python module > 'alerts' > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES > member > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 1 mgr[py] Loading python module > 'balancer' > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES > member > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 1 mgr[py] Loading python module > 'cephadm' > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.568+0000 7f5a66492680 1 mgr[py] Loading python module > 'crash' > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 -1 mgr[py] Module crash has missing NOTIFY_TYPES > member > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 1 mgr[py] Loading python module > 'dashboard' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.276+0000 7f5a66492680 1 mgr[py] Loading python module > 'devicehealth' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.415+0000 7f5a66492680 1 mgr[py] Loading python module > 'diskprediction_local' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES > member > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 1 mgr[py] Loading python module > 'influx' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 -1 mgr[py] Module influx has missing NOTIFY_TYPES > member > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 1 mgr[py] Loading python module > 'insights' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.919+0000 7f5a66492680 1 mgr[py] Loading python module > 'iostat' > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES > member > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 1 mgr[py] Loading python module > 'k8sevents' > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.533+0000 7f5a66492680 1 mgr[py] Loading python module > 'localpool' > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.634+0000 7f5a66492680 1 mgr[py] Loading python module > 'mds_autoscaler' > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.894+0000 7f5a66492680 1 mgr[py] Loading python module > 'mirroring' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.019+0000 7f5a66492680 1 mgr[py] Loading python module > 'nfs' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 1 mgr[py] Loading python module > 'orchestrator' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 1 mgr[py] Loading python module > 'osd_support' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 1 mgr[py] Loading python module > 'pg_autoscaler' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 1 mgr[py] Loading python module > 'progress' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 -1 mgr[py] Module progress has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 1 mgr[py] Loading python module > 'prometheus' > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES > member > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 1 mgr[py] Loading python module > 'rbd_support' > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES > member > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 1 mgr[py] Loading python module > 'restful' > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.799+0000 7f5a66492680 1 mgr[py] Loading python module > 'rook' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 -1 mgr[py] Module rook has missing NOTIFY_TYPES > member > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 1 mgr[py] Loading python module > 'selftest' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES > member > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 1 mgr[py] Loading python module > 'snap_schedule' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES > member > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 1 mgr[py] Loading python module > 'stats' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.660+0000 7f5a66492680 1 mgr[py] Loading python module > 'status' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 -1 mgr[py] Module status has missing NOTIFY_TYPES > member > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 1 mgr[py] Loading python module > 'telegraf' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 1 mgr[py] Loading python module > 'telemetry' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 1 mgr[py] Loading python module > 'test_orchestrator' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 1 mgr[py] Loading python module > 'volumes' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 1 mgr[py] Loading python module > 'zabbix' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.792+0000 7f5a66492680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.793+0000 7f5a52600700 0 ms_deliver_dispatch: unhandled message 0x556eb85f26e0 mon_map magic: 0 v1 from mon.2 > v2:192.168.47.72:3300/0 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from module: > dashboard > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from module: > prometheus > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f58f7800700 0 [dashboard INFO root] server: ssl=yes host=:: > port=8443 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] server_addr: :: server_port: > 9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] Starting > engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Bus > STARTING > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Bus > STARTING > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.797+0000 7f5a52600700 0 client.0 ms_handle_reset on > v2:192.168.47.73:6842/917991990 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.825+0000 7f58f7800700 0 [dashboard INFO root] Configured CherryPy, starting > engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.826+0000 7f58f7800700 0 [dashboard INFO root] Starting > engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy > Checker: > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty > config. > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Serving on > http://:::9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Serving on > http://:::9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Bus > STARTED > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Bus > STARTED > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO root] Engine > started. > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.939+0000 7f58f7800700 0 [dashboard INFO root] Engine > started... > Apr 05 20:33:58 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:58 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:58.569+0000 7f59b8000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:00 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:00 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:00.702+0000 7f59af400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:03.286+0000 7f59a6800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:08 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:08 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:08.569+0000 7f599dc00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:10 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:10 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:10.702+0000 7f5995000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:13.286+0000 7f598c400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:18 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:18 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:18.569+0000 7f5983800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:20 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:20 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:20.702+0000 7f597ac00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > > cn03 > —— > Apr 05 20:33:42 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:42] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:42 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:42.753+0000 7f2724a00700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:42] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:43 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:43] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:43.772+0000 7f271be00700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:47 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:47] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:47 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:47.037+0000 7f2713200700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:33:47] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.867+0000 7f27d1a00700 1 mgr handle_mgr_map > Activating! > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.868+0000 7f27d1a00700 1 mgr handle_mgr_map > I am now activating > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > root] Stopping engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > root] Stopped engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > root] Stopping engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > root] Stopped engine > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.879+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.890+0000 7f2675a00700 0 [dashboard INFO > root] Engine stopped. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE HTTP Server > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.891+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE HTTP Server > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > root] Engine stopped. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts INFO root] > Init > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 1 mgr load > Constructed class from module: alerts > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f2663400700 0 [balancer DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f2663400700 1 mgr load > Constructed class from module: balancer > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f26f8e00700 0 [alerts INFO root] > Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.901+0000 7f2663400700 0 [cephadm DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.901+0000 7f26f0200700 0 [balancer INFO > root] Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > root] Optimize plan auto_2025-04-05_20:33:48 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > root] Mode upmap, max misplaced 0.050000 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO > root] Some PGs (1.000000) are unknown; try again later > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed > to construct class in 'cephadm' > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load > Traceback (most recent call last): > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.upgrade = CephadmUpgrade(self) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.upgrade_state: Optional[UpgradeState] = > UpgradeState.from_json(json.loads(t)) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > return cls(**c) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > TypeError: __init__() got an unexpected keyword argument > 'daemon_types' > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() > Failed to run module in active mode ('cephadm') > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [crash DEBUG root] > setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > Constructed class from module: crash > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [dashboard DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > Constructed class from module: dashboard > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.920+0000 7f2660c00700 0 [dashboard INFO > access_control] Loading user roles DB version=2 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > sso] Loading SSO DB version=1 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > root] server: ssl=yes host=:: port=8443 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [devicehealth > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > Constructed class from module: devicehealth > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [insights DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > Constructed class from module: insights > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.923+0000 7f2660200700 0 [devicehealth INFO > root] Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f2663400700 0 [iostat DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f2663400700 1 mgr load > Constructed class from module: iostat > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f265f800700 0 [insights INFO > root] Reset curr health slot key health_history/2025-04-05_20 next > flush None checks check count 0 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.926+0000 7f2663400700 0 [mirroring DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.931+0000 7f2663400700 1 mgr load > Constructed class from module: mirroring > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [nfs DEBUG root] > setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > Constructed class from module: nfs > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [orchestrator > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > Constructed class from module: orchestrator > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.933+0000 7f2663400700 0 [pg_autoscaler > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.933+0000 7f2663400700 1 mgr load > Constructed class from module: pg_autoscaler > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2663400700 0 [progress DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2663400700 1 mgr load > Constructed class from module: progress > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2657600700 0 [pg_autoscaler > INFO root] _maybe_adjust > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.940+0000 7f2656c00700 0 [progress INFO > root] Loading... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.940+0000 7f2663400700 0 [prometheus DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2663400700 1 mgr load > Constructed class from module: prometheus > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > root] server_addr: :: server_port: 9283 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > root] Cache enabled > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > root] Loaded [<progress.module.GhostEvent object at 0x7f27363ac400>, > <progress.module.GhostEvent object at 0x7f27363ac438>, > <progress.module.GhostEvent object at 0x7f27363ac3c8>, > <progress.module.GhostEvent object at 0x7f27363ac5c0>, > <progress.module.GhostEvent object at 0x7f27363ac4e0>, > <progress.module.GhostEvent object at 0x7f27363ac4a8>, > <progress.module.GhostEvent object at 0x7f278d317e10>, > <progress.module.GhostEvent object at 0x7f278d4397b8>, > <progress.module.GhostEvent object at 0x7f278d439908>, > <progress.module.GhostEvent object at 0x7f27363ed2e8>, > <progress.module.GhostEvent object at 0x7f27363ed4e0>, > <progress.module.GhostEvent object at 0x7f27363ed358>, > <progress.module.GhostEvent object at 0x7f27363ed278>, > <progress.module.GhostEvent object at 0x7f27363edeb8>, > <progress.module.GhostEvent object at 0x7f27363ed438>, > <progress.module.GhostEvent object at 0x7f27363ed400>, > <progress.module.GhostEvent object at 0x7f27363edb00>, > <progress.module.GhostEvent object at 0x7f27363ed780>, > <progress.module.GhostEvent object at 0x7f27363ed240>, > <progress.module.GhostEvent object at 0x7f27363edba8>, > <progress.module.GhostEvent object at 0x7f27363ed2b0>, > <progress.module.GhostEvent object at 0x7f27363ed0b8>, > <progress.module.GhostEvent object at 0x7f27363edcf8>, > <progress.module.GhostEvent object at 0x7f27363ed128>, > <progress.module.GhostEvent object at 0x7f27363ed080>, > <progress.module.GhostEvent object at 0x7f27363edc18>, > <progress.module.GhostEvent object at 0x7f27363ed860>, > <progress.module.GhostEvent object at 0x7f27363eda58>, > <progress.module.GhostEvent object at 0x7f27363eda90>, > <progress.module.GhostEvent object at 0x7f27363ed160>, > <progress.module.GhostEvent object at 0x7f27363ed908>, > <progress.module.GhostEvent object at 0x7f27363ed940>, > <progress.module.GhostEvent object at 0x7f27363ed8d0>, > <progress.module.GhostEvent object at 0x7f27363ed978>, > <progress.module.GhostEvent object at 0x7f27363ed748>, > <progress.module.GhostEvent object at 0x7f27363ed898>, > <progress.module.GhostEvent object at 0x7f27363eda20>, > <progress.module.GhostEvent object at 0x7f27363ed9b0>, > <progress.module.GhostEvent object at 0x7f27363ed9e8>, > <progress.module.GhostEvent object at 0x7f27363ed828>, > <progress.module.GhostEvent object at 0x7f27363ed7f0>, > <progress.module.GhostEvent object at 0x7f27363ed7b8>, > <progress.module.GhostEvent object at 0x7f27363ed668>, > <progress.module.GhostEvent object at 0x7f27363ed320>, > <progress.module.GhostEvent object at 0x7f27363ede48>, > <progress.module.GhostEvent object at 0x7f27363edc50>, > <progress.module.GhostEvent object at 0x7f27363ede80>, > <progress.module.GhostEvent object at 0x7f27363edf60>, > <progress.module.GhostEvent object at 0x7f27363edd68>, > <progress.module.GhostEvent object at 0x7f27363edda0>] historic events > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > root] Loaded OSDMap, ready. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.946+0000 7f2663400700 0 [rbd_support DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.946+0000 7f2655800700 0 [prometheus INFO > root] starting metric collection thread > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.947+0000 7f2656200700 0 [prometheus INFO > root] Starting engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STARTING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.948+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STARTING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.948+0000 7f2660c00700 0 [dashboard INFO > root] Configured CherryPy, starting engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.953+0000 7f2663400700 0 [rbd_support INFO > root] MirrorSnapshotScheduleHandler: load_schedules > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.957+0000 7f2663400700 0 [rbd_support INFO > root] load_schedules: csxyz, start_after= > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.989+0000 7f2649a00700 0 [rbd_support INFO > root] MirrorSnapshotScheduleHandler: starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.990+0000 7f2649000700 0 [rbd_support INFO > root] PerfHandler: starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.991+0000 7f2663400700 0 [rbd_support INFO > root] load_task_task: csxyz, start_after= > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.019+0000 7f2648600700 0 [rbd_support INFO > root] TaskHandler: starting > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.020+0000 7f2663400700 0 [rbd_support INFO > root] TrashPurgeScheduleHandler: load_schedules > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.022+0000 7f2663400700 0 [rbd_support INFO > root] load_schedules: csxyz, start_after= > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.024+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CephFS -> /api/cephfs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.026+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CephFsUi -> /ui-api/cephfs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > IscsiUi -> /ui-api/iscsi > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Iscsi -> /api/iscsi > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > IscsiTarget -> /api/iscsi/target > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGanesha -> /api/nfs-ganesha > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaCluster -> /api/nfs-ganesha/cluster > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaExports -> /api/nfs-ganesha/export > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaUi -> /ui-api/nfs-ganesha > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Rbd > -> /api/block/image > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.034+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdSnapshot -> /api/block/image/{image_spec}/snap > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdTrash -> /api/block/image/trash > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdNamespace -> /api/block/pool/{pool_name}/namespace > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroring -> /api/block/mirroring > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringSummary -> /api/block/mirroring/summary > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolMode -> /api/block/mirroring/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolBootstrap -> > /api/block/mirroring/pool/{pool_name}/bootstrap > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolPeer -> /api/block/mirroring/pool/{pool_name}/peer > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.038+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Rgw > -> /api/rgw > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwDaemon -> /api/rgw/daemon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwSite -> /api/rgw/site > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.040+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwBucket -> /api/rgw/bucket > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.041+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwUser -> /api/rgw/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Auth -> /api/auth > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Cluster -> /api/cluster > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.045+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ClusterConfiguration -> /api/cluster_conf > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.046+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CrushRule -> /api/crush_rule > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.047+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CrushRuleUi -> /ui-api/crush_rule > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Orchestrator -> /api/orchestrator > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Daemon -> /api/daemon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Docs -> /docs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ErasureCodeProfile -> /api/erasure_code_profile > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ErasureCodeProfileUi -> /ui-api/erasure_code_profile > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackController -> /api/feedback > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.051+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackApiController -> /api/feedback/api_key > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackUiController -> /ui-api/feedback/api_key > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FrontendLogging -> /ui-api/logging > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Grafana -> /api/grafana > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Host -> /api/host > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > HostUi -> /ui-api/host > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Health -> /api/health > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > HomeController -> / > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:49] ENGINE Serving on http://:::9283 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Serving on > http://:::9283 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:49] ENGINE Bus STARTED > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Bus STARTED > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > root] Engine started. > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > LangsController -> /ui-api/langs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Logs -> /api/logs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MgrModules -> /api/mgr/module > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Monitor -> /api/monitor > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Osd > -> /api/osd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.060+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > OsdFlagsController -> /api/osd/flags > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MdsPerfCounter -> /api/perf_counters/mds > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MonPerfCounter -> /api/perf_counters/mon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > OsdPerfCounter -> /api/perf_counters/osd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwPerfCounter -> /api/perf_counters/rgw > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirrorPerfCounter -> /api/perf_counters/rbd-mirror > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MgrPerfCounter -> /api/perf_counters/mgr > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > TcmuRunnerPerfCounter -> /api/perf_counters/tcmu-runner > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PerfCounters -> /api/perf_counters > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.065+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Pool -> /api/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.066+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PoolUi -> /ui-api/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PrometheusReceiver -> /api/prometheus_receiver > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Prometheus -> /api/prometheus > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.068+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PrometheusNotifications -> /api/prometheus/notifications > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.069+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Role -> /api/role > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Scope -> /ui-api/scope > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Saml2 -> /auth/saml2 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Service -> /api/service > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.071+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Settings -> /api/settings > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > StandardSettings -> /ui-api/standard_settings > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Summary -> /api/summary > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Task -> /api/task > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Telemetry -> /api/telemetry > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > User -> /api/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > UserPasswordPolicy -> /api/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > UserChangePassword -> /api/user/{username} > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeatureTogglesEndpoint -> /api/feature_toggles > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MessageOfTheDay -> /ui-api/motd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.090+0000 7f2647200700 0 [rbd_support INFO > root] TrashPurgeScheduleHandler: starting > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.096+0000 7f2663400700 1 mgr load > Constructed class from module: rbd_support > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.102+0000 7f2663400700 0 [restful DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.102+0000 7f2663400700 1 mgr load > Constructed class from module: restful > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.108+0000 7f2645e00700 0 [restful INFO > root] server_addr: :: server_port: 8003 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.109+0000 7f2663400700 0 [snap_schedule > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.116+0000 7f2645e00700 0 [restful WARNING > root] server not running: no certificate configured > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.257+0000 7f2660c00700 0 [dashboard INFO > dashboard.module] Engine started. > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed > to construct class in 'snap_schedule' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load > Traceback (most recent call last): > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in > __init__ > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.client = SnapSchedClient(self) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > 158, in __init__ > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > with self.get_schedule_db(fs_name) as conn_mgr: > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > 192, in get_schedule_db > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > db.executescript(dump) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > sqlite3.OperationalError: table schedules already exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() > Failed to run module in active mode ('snap_schedule') > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 0 [status DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 1 mgr load > Constructed class from module: status > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.275+0000 7f2663400700 0 [telemetry DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.276+0000 7f2663400700 1 mgr load > Constructed class from module: telemetry > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.276+0000 7f2663400700 0 [volumes DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.284+0000 7f2660200700 0 [devicehealth INFO > root] Check health > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 1 mgr load > Constructed class from module: volumes > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.883+0000 7f2662a00700 0 > log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail > Apr 05 20:33:50 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:50.875+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail > Apr 05 20:33:52 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:52] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:33:52 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:52.753+0000 7f264f400700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:52] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:33:52 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:52.876+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v5: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:53] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:53.772+0000 7f264ea00700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:33:53] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:53.870+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:53.871+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:53.871+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:54 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:54.877+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v6: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 0 > B/s rd, 12 KiB/s wr, 2 op/s > Apr 05 20:33:55 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:55.796+0000 7f2662a00700 2 mgr.server > handle_open ignoring open from mgr.cn02.ceph.xyz.corp.ggixgj > 192.168.47.72:0/2580067884; not ready for session (expect reconnect) > Apr 05 20:33:56 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:56.878+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v7: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 439 B/s rd, 27 KiB/s wr, 6 op/s > Apr 05 20:33:57 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:57] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:33:57 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:57.038+0000 7f264e000700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:33:57] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:33:58 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:58.870+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:58 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:58.878+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v8: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 341 B/s rd, 21 KiB/s wr, 4 op/s > Apr 05 20:33:58 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:58.897+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:58 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:58.897+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:00 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:00.880+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v9: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 279 B/s rd, 24 KiB/s wr, 6 op/s > Apr 05 20:34:02 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:02] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:34:02 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:02.753+0000 7f264d600700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:34:02 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:02.880+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v10: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:03] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:03.772+0000 7f264cc00700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:03.870+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:03.873+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:03.874+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:04 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:04.881+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v11: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 59 KiB/s wr, 11 op/s > Apr 05 20:34:06 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:06.882+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v12: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 511 B/s rd, 66 KiB/s wr, 13 op/s > Apr 05 20:34:07 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:07] "GET /metrics > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > Apr 05 20:34:07 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:07.037+0000 7f264c200700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:07] "GET /metrics HTTP/1.1" 200 267239 "" > "Prometheus/2.33.4" > Apr 05 20:34:08 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:08.871+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:34:08 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:08.883+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v13: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 55 KiB/s wr, 11 op/s > Apr 05 20:34:08 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:08.907+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:34:08 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:08.907+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:10 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:10.884+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v14: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 79 KiB/s wr, 16 op/s > Apr 05 20:34:12 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:12] "GET /metrics > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > Apr 05 20:34:12 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:12.753+0000 7f264b800700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 267239 "" > "Prometheus/2.33.4" > Apr 05 20:34:12 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:12.884+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v15: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 73 KiB/s wr, 14 op/s > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:13] "GET /metrics > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:13.772+0000 7f264ae00700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 267239 "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:13.871+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:13.883+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:13.883+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:14 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:14.886+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v16: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 103 KiB/s wr, 113 op/s > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.754+0000 7f2644000700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.089s] [1.7K] > [6c8ffcbf-e0d6-4f01-a24a-693d11b5fcdb] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.769+0000 7f2643600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.003s] [1.7K] > [d20dd549-caaf-4b35-9784-465556871a83] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.787+0000 7f2642c00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] > [2efb9ce6-0ec0-4525-afcf-5d04356c0f18] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.803+0000 7f2642200700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] > [2f0b6080-a99d-4fdf-b7f9-c013a4cf2b78] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.825+0000 7f2641800700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] > [5c67930f-511b-4e2a-993e-13bc0c73fe2c] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.842+0000 7f2640e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] > [043cc2d4-1863-4072-a15e-91a04422c2d3] / > Apr 05 20:34:16 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:16.887+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v17: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 66 KiB/s wr, 106 op/s > Apr 05 20:34:17 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:17] "GET /metrics > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > Apr 05 20:34:17 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:17.037+0000 7f264a400700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:17] "GET /metrics HTTP/1.1" 200 267239 "" > "Prometheus/2.33.4" > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.871+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.887+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v18: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 54 KiB/s wr, 103 op/s > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.916+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.916+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.931+0000 7f2659e00700 0 [mirroring INFO > mgr_util] scanning for idle connections.. > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.931+0000 7f2659e00700 0 [mirroring INFO > mgr_util] cleaning up connections: [] > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.109+0000 7f2645400700 0 [snap_schedule > INFO mgr_util] scanning for idle connections.. > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.109+0000 7f2645400700 0 [snap_schedule > INFO mgr_util] cleaning up connections: [] > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.277+0000 7f2632800700 0 [volumes INFO > mgr_util] scanning for idle connections.. > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.277+0000 7f2632800700 0 [volumes INFO > mgr_util] cleaning up connections: [] > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.282+0000 7f262d800700 0 [volumes INFO > mgr_util] scanning for idle connections.. > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.282+0000 7f262d800700 0 [volumes INFO > mgr_util] cleaning up connections: [] > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.287+0000 7f2629c00700 0 [volumes INFO > mgr_util] scanning for idle connections.. > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.287+0000 7f2629c00700 0 [volumes INFO > mgr_util] cleaning up connections: [] > Apr 05 20:34:20 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:20.889+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v19: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 70 KiB/s wr, 107 op/s > Apr 05 20:34:22 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:22] "GET /metrics > HTTP/1.1" 200 267240 "" "Prometheus/2.33.4" > Apr 05 20:34:22 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:22.753+0000 7f264fe00700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:22] "GET /metrics HTTP/1.1" 200 267240 "" > "Prometheus/2.33.4" > Apr 05 20:34:22 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:22.889+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v20: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 45 KiB/s wr, 102 op/s > > Thank you > -jeremy > >> On Saturday, Apr 05, 2025 at 1:32 PM, Eugen Block <eblock(a)nde.ag >> (mailto:eblock@nde.ag)> wrote: >> The mgr logs should contain a stack trace, can you check again? >> >> >> Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: >> >> > No. Same issue unfortunately. >> > >> > > On Saturday, Apr 05, 2025 at 1:27 PM, Eugen Block <eblock(a)nde.ag >> > > (mailto:eblock@nde.ag)> wrote: >> > > I don't see the module error in the logs you provided. Has it cleared? >> > > A mgr failover is often helpful in such cases, basically it's the >> > > first thing I suggest since two or three years. >> > > >> > > Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: >> > > >> > > > Thank you so much for the detailed instructions. Here’s logs from >> > > > the failover to a new node. >> > > > >> > > > Apr 05 20:06:08 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 >> - >> > > "" >> > > > "Prometheus/2.33.4" >> > > > Apr 05 20:06:08 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:08.568+0000 7efcbd800700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 >> - >> > > "" >> > > > "Prometheus/2.33.4" >> > > > Apr 05 20:06:10 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 >> - >> > > "" >> > > > "Prometheus/2.33.4" >> > > > Apr 05 20:06:10 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:10.702+0000 7efcb4c00700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 >> - >> > > "" >> > > > "Prometheus/2.33.4" >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.027+0000 7efd95200700 1 >> mgr >> > > handle_mgr_map >> > > > Activating! >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.028+0000 7efd95200700 1 mgr handle_mgr_map I >> am >> > > now >> > > > activating >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO >> root] >> > > Stopping >> > > > engine... >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO >> root] >> > > Stopped >> > > > engine... >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO >> root] >> > > Stopping >> > > > engine... >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO >> root] >> > > Stopped >> > > > engine >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] >> ENGINE >> > > Bus >> > > > STOPPING >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] >> ENGINE >> > > Bus >> > > > STOPPING >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.053+0000 7efc3a400700 0 [dashboard INFO >> root] >> > > Engine >> > > > stopped. >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', >> 9283)) >> > > shut >> > > > down >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', >> 9283)) >> > > shut >> > > > down >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] >> ENGINE >> > > Bus >> > > > STOPPED >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] >> ENGINE >> > > Bus >> > > > STOPPED >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO >> root] >> > > Engine >> > > > stopped. >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.056+0000 7efc27e00700 0 [alerts DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 0 [alerts >> INFO >> > > root] >> > > > Init >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > alerts >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 0 [balancer DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > balancer >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efcfac00700 0 [alerts >> INFO >> > > root] >> > > > Starting >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efc27e00700 0 [cephadm DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efcb4c00700 0 [balancer >> INFO >> > > root] >> > > > Starting >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] >> Optimize >> > > plan >> > > > auto_2025-04-05_20:06:12 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Mode upmap, >> max >> > > misplaced >> > > > 0.050000 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Some PGs (1.000000) are unknown; >> try >> > > again >> > > > later >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.212+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > cephadm >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 0 [crash DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > crash >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 0 [dashboard DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > dashboard >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.216+0000 7efc23800700 0 [dashboard INFO access_control] Loading user >> roles >> > > DB >> > > > version=2 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO sso] Loading >> SSO >> > > DB >> > > > version=1 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO root] server: >> ssl=yes >> > > host=:: >> > > > port=8443 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 0 [devicehealth DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > devicehealth >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 0 [insights DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > insights >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc20600700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:06:12 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc21a00700 0 [devicehealth >> INFO >> > > root] >> > > > Starting >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 0 [iostat DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > iostat >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.224+0000 7efc27e00700 0 [mirroring DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.227+0000 7efc21a00700 0 [devicehealth INFO >> root] >> > > Check >> > > > health >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.235+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > mirroring >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 0 [nfs DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > nfs >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 0 [orchestrator DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > orchestrator >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 0 [pg_autoscaler DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > pg_autoscaler >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc10200700 0 [pg_autoscaler >> INFO >> > > root] >> > > > _maybe_adjust >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 0 [progress DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > progress >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.247+0000 7efc27e00700 0 [prometheus DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.248+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > prometheus >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc0ee00700 0 [progress >> INFO >> > > root] >> > > > Loading... >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc27e00700 0 [rbd_support DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7efcf2127710>, <progress.module.GhostEvent object at 0x7efcf2127cc0>, <progress.module.GhostEvent object at 0x7efcf2127208>, <progress.module.GhostEvent object at 0x7efcf2127f28>, <progress.module.GhostEvent object at 0x7efcf21270f0>, <progress.module.GhostEvent object at 0x7efcf2127eb8>, <progress.module.GhostEvent object at 0x7efcf2127b38>, <progress.module.GhostEvent object at 0x7efcf2127a20>, <progress.module.GhostEvent object at 0x7efcf2127da0>, <progress.module.GhostEvent object at 0x7efcf2127f98>, <progress.module.GhostEvent object at 0x7efcf2127780>, <progress.module.GhostEvent object at 0x7efcf2127320>, <progress.module.GhostEvent object at 0x7efcf2127588>, <progress.module.GhostEvent object at 0x7efcf2127518>, <progress.module.GhostEvent object at 0x7efcf2127c50>, <progress.module.GhostEvent object at 0x7efcf21279e8>, <progress.module.GhostEvent object at 0x7efcf2127d68>, <progress.module.GhostEvent object at 0x7efcf2127470>, <progress.module.GhostEvent object at >> 0x7efcf > [Text gekürzt]

2 6

Re: Cephadm upgrade from 16.2.15 -> 17.2.0
by Eugen Block 05 Apr '25

05 Apr '25

Please don't drop the list from your response. The orchestrator might be down, but you should have an active mgr (ceph -s). On the host running the mgr, you can just run 'cephadm logs --name mgr.<your_mgr>', or 'journalctl -u ceph-<ceph_fsid>@mgr.<your_mgr>'. To get fresh logs, you can just run 'ceph mgr fail', look on which host a different mgr takes over and gather logs from there. Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: > Thanks for the clarification on version jumps. I interpreted the > docs to mean that I had to incrementally step through each release. > This will save a lot of time. > > Since the orchestrator is down and I’m not sure how to trigger a > failure, I don’t know how to gather those logs. I apologize. I > appreciate your assistance. > > -jeremy > >> On Saturday, Apr 05, 2025 at 4:01 AM, Eugen Block <eblock(a)nde.ag >> (mailto:eblock@nde.ag)> wrote: >> It might be a different issue, can you paste the entire stack trace >> from the mgr when it's failing? >> >> Also, you could go directly from Pacific to Reef, there's no need to >> upgrade to Quincy (which is EOL). And I would also recommend to >> upgrade to a latest major version, e.g. 17.2.8, not .0, otherwise you >> might hit bugs which have already been fixed in later releases. >> >> Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: >> >> > I ran in to the “Error ENOENT: Module not found” issue with the >> > orchestrator. I see the note in the cephadm upgrade docs but I don’t >> > quite know what action to take to fix this: >> > >> > ceph versions >> > { >> > "mon": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 3 >> > }, >> > "mgr": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 1, >> > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) >> > quincy (stable)": 2 >> > }, >> > "osd": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 15 >> > }, >> > "mds": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 3 >> > }, >> > "overall": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 22, >> > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) >> > quincy (stable)": 2 >> > } >> > } >> > >> > ceph orch upgrade status >> > Error ENOENT: Module not found >> > >> > ceph config-key get mgr/cephadm/osd_remove_queue >> > [] >> > >> > Thanks. I’m trying to upgrade this cluster all the way to current. >> > >> > -jeremy >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 5

Cephadm upgrade from 16.2.15 -> 17.2.0
by Jeremy Hansen 05 Apr '25

05 Apr '25

I ran in to the “Error ENOENT: Module not found” issue with the orchestrator. I see the note in the cephadm upgrade docs but I don’t quite know what action to take to fix this: ceph versions { "mon": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 1, "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 }, "osd": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 15 }, "mds": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "overall": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 22, "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 } } ceph orch upgrade status Error ENOENT: Module not found ceph config-key get mgr/cephadm/osd_remove_queue [] Thanks. I’m trying to upgrade this cluster all the way to current. -jeremy

2 1

Recovery across datacenters after host failure
by Eugen Block 04 Apr '25

04 Apr '25

Hi all, let's say I have two DCs with replicated pools (size 4) and a tiebreaker MON somewhere else. Is it possible to control the recovery traffic in case of a host failure? Both DCs have enough replicas, so in theory it should be possible to recover within the DC with the failed host, right? This would reduce the required bandwidth between the DCs (ignoring an entire DC outage for now). But I don't think it currently works that way, at least nothing I've seen so far allows this assumption. I've been scanning the docs for hints, but to no avail yet. Assuming I didn't miss anything and this is not possible yet, I'm wondering if the concepts described for the "official" stretch mode [0],[1],[2] could be adapted: > While in stretch mode, OSDs will connect only to monitors within the > data center in which they are located. OSDs DO NOT connect to the > tiebreaker monitor. And for MON election: > When using stretch mode, the monitor election strategy must be set > to connectivity. This strategy tracks network connectivity between > the monitors and is used to determine which zone should be favored > when the cluster is in a netsplit scenario. The inter-OSD connections could be scored as well to limit the recovery traffic to the local DC. Or has this been already discussed and ruled out? I would appreciate any insights! Thanks! Eugen [0] https://docs.ceph.com/en/reef/rados/operations/stretch-mode/ [1] https://docs.ceph.com/en/reef/rados/operations/stretch-mode/#connectivity-m… [2] https://docs.ceph.com/en/reef/rados/operations/change-mon-elections/

1 0