ceph-users

Download

ceph-users@ceph.io

42 participants
9158 discussions

File immutability on CephFS ?
by Fabien Sirjean 09 Apr '25

09 Apr '25

Dear all, If I am to believe this issue [1], it seems that it is still not possible to make files immutable (chattr +i) in cephfs. Do you have any update on this matter ? Thanks a lot for all the good work! Cheers, Fabien [1] : https://tracker.ceph.com/issues/10679

2 1

squid 19.2.2 hotfix QE validation status
by Yuri Weinstein 08 Apr '25

08 Apr '25

This is a hotfix release to address only one issue: https://github.com/ceph/ceph/pull/62711 and has limited testing. Details of this release are summarized here: https://tracker.ceph.com/issues/70822#note-1 Release Notes - TBD LRC upgrade - excluded from this release Gibba upgrade - excluded from this release smoke - Adam E, Casey approved? rgw - Adam E, Casey approved?

2 1

irc/slack/discord bridge failing
by Alvaro Soto 08 Apr '25

08 Apr '25

Hi all, Our IRC/Slack/Discord bridge started failing since ~2 days ago. Slack deprecated the use of legacy bots[1], so at the moment, IRC/Discord is the only bridge working. [1] ~~~ time="2025-04-08T18:09:30Z" level=error msg="Could not retrieve channels: slack.SlackErrorResponse{Err:"legacy_custom_bots_deprecated", ResponseMetadata:slack.ResponseMetadata{Cursor:"", Messages:[]string(nil), Warnings:[]string(nil)}}" prefix=slack ~~~ I'll keep you posted. Cheers! -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 0

Re: Ceph squid fresh install
by quaglio＠bol.com.br 08 Apr '25

08 Apr '25

1 0

Re: Ceph squid fresh install
by quaglio＠bol.com.br 08 Apr '25

08 Apr '25

1 0

Re: Ceph squid fresh install
by quaglio＠bol.com.br 08 Apr '25

08 Apr '25

2 3

Data Loss Bug Identified on Squid
by Adam Emerson 07 Apr '25

07 Apr '25

https://tracker.ceph.com/issues/70746 tracks a data loss bug on squid when CopyObject is used to copy an object onto itself. S3 clients typically do this when they want to change the metadata of an existing object. Due to a regression caused by an earlier fix for https://tracker.ceph.com/issues/66286, any tail objects associated with such objects are erroneously marked for garbage collection. This regression was introduced with the initial 19.2.0 squid release and is still present in 19.2.1. There is no workaround to prevent clients from triggering this bug, so we're preparing 19.2.2 as a hotfix release to resolve this as soon as possible.

1 0

ceph.log using seconds since epoch instead of date/time stamp
by Stillwell, Bryan 07 Apr '25

07 Apr '25

Does anyone know why log entries in ceph.log and ceph.audit.log would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but ceph-mon.*.log and ceph-osd.*.log use date/time stamps? Example: # ceph.log 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok # ceph-mon.log 2025-02-21T00:00:01.814-0500 7fe33378e700 0 mon.XXXXXX@0(leader) e2 handle_command mon_command({"prefix": "osd dump"} v 0) v1 This doesn't seem to be the case on my home cluster running Squid 19.2.1/Ubuntu 24.04. I found this Proxmox forum post that seems to imply that could be a distro setting (same Ceph version, but bookworm is affected, and bullseye is not): https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.… Thanks, Bryan

3 4

Re: Cephadm upgrade from 16.2.15 -> 17.2.0
by Eugen Block 07 Apr '25

07 Apr '25

It would help if you only pasted the relevant parts. Anyway, these two sections stand out: ---snip--- Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again later Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed to construct class in 'cephadm' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade = CephadmUpgrade(self) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: return cls(**c) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: TypeError: __init__() got an unexpected keyword argument 'daemon_types' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('cephadm') Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed to construct class in 'snap_schedule' Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.client = SnapSchedClient(self) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 158, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: with self.get_schedule_db(fs_name) as conn_mgr: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 192, in get_schedule_db Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: db.executescript(dump) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: sqlite3.OperationalError: table schedules already exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('snap_schedule') ---snip--- Your cluster seems to be in an error state (ceph -s) because of an unknown PG. It's recommended to have a healthy cluster before attemping an upgrade. It's possible that these errors come from the not upgraded MGR, I'm not sure. Since the upgrade was only successful for two MGRs, I am thinking about downgrading both MGRs back to 16.2.15, then retry an upgrade to a newer version, either 17.2.8 or 18.2.4. I haven't checked the snap_schedule error yet, though. Maybe someone else knows that already. Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: > This is a small three node cluster. Here’s logs from all three nodes > during the failover. The current manager was cn02 and it failed over > to cn03. > > cn01 > —— > Apr 05 20:33:35 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:35.823+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v821: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.170+0000 7f2aaec00700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.823+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v822: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:33:39 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:39] "GET /metrics > HTTP/1.1" 200 267241 "" "Prometheus/2.33.4" > Apr 05 20:33:39 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:39.825+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v823: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 97 KiB/s wr, 12 op/s > Apr 05 20:33:41 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:41.826+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v824: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 85 KiB/s wr, 10 op/s > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.170+0000 7f2aaec00700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:42] "GET /metrics > HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics > HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:43.827+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v825: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 511 B/s rd, 133 KiB/s wr, 18 op/s > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 -1 mgr > handle_mgr_map I was active but no longer am > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn e: > '/usr/bin/ceph-mgr' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 0: > '/usr/bin/ceph-mgr' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 1: '-n' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 2: > 'mgr.cn01.tjmtph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 3: '-f' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 4: > '--setuser' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 5: > 'ceph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 6: > '--setgroup' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 7: > 'ceph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 8: > '--default-log-to-file=false' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 9: > '--default-log-to-stderr=true' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 10: > '--default-log-stderr-prefix=debug ' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn > respawning with exe /usr/bin/ceph-mgr > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn > exe_path /proc/self/exe > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ignoring --setuser ceph since I am not root > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ignoring --setgroup ceph since I am not root > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 ceph version > 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), > process ceph-mgr, pid 2 > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 pidfile_write: > ignore empty --pid-file > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.477+0000 7fb814447000 1 mgr[py] Loading > python module 'alerts' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.612+0000 7fb814447000 -1 mgr[py] Module > alerts has missing NOTIFY_TYPES member > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.612+0000 7fb814447000 1 mgr[py] Loading > python module 'balancer' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.726+0000 7fb814447000 -1 mgr[py] Module > balancer has missing NOTIFY_TYPES member > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.726+0000 7fb814447000 1 mgr[py] Loading > python module 'cephadm' > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.417+0000 7fb814447000 1 mgr[py] Loading > python module 'crash' > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.540+0000 7fb814447000 -1 mgr[py] Module > crash has missing NOTIFY_TYPES member > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.540+0000 7fb814447000 1 mgr[py] Loading > python module 'dashboard' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.210+0000 7fb814447000 1 mgr[py] Loading > python module 'devicehealth' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.331+0000 7fb814447000 -1 mgr[py] Module > devicehealth has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.331+0000 7fb814447000 1 mgr[py] Loading > python module 'diskprediction_local' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.653+0000 7fb814447000 -1 mgr[py] Module > diskprediction_local has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.654+0000 7fb814447000 1 mgr[py] Loading > python module 'influx' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.769+0000 7fb814447000 -1 mgr[py] Module > influx has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.769+0000 7fb814447000 1 mgr[py] Loading > python module 'insights' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.884+0000 7fb814447000 1 mgr[py] Loading > python module 'iostat' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.082+0000 7fb814447000 -1 mgr[py] Module > iostat has missing NOTIFY_TYPES member > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.082+0000 7fb814447000 1 mgr[py] Loading > python module 'k8sevents' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.679+0000 7fb814447000 1 mgr[py] Loading > python module 'localpool' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.796+0000 7fb814447000 1 mgr[py] Loading > python module 'mds_autoscaler' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.983+0000 7fb814447000 1 mgr[py] Loading > python module 'mirroring' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.118+0000 7fb814447000 1 mgr[py] Loading > python module 'nfs' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.315+0000 7fb814447000 -1 mgr[py] Module > nfs has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.315+0000 7fb814447000 1 mgr[py] Loading > python module 'orchestrator' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.607+0000 7fb814447000 -1 mgr[py] Module > orchestrator has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.607+0000 7fb814447000 1 mgr[py] Loading > python module 'osd_perf_query' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.723+0000 7fb814447000 -1 mgr[py] Module > osd_perf_query has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.723+0000 7fb814447000 1 mgr[py] Loading > python module 'osd_support' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.837+0000 7fb814447000 -1 mgr[py] Module > osd_support has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.837+0000 7fb814447000 1 mgr[py] Loading > python module 'pg_autoscaler' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.970+0000 7fb814447000 -1 mgr[py] Module > pg_autoscaler has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.970+0000 7fb814447000 1 mgr[py] Loading > python module 'progress' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.082+0000 7fb814447000 -1 mgr[py] Module > progress has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.082+0000 7fb814447000 1 mgr[py] Loading > python module 'prometheus' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.617+0000 7fb814447000 -1 mgr[py] Module > prometheus has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.617+0000 7fb814447000 1 mgr[py] Loading > python module 'rbd_support' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.761+0000 7fb814447000 -1 mgr[py] Module > rbd_support has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.761+0000 7fb814447000 1 mgr[py] Loading > python module 'restful' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.023+0000 7fb814447000 1 mgr[py] Loading > python module 'rook' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.788+0000 7fb814447000 -1 mgr[py] Module > rook has missing NOTIFY_TYPES member > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.788+0000 7fb814447000 1 mgr[py] Loading > python module 'selftest' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.912+0000 7fb814447000 -1 mgr[py] Module > selftest has missing NOTIFY_TYPES member > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.912+0000 7fb814447000 1 mgr[py] Loading > python module 'snap_schedule' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.038+0000 7fb814447000 -1 mgr[py] Module > snap_schedule has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.038+0000 7fb814447000 1 mgr[py] Loading > python module 'stats' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.155+0000 7fb814447000 1 mgr[py] Loading > python module 'status' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.280+0000 7fb814447000 -1 mgr[py] Module > status has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.280+0000 7fb814447000 1 mgr[py] Loading > python module 'telegraf' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.398+0000 7fb814447000 -1 mgr[py] Module > telegraf has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.398+0000 7fb814447000 1 mgr[py] Loading > python module 'telemetry' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.594+0000 7fb814447000 -1 mgr[py] Module > telemetry has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.594+0000 7fb814447000 1 mgr[py] Loading > python module 'test_orchestrator' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.771+0000 7fb814447000 -1 mgr[py] Module > test_orchestrator has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.771+0000 7fb814447000 1 mgr[py] Loading > python module 'volumes' > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.225+0000 7fb814447000 -1 mgr[py] Module > volumes has missing NOTIFY_TYPES member > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.225+0000 7fb814447000 1 mgr[py] Loading > python module 'zabbix' > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.342+0000 7fb814447000 -1 mgr[py] Module > zabbix has missing NOTIFY_TYPES member > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.344+0000 7fb807c00700 0 > ms_deliver_dispatch: unhandled message 0x555c78fe1080 mon_map magic: > 0 v1 from mon.0 v2:192.168.47.71:3300/0 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.344+0000 7fb808600700 0 [dashboard DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load > Constructed class from module: dashboard > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 0 [prometheus DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load > Constructed class from module: prometheus > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb6abe00700 0 [dashboard INFO > root] server: ssl=yes host=:: port=8443 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > root] server_addr: :: server_port: 9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > root] Starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Bus STARTING > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTING > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO > root] Configured CherryPy, starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO > root] Starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > CherryPy Checker: > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > The Application mounted at '' has an empty config. > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Serving on http://:::9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Serving on > http://:::9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Bus STARTED > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTED > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.451+0000 7fb6b4a00700 0 [prometheus INFO > root] Engine started. > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.489+0000 7fb6abe00700 0 [dashboard INFO > root] Engine started... > Apr 05 20:33:59 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:59] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:59 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:59.603+0000 7fb76c600700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:59] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:02 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:02] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:02 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:02.476+0000 7fb763a00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:03 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:03.061+0000 7fb75ae00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:09 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:09] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:09 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:09.602+0000 7fb752200700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:09] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:12 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:12] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:12 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:12.476+0000 7fb749600700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:13 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:13.060+0000 7fb740a00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.493+0000 7fb714e00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.494+0000 7fb714e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52368] [GET] [303] [0.003s] [96.0B] > [0031fbdd-947f-4c6c-943d-55cf256edfba] /api/summary > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.512+0000 7fb70c200700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.513+0000 7fb70c200700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52369] [GET] [303] [0.001s] [96.0B] > [76c9a6ab-aae0-4ef9-815e-dbd0ac8f8b49] /api/feature_toggles > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.527+0000 7fb703600700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.528+0000 7fb703600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52370] [GET] [303] [0.001s] [96.0B] > [db017da3-e6f6-45fc-9d99-7b9f3b3859f2] /api/health/minimal > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.541+0000 7fb6faa00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.542+0000 7fb6faa00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52371] [GET] [303] [0.001s] [96.0B] > [564ba322-b99c-49de-a3a3-e0ed422bfae7] /api/prometheus > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.616+0000 7fb6f1e00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.617+0000 7fb6f1e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52372] [GET] [303] [0.001s] [96.0B] > [1f349e52-9a10-4fda-b13f-621785208ea9] /api/prometheus/rules > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52368] [GET] [303] [0.001s] [96.0B] > [30d749d7-c972-44f0-88c7-b70078e2385c] /api/prometheus/notifications > Apr 05 20:34:19 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:19] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:19 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:19.601+0000 7fb737e00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:19] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > > cn02 > —— > Apr 05 20:33:40 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:40 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:40.701+0000 7ff710400700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:43 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:43 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:43.287+0000 7ff707800700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.370+0000 7ff7e7c00700 1 mgr handle_mgr_map > Activating! > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.371+0000 7ff7e7c00700 1 mgr handle_mgr_map I am now > activating > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root] Stopping > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root] Stopped > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root] Stopping > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root] Stopped > engine > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > STOPPING > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.383+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > STOPPING > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.394+0000 7ff68d000700 0 [dashboard INFO root] Engine > stopped. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut > down > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.395+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut > down > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > STOPPED > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > STOPPED > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO root] Engine > stopped. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts INFO root] > Init > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 1 mgr load Constructed class from module: > alerts > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 0 [balancer DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 1 mgr load Constructed class from module: > balancer > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff74d800700 0 [alerts INFO root] > Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.401+0000 7ff67aa00700 0 [cephadm DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.402+0000 7ff710400700 0 [balancer INFO root] > Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Optimize plan > auto_2025-04-05_20:33:45 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Mode upmap, max misplaced > 0.050000 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.409+0000 7ff710400700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again > later > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.551+0000 7ff67aa00700 1 mgr load Constructed class from module: > cephadm > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 0 [crash DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 1 mgr load Constructed class from module: > crash > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 1 mgr load Constructed class from module: > dashboard > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 1 mgr load Constructed class from module: > devicehealth > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO access_control] Loading user roles DB > version=2 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO sso] Loading SSO DB > version=1 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO root] server: ssl=yes host=:: > port=8443 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.557+0000 7ff674600700 0 [devicehealth INFO root] > Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.558+0000 7ff67aa00700 0 [insights DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.559+0000 7ff67aa00700 1 mgr load Constructed class from module: > insights > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.561+0000 7ff674600700 0 [devicehealth INFO root] Check > health > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.566+0000 7ff671e00700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 0 [iostat DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 1 mgr load Constructed class from module: > iostat > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.572+0000 7ff67aa00700 0 [mirroring DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.579+0000 7ff67aa00700 1 mgr load Constructed class from module: > mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 0 [nfs DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 1 mgr load Constructed class from module: > nfs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 1 mgr load Constructed class from module: > orchestrator > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 1 mgr load Constructed class from module: > pg_autoscaler > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.625+0000 7ff662a00700 0 [pg_autoscaler INFO root] > _maybe_adjust > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 0 [progress DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 1 mgr load Constructed class from module: > progress > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.72' (ECDSA) to the list of known > hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.73' (ECDSA) to the list of known > hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 1 mgr load Constructed class from module: > prometheus > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff661600700 0 [progress INFO root] > Loading... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] server_addr: :: server_port: > 9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] Cache > enabled > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff65ee00700 0 [prometheus INFO root] starting metric collection > thread > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.640+0000 7ff67aa00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7ff7443993c8>, <progress.module.GhostEvent object at 0x7ff744399400>, <progress.module.GhostEvent object at 0x7ff744399438>, <progress.module.GhostEvent object at 0x7ff744399470>, <progress.module.GhostEvent object at 0x7ff7443994a8>, <progress.module.GhostEvent object at 0x7ff7443994e0>, <progress.module.GhostEvent object at 0x7ff744399518>, <progress.module.GhostEvent object at 0x7ff744399550>, <progress.module.GhostEvent object at 0x7ff744399588>, <progress.module.GhostEvent object at 0x7ff7443995c0>, <progress.module.GhostEvent object at 0x7ff7443995f8>, <progress.module.GhostEvent object at 0x7ff744399630>, <progress.module.GhostEvent object at 0x7ff744399668>, <progress.module.GhostEvent object at 0x7ff7443996a0>, <progress.module.GhostEvent object at 0x7ff7443996d8>, <progress.module.GhostEvent object at 0x7ff744399710>, <progress.module.GhostEvent object at 0x7ff744399748>, <progress.module.GhostEvent object at 0x7ff744399780>, <progress.module.GhostEvent object at 0x7ff7443997b8>, <progress.module.GhostEvent object at 0x7ff7443997f0>, <progress.module.GhostEvent object at 0x7ff744399828>, <progress.module.GhostEvent object at 0x7ff744399860>, <progress.module.GhostEvent object at 0x7ff744399898>, <progress.module.GhostEvent object at 0x7ff7443998d0>, <progress.module.GhostEvent object at 0x7ff744399908>, <progress.module.GhostEvent object at 0x7ff744399940>, <progress.module.GhostEvent object at 0x7ff744399978>, <progress.module.GhostEvent object at 0x7ff7443999b0>, <progress.module.GhostEvent object at 0x7ff7443999e8>, <progress.module.GhostEvent object at 0x7ff744399a20>, <progress.module.GhostEvent object at 0x7ff744399a58>, <progress.module.GhostEvent object at 0x7ff744399a90>, <progress.module.GhostEvent object at 0x7ff744399ac8>, <progress.module.GhostEvent object at 0x7ff744399b00>, <progress.module.GhostEvent object at 0x7ff744399b38>, <progress.module.GhostEvent object at 0x7ff744399b70>, <progress.module.GhostEvent object at 0x7ff744399ba8>, <progress.module.GhostEvent object at 0x7ff744399be0>, <progress.module.GhostEvent object at 0x7ff744399c18>, <progress.module.GhostEvent object at 0x7ff744399c50>, <progress.module.GhostEvent object at 0x7ff744399c88>, <progress.module.GhostEvent object at 0x7ff744399cc0>, <progress.module.GhostEvent object at 0x7ff744399cf8>, <progress.module.GhostEvent object at 0x7ff744399d30>, <progress.module.GhostEvent object at 0x7ff744399d68>, <progress.module.GhostEvent object at 0x7ff744399da0>, <progress.module.GhostEvent object at 0x7ff744399dd8>, <progress.module.GhostEvent object at 0x7ff744399e10>, <progress.module.GhostEvent object at 0x7ff744399e48>, <progress.module.GhostEvent object at 0x7ff744399e80>] historic > events > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded OSDMap, > ready. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Processing OSDMap change > 27185..27185 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff676400700 0 [dashboard INFO root] Configured CherryPy, starting > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff660200700 0 [prometheus INFO root] Starting > engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > STARTING > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.647+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > STARTING > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] recovery thread > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] starting > setup > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff67aa00700 1 mgr load Constructed class from module: > rbd_support > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.71' (ECDSA) to the list of known > hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 0 [restful DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 1 mgr load Constructed class from module: > restful > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.660+0000 7ff651c00700 0 [restful INFO root] server_addr: :: server_port: > 8003 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.661+0000 7ff654e00700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: > load_schedules > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.697+0000 7ff651c00700 0 [restful WARNING root] server not running: no certificate > configured > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.705+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules: csxyz, > start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.706+0000 7ff67aa00700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.707+0000 7ff64f400700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.708+0000 7ff64ea00700 0 [rbd_support INFO root] PerfHandler: > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.709+0000 7ff654e00700 0 [rbd_support INFO root] load_task_task: csxyz, > start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.718+0000 7ff649a00700 0 [rbd_support INFO root] TaskHandler: > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.719+0000 7ff654e00700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: > load_schedules > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.721+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules: csxyz, > start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff649000700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: > starting > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff654e00700 0 [rbd_support INFO root] setup > complete > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.726+0000 7ff67aa00700 1 mgr load Constructed class from module: > snap_schedule > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 0 [status DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 1 mgr load Constructed class from module: > status > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.728+0000 7ff67aa00700 0 [telemetry DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 1 mgr load Constructed class from module: > telemetry > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 0 [volumes DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume > 'cssnfsxyz' > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume > 'cssnfsxyz' > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 1 mgr load Constructed class from module: > volumes > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > exists > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy > Checker: > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty > config. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Serving on > http://:::9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Serving on > http://:::9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > STARTED > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > STARTED > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO root] Engine > started. > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.785+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS -> > /api/cephfs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.787+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi -> > /ui-api/cephfs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi -> > /ui-api/iscsi > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi -> > /api/iscsi > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget -> > /api/iscsi/target > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster -> > /api/nfs-ganesha/cluster > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports -> > /api/nfs-ganesha/export > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi -> > /ui-api/nfs-ganesha > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd -> > /api/block/image > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdStatus -> > /ui-api/block/rbd > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot -> > /api/block/image/{image_spec}/snap > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.796+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash -> > /api/block/image/trash > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace -> > /api/block/pool/{pool_name}/namespace > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool -> > /api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.799+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi -> > /ui-api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.800+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RBDPool -> > /api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator -> > /ui-api/orchestrator > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service -> > /api/service > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring -> > /api/block/mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary -> > /api/block/mirroring/summary > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode -> > /api/block/mirroring/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap -> > /api/block/mirroring/pool/{pool_name}/bootstrap > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer -> > /api/block/mirroring/pool/{pool_name}/peer > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.804+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringStatus -> > /ui-api/block/mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw -> > /ui-api/rgw > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon -> > /api/rgw/daemon > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite -> > /api/rgw/site > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket -> > /api/rgw/bucket > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.807+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser -> > /api/rgw/user > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.810+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth -> > /api/auth > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster -> > /api/cluster > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration -> > /api/cluster_conf > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.813+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule -> > /api/crush_rule > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi -> > /ui-api/crush_rule > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon -> > /api/daemon > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs -> > /docs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile -> > /api/erasure_code_profile > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.816+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi -> > /ui-api/erasure_code_profile > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging -> > /ui-api/logging > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana -> > /api/grafana > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host -> > /api/host > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.819+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi -> > /ui-api/host > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health -> > /api/health > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController -> > / > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.822+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController -> > /ui-api/langs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LoginController -> > /ui-api/login > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs -> > /api/logs > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules -> > /api/mgr/module > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor -> > /api/monitor > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd -> > /api/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.828+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdUi -> > /ui-api/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.830+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController -> > /api/osd/flags > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter -> > /api/perf_counters/mds > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter -> > /api/perf_counters/mon > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter -> > /api/perf_counters/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter -> > /api/perf_counters/rgw > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter -> > /api/perf_counters/rbd-mirror > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter -> > /api/perf_counters/mgr > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter -> > /api/perf_counters/tcmu-runner > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters -> > /api/perf_counters > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver -> > /api/prometheus_receiver > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus -> > /api/prometheus > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications -> > /api/prometheus/notifications > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role -> > /api/role > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope -> > /ui-api/scope > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2 -> > /auth/saml2 > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings -> > /api/settings > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings -> > /ui-api/standard_settings > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary -> > /api/summary > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task -> > /api/task > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry -> > /api/telemetry > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.842+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User -> > /api/user > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy -> > /api/user > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword -> > /api/user/{username} > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint -> > /api/feature_toggles > Apr 05 20:33:45 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.844+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay -> > /ui-api/motd > Apr 05 20:33:46 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.050+0000 7ff676400700 0 [dashboard INFO dashboard.module] Engine > started. > Apr 05 20:33:46 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.378+0000 7ff67a000700 0 log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB > avail > Apr 05 20:33:47 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:47.379+0000 7ff678c00700 0 log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB > avail > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104 "" > "Prometheus/2.33.4" > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.568+0000 7ff65b200700 0 [prometheus INFO cherrypy.access.140699978276536] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104 "" > "Prometheus/2.33.4" > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 [cephadm INFO cephadm.serve] Updating > cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 log_channel(cephadm) log [INF] : Updating > cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 [cephadm INFO cephadm.serve] Updating > cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve] Updating > cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 log_channel(cephadm) log [INF] : Updating > cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] : Updating > cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 [cephadm INFO cephadm.serve] Updating > cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 log_channel(cephadm) log [INF] : Updating > cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 [cephadm INFO cephadm.serve] Updating > cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 log_channel(cephadm) log [INF] : Updating > cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve] Updating > cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] : Updating > cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.812+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 824b8fde-5a37-4436-af45-a7d8255e2c66 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 621cb660-d584-45e4-8033-4a476e804bba does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 007a4dd2-46d3-4655-9c93-c02f97e555cb does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 6b0fbff2-881e-4203-9149-f0977bc13eab does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 8a1a90b8-1f2a-48c8-924f-6e74032b6319 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.815+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3c322d26-9edb-4d26-b4bc-97c330f5a1d8 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev f3e5a517-d643-4fa5-b127-57716264e4e3 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3fb020c7-92e8-4eb1-a884-2d04e5bca889 does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.840+0000 7ff6a7400700 0 [progress WARNING root] complete: ev bf9e0705-075e-4134-b77f-aa64fe92112a does not > exist > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade myself > (mgr.cn02.ceph.xyz.corp.ggixgj) > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade myself > (mgr.cn02.ceph.xyz.corp.ggixgj) > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [progress INFO root] update: starting ev e7e1a809-558d-43a7-842a-c6229fdc57af (Upgrade to > 17.2.0) > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 [cephadm INFO cephadm.services.cephadmservice] Failing over to other > MGR > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Failing over to other > MGR > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 -1 mgr handle_mgr_map I was active but no longer > am > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn e: > '/usr/bin/ceph-mgr' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 0: > '/usr/bin/ceph-mgr' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 1: > '-n' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 2: > 'mgr.cn02.ceph.xyz.corp.ggixgj' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 3: > '-f' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 4: > '--setuser' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 5: > 'ceph' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 6: > '--setgroup' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 7: > 'ceph' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 8: > '--default-log-to-file=false' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 9: > '--default-log-to-stderr=true' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 10: '--default-log-stderr-prefix=debug > ' > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn respawning with exe > /usr/bin/ceph-mgr > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn exe_path > /proc/self/exe > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setuser ceph since I am not > root > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setgroup ceph since I am not > root > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable), process ceph-mgr, pid > 2 > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 pidfile_write: ignore empty > --pid-file > Apr 05 20:33:48 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.976+0000 7f5a66492680 1 mgr[py] Loading python module > 'alerts' > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES > member > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 1 mgr[py] Loading python module > 'balancer' > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES > member > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 1 mgr[py] Loading python module > 'cephadm' > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.568+0000 7f5a66492680 1 mgr[py] Loading python module > 'crash' > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 -1 mgr[py] Module crash has missing NOTIFY_TYPES > member > Apr 05 20:33:49 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 1 mgr[py] Loading python module > 'dashboard' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.276+0000 7f5a66492680 1 mgr[py] Loading python module > 'devicehealth' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.415+0000 7f5a66492680 1 mgr[py] Loading python module > 'diskprediction_local' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES > member > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 1 mgr[py] Loading python module > 'influx' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 -1 mgr[py] Module influx has missing NOTIFY_TYPES > member > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 1 mgr[py] Loading python module > 'insights' > Apr 05 20:33:50 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.919+0000 7f5a66492680 1 mgr[py] Loading python module > 'iostat' > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES > member > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 1 mgr[py] Loading python module > 'k8sevents' > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.533+0000 7f5a66492680 1 mgr[py] Loading python module > 'localpool' > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.634+0000 7f5a66492680 1 mgr[py] Loading python module > 'mds_autoscaler' > Apr 05 20:33:51 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.894+0000 7f5a66492680 1 mgr[py] Loading python module > 'mirroring' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.019+0000 7f5a66492680 1 mgr[py] Loading python module > 'nfs' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 1 mgr[py] Loading python module > 'orchestrator' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 1 mgr[py] Loading python module > 'osd_support' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 1 mgr[py] Loading python module > 'pg_autoscaler' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 1 mgr[py] Loading python module > 'progress' > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 -1 mgr[py] Module progress has missing NOTIFY_TYPES > member > Apr 05 20:33:52 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 1 mgr[py] Loading python module > 'prometheus' > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES > member > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 1 mgr[py] Loading python module > 'rbd_support' > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES > member > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 1 mgr[py] Loading python module > 'restful' > Apr 05 20:33:53 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.799+0000 7f5a66492680 1 mgr[py] Loading python module > 'rook' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 -1 mgr[py] Module rook has missing NOTIFY_TYPES > member > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 1 mgr[py] Loading python module > 'selftest' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES > member > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 1 mgr[py] Loading python module > 'snap_schedule' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES > member > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 1 mgr[py] Loading python module > 'stats' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.660+0000 7f5a66492680 1 mgr[py] Loading python module > 'status' > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 -1 mgr[py] Module status has missing NOTIFY_TYPES > member > Apr 05 20:33:54 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 1 mgr[py] Loading python module > 'telegraf' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 1 mgr[py] Loading python module > 'telemetry' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 1 mgr[py] Loading python module > 'test_orchestrator' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 1 mgr[py] Loading python module > 'volumes' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 1 mgr[py] Loading python module > 'zabbix' > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.792+0000 7f5a66492680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES > member > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.793+0000 7f5a52600700 0 ms_deliver_dispatch: unhandled message 0x556eb85f26e0 mon_map magic: 0 v1 from mon.2 > v2:192.168.47.72:3300/0 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from module: > dashboard > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO > (2/5) > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from module: > prometheus > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f58f7800700 0 [dashboard INFO root] server: ssl=yes host=:: > port=8443 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] server_addr: :: server_port: > 9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] Starting > engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Bus > STARTING > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Bus > STARTING > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.797+0000 7f5a52600700 0 client.0 ms_handle_reset on > v2:192.168.47.73:6842/917991990 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.825+0000 7f58f7800700 0 [dashboard INFO root] Configured CherryPy, starting > engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.826+0000 7f58f7800700 0 [dashboard INFO root] Starting > engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy > Checker: > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty > config. > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Serving on > http://:::9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Serving on > http://:::9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Bus > STARTED > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Bus > STARTED > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO root] Engine > started. > Apr 05 20:33:55 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.939+0000 7f58f7800700 0 [dashboard INFO root] Engine > started... > Apr 05 20:33:58 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:58 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:58.569+0000 7f59b8000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:00 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:00 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:00.702+0000 7f59af400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:03.286+0000 7f59a6800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:08 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:08 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:08.569+0000 7f599dc00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:10 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:10 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:10.702+0000 7f5995000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:13.286+0000 7f598c400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:18 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:18 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:18.569+0000 7f5983800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:20 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:20 cn02.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:20.702+0000 7f597ac00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > > cn03 > —— > Apr 05 20:33:42 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:42] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:42 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:42.753+0000 7f2724a00700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:42] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:43 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:43] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:43.772+0000 7f271be00700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:47 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:47] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:47 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:47.037+0000 7f2713200700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:33:47] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.867+0000 7f27d1a00700 1 mgr handle_mgr_map > Activating! > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.868+0000 7f27d1a00700 1 mgr handle_mgr_map > I am now activating > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > root] Stopping engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > root] Stopped engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > root] Stopping engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > root] Stopped engine > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.879+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.890+0000 7f2675a00700 0 [dashboard INFO > root] Engine stopped. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE HTTP Server > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.891+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE HTTP Server > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > root] Engine stopped. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts INFO root] > Init > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 1 mgr load > Constructed class from module: alerts > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f2663400700 0 [balancer DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f2663400700 1 mgr load > Constructed class from module: balancer > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f26f8e00700 0 [alerts INFO root] > Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.901+0000 7f2663400700 0 [cephadm DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.901+0000 7f26f0200700 0 [balancer INFO > root] Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > root] Optimize plan auto_2025-04-05_20:33:48 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > root] Mode upmap, max misplaced 0.050000 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO > root] Some PGs (1.000000) are unknown; try again later > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed > to construct class in 'cephadm' > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load > Traceback (most recent call last): > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.upgrade = CephadmUpgrade(self) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.upgrade_state: Optional[UpgradeState] = > UpgradeState.from_json(json.loads(t)) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > return cls(**c) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > TypeError: __init__() got an unexpected keyword argument > 'daemon_types' > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() > Failed to run module in active mode ('cephadm') > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [crash DEBUG root] > setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > Constructed class from module: crash > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [dashboard DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > Constructed class from module: dashboard > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.920+0000 7f2660c00700 0 [dashboard INFO > access_control] Loading user roles DB version=2 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > sso] Loading SSO DB version=1 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > root] server: ssl=yes host=:: port=8443 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [devicehealth > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > Constructed class from module: devicehealth > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [insights DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > Constructed class from module: insights > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.923+0000 7f2660200700 0 [devicehealth INFO > root] Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f2663400700 0 [iostat DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f2663400700 1 mgr load > Constructed class from module: iostat > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f265f800700 0 [insights INFO > root] Reset curr health slot key health_history/2025-04-05_20 next > flush None checks check count 0 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.926+0000 7f2663400700 0 [mirroring DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.931+0000 7f2663400700 1 mgr load > Constructed class from module: mirroring > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [nfs DEBUG root] > setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > Constructed class from module: nfs > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [orchestrator > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > Constructed class from module: orchestrator > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.933+0000 7f2663400700 0 [pg_autoscaler > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.933+0000 7f2663400700 1 mgr load > Constructed class from module: pg_autoscaler > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2663400700 0 [progress DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2663400700 1 mgr load > Constructed class from module: progress > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2657600700 0 [pg_autoscaler > INFO root] _maybe_adjust > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.940+0000 7f2656c00700 0 [progress INFO > root] Loading... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.940+0000 7f2663400700 0 [prometheus DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2663400700 1 mgr load > Constructed class from module: prometheus > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > root] server_addr: :: server_port: 9283 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > root] Cache enabled > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > root] Loaded [<progress.module.GhostEvent object at 0x7f27363ac400>, > <progress.module.GhostEvent object at 0x7f27363ac438>, > <progress.module.GhostEvent object at 0x7f27363ac3c8>, > <progress.module.GhostEvent object at 0x7f27363ac5c0>, > <progress.module.GhostEvent object at 0x7f27363ac4e0>, > <progress.module.GhostEvent object at 0x7f27363ac4a8>, > <progress.module.GhostEvent object at 0x7f278d317e10>, > <progress.module.GhostEvent object at 0x7f278d4397b8>, > <progress.module.GhostEvent object at 0x7f278d439908>, > <progress.module.GhostEvent object at 0x7f27363ed2e8>, > <progress.module.GhostEvent object at 0x7f27363ed4e0>, > <progress.module.GhostEvent object at 0x7f27363ed358>, > <progress.module.GhostEvent object at 0x7f27363ed278>, > <progress.module.GhostEvent object at 0x7f27363edeb8>, > <progress.module.GhostEvent object at 0x7f27363ed438>, > <progress.module.GhostEvent object at 0x7f27363ed400>, > <progress.module.GhostEvent object at 0x7f27363edb00>, > <progress.module.GhostEvent object at 0x7f27363ed780>, > <progress.module.GhostEvent object at 0x7f27363ed240>, > <progress.module.GhostEvent object at 0x7f27363edba8>, > <progress.module.GhostEvent object at 0x7f27363ed2b0>, > <progress.module.GhostEvent object at 0x7f27363ed0b8>, > <progress.module.GhostEvent object at 0x7f27363edcf8>, > <progress.module.GhostEvent object at 0x7f27363ed128>, > <progress.module.GhostEvent object at 0x7f27363ed080>, > <progress.module.GhostEvent object at 0x7f27363edc18>, > <progress.module.GhostEvent object at 0x7f27363ed860>, > <progress.module.GhostEvent object at 0x7f27363eda58>, > <progress.module.GhostEvent object at 0x7f27363eda90>, > <progress.module.GhostEvent object at 0x7f27363ed160>, > <progress.module.GhostEvent object at 0x7f27363ed908>, > <progress.module.GhostEvent object at 0x7f27363ed940>, > <progress.module.GhostEvent object at 0x7f27363ed8d0>, > <progress.module.GhostEvent object at 0x7f27363ed978>, > <progress.module.GhostEvent object at 0x7f27363ed748>, > <progress.module.GhostEvent object at 0x7f27363ed898>, > <progress.module.GhostEvent object at 0x7f27363eda20>, > <progress.module.GhostEvent object at 0x7f27363ed9b0>, > <progress.module.GhostEvent object at 0x7f27363ed9e8>, > <progress.module.GhostEvent object at 0x7f27363ed828>, > <progress.module.GhostEvent object at 0x7f27363ed7f0>, > <progress.module.GhostEvent object at 0x7f27363ed7b8>, > <progress.module.GhostEvent object at 0x7f27363ed668>, > <progress.module.GhostEvent object at 0x7f27363ed320>, > <progress.module.GhostEvent object at 0x7f27363ede48>, > <progress.module.GhostEvent object at 0x7f27363edc50>, > <progress.module.GhostEvent object at 0x7f27363ede80>, > <progress.module.GhostEvent object at 0x7f27363edf60>, > <progress.module.GhostEvent object at 0x7f27363edd68>, > <progress.module.GhostEvent object at 0x7f27363edda0>] historic events > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > root] Loaded OSDMap, ready. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.946+0000 7f2663400700 0 [rbd_support DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.946+0000 7f2655800700 0 [prometheus INFO > root] starting metric collection thread > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.947+0000 7f2656200700 0 [prometheus INFO > root] Starting engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STARTING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.948+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STARTING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.948+0000 7f2660c00700 0 [dashboard INFO > root] Configured CherryPy, starting engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.953+0000 7f2663400700 0 [rbd_support INFO > root] MirrorSnapshotScheduleHandler: load_schedules > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.957+0000 7f2663400700 0 [rbd_support INFO > root] load_schedules: csxyz, start_after= > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.989+0000 7f2649a00700 0 [rbd_support INFO > root] MirrorSnapshotScheduleHandler: starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.990+0000 7f2649000700 0 [rbd_support INFO > root] PerfHandler: starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.991+0000 7f2663400700 0 [rbd_support INFO > root] load_task_task: csxyz, start_after= > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.019+0000 7f2648600700 0 [rbd_support INFO > root] TaskHandler: starting > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.020+0000 7f2663400700 0 [rbd_support INFO > root] TrashPurgeScheduleHandler: load_schedules > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.022+0000 7f2663400700 0 [rbd_support INFO > root] load_schedules: csxyz, start_after= > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.024+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CephFS -> /api/cephfs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.026+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CephFsUi -> /ui-api/cephfs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > IscsiUi -> /ui-api/iscsi > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Iscsi -> /api/iscsi > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > IscsiTarget -> /api/iscsi/target > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGanesha -> /api/nfs-ganesha > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaCluster -> /api/nfs-ganesha/cluster > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaExports -> /api/nfs-ganesha/export > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaUi -> /ui-api/nfs-ganesha > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Rbd > -> /api/block/image > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.034+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdSnapshot -> /api/block/image/{image_spec}/snap > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdTrash -> /api/block/image/trash > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdNamespace -> /api/block/pool/{pool_name}/namespace > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroring -> /api/block/mirroring > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringSummary -> /api/block/mirroring/summary > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolMode -> /api/block/mirroring/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolBootstrap -> > /api/block/mirroring/pool/{pool_name}/bootstrap > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolPeer -> /api/block/mirroring/pool/{pool_name}/peer > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.038+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Rgw > -> /api/rgw > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwDaemon -> /api/rgw/daemon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwSite -> /api/rgw/site > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.040+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwBucket -> /api/rgw/bucket > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.041+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwUser -> /api/rgw/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Auth -> /api/auth > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Cluster -> /api/cluster > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.045+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ClusterConfiguration -> /api/cluster_conf > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.046+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CrushRule -> /api/crush_rule > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.047+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CrushRuleUi -> /ui-api/crush_rule > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Orchestrator -> /api/orchestrator > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Daemon -> /api/daemon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Docs -> /docs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ErasureCodeProfile -> /api/erasure_code_profile > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ErasureCodeProfileUi -> /ui-api/erasure_code_profile > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackController -> /api/feedback > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.051+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackApiController -> /api/feedback/api_key > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackUiController -> /ui-api/feedback/api_key > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FrontendLogging -> /ui-api/logging > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Grafana -> /api/grafana > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Host -> /api/host > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > HostUi -> /ui-api/host > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Health -> /api/health > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > HomeController -> / > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:49] ENGINE Serving on http://:::9283 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Serving on > http://:::9283 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:49] ENGINE Bus STARTED > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Bus STARTED > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > root] Engine started. > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > LangsController -> /ui-api/langs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Logs -> /api/logs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MgrModules -> /api/mgr/module > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Monitor -> /api/monitor > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Osd > -> /api/osd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.060+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > OsdFlagsController -> /api/osd/flags > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MdsPerfCounter -> /api/perf_counters/mds > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MonPerfCounter -> /api/perf_counters/mon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > OsdPerfCounter -> /api/perf_counters/osd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwPerfCounter -> /api/perf_counters/rgw > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirrorPerfCounter -> /api/perf_counters/rbd-mirror > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MgrPerfCounter -> /api/perf_counters/mgr > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > TcmuRunnerPerfCounter -> /api/perf_counters/tcmu-runner > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PerfCounters -> /api/perf_counters > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.065+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Pool -> /api/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.066+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PoolUi -> /ui-api/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PrometheusReceiver -> /api/prometheus_receiver > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Prometheus -> /api/prometheus > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.068+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PrometheusNotifications -> /api/prometheus/notifications > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.069+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Role -> /api/role > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Scope -> /ui-api/scope > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Saml2 -> /auth/saml2 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Service -> /api/service > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.071+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Settings -> /api/settings > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > StandardSettings -> /ui-api/standard_settings > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Summary -> /api/summary > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Task -> /api/task > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Telemetry -> /api/telemetry > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > User -> /api/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > UserPasswordPolicy -> /api/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > UserChangePassword -> /api/user/{username} > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeatureTogglesEndpoint -> /api/feature_toggles > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MessageOfTheDay -> /ui-api/motd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.090+0000 7f2647200700 0 [rbd_support INFO > root] TrashPurgeScheduleHandler: starting > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.096+0000 7f2663400700 1 mgr load > Constructed class from module: rbd_support > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.102+0000 7f2663400700 0 [restful DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.102+0000 7f2663400700 1 mgr load > Constructed class from module: restful > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.108+0000 7f2645e00700 0 [restful INFO > root] server_addr: :: server_port: 8003 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.109+0000 7f2663400700 0 [snap_schedule > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.116+0000 7f2645e00700 0 [restful WARNING > root] server not running: no certificate configured > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.257+0000 7f2660c00700 0 [dashboard INFO > dashboard.module] Engine started. > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed > to construct class in 'snap_schedule' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load > Traceback (most recent call last): > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in > __init__ > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.client = SnapSchedClient(self) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > 158, in __init__ > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > with self.get_schedule_db(fs_name) as conn_mgr: > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > 192, in get_schedule_db > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > db.executescript(dump) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > sqlite3.OperationalError: table schedules already exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() > Failed to run module in active mode ('snap_schedule') > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 0 [status DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 1 mgr load > Constructed class from module: status > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.275+0000 7f2663400700 0 [telemetry DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.276+0000 7f2663400700 1 mgr load > Constructed class from module: telemetry > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.276+0000 7f2663400700 0 [volumes DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.284+0000 7f2660200700 0 [devicehealth INFO > root] Check health > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 1 mgr load > Constructed class from module: volumes > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.883+0000 7f2662a00700 0 > log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail > Apr 05 20:33:50 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:50.875+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail > Apr 05 20:33:52 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:52] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:33:52 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:52.753+0000 7f264f400700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:52] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:33:52 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:52.876+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v5: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:53] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:53.772+0000 7f264ea00700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:33:53] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:53.870+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:53.871+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:53 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:53.871+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:54 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:54.877+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v6: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 0 > B/s rd, 12 KiB/s wr, 2 op/s > Apr 05 20:33:55 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:55.796+0000 7f2662a00700 2 mgr.server > handle_open ignoring open from mgr.cn02.ceph.xyz.corp.ggixgj > 192.168.47.72:0/2580067884; not ready for session (expect reconnect) > Apr 05 20:33:56 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:56.878+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v7: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 439 B/s rd, 27 KiB/s wr, 6 op/s > Apr 05 20:33:57 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:57] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:33:57 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:57.038+0000 7f264e000700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:33:57] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:33:58 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:58.870+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:58 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:58.878+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v8: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 341 B/s rd, 21 KiB/s wr, 4 op/s > Apr 05 20:33:58 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:58.897+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:58 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:58.897+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:00 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:00.880+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v9: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 279 B/s rd, 24 KiB/s wr, 6 op/s > Apr 05 20:34:02 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:02] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:34:02 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:02.753+0000 7f264d600700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:34:02 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:02.880+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v10: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:03] "GET /metrics > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:03.772+0000 7f264cc00700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 33671 "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:03.870+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:03.873+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:34:03 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:03.874+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:04 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:04.881+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v11: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 59 KiB/s wr, 11 op/s > Apr 05 20:34:06 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:06.882+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v12: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 511 B/s rd, 66 KiB/s wr, 13 op/s > Apr 05 20:34:07 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:07] "GET /metrics > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > Apr 05 20:34:07 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:07.037+0000 7f264c200700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:07] "GET /metrics HTTP/1.1" 200 267239 "" > "Prometheus/2.33.4" > Apr 05 20:34:08 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:08.871+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:34:08 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:08.883+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v13: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 55 KiB/s wr, 11 op/s > Apr 05 20:34:08 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:08.907+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:34:08 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:08.907+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:10 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:10.884+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v14: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 79 KiB/s wr, 16 op/s > Apr 05 20:34:12 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:12] "GET /metrics > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > Apr 05 20:34:12 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:12.753+0000 7f264b800700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 267239 "" > "Prometheus/2.33.4" > Apr 05 20:34:12 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:12.884+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v15: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 73 KiB/s wr, 14 op/s > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:13] "GET /metrics > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:13.772+0000 7f264ae00700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 267239 "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:13.871+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:13.883+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:34:13 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:13.883+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:14 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:14.886+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v16: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 103 KiB/s wr, 113 op/s > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.754+0000 7f2644000700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.089s] [1.7K] > [6c8ffcbf-e0d6-4f01-a24a-693d11b5fcdb] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.769+0000 7f2643600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.003s] [1.7K] > [d20dd549-caaf-4b35-9784-465556871a83] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.787+0000 7f2642c00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] > [2efb9ce6-0ec0-4525-afcf-5d04356c0f18] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.803+0000 7f2642200700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] > [2f0b6080-a99d-4fdf-b7f9-c013a4cf2b78] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.825+0000 7f2641800700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] > [5c67930f-511b-4e2a-993e-13bc0c73fe2c] / > Apr 05 20:34:15 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:15.842+0000 7f2640e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] > [043cc2d4-1863-4072-a15e-91a04422c2d3] / > Apr 05 20:34:16 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:16.887+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v17: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 66 KiB/s wr, 106 op/s > Apr 05 20:34:17 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:17] "GET /metrics > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > Apr 05 20:34:17 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:17.037+0000 7f264a400700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:17] "GET /metrics HTTP/1.1" 200 267239 "" > "Prometheus/2.33.4" > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.871+0000 7f2663400700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.887+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v18: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 54 KiB/s wr, 103 op/s > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.916+0000 7f265f800700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.916+0000 7f265f800700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.931+0000 7f2659e00700 0 [mirroring INFO > mgr_util] scanning for idle connections.. > Apr 05 20:34:18 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:18.931+0000 7f2659e00700 0 [mirroring INFO > mgr_util] cleaning up connections: [] > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.109+0000 7f2645400700 0 [snap_schedule > INFO mgr_util] scanning for idle connections.. > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.109+0000 7f2645400700 0 [snap_schedule > INFO mgr_util] cleaning up connections: [] > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.277+0000 7f2632800700 0 [volumes INFO > mgr_util] scanning for idle connections.. > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.277+0000 7f2632800700 0 [volumes INFO > mgr_util] cleaning up connections: [] > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.282+0000 7f262d800700 0 [volumes INFO > mgr_util] scanning for idle connections.. > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.282+0000 7f262d800700 0 [volumes INFO > mgr_util] cleaning up connections: [] > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.287+0000 7f2629c00700 0 [volumes INFO > mgr_util] scanning for idle connections.. > Apr 05 20:34:19 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:19.287+0000 7f2629c00700 0 [volumes INFO > mgr_util] cleaning up connections: [] > Apr 05 20:34:20 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:20.889+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v19: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 70 KiB/s wr, 107 op/s > Apr 05 20:34:22 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:22] "GET /metrics > HTTP/1.1" 200 267240 "" "Prometheus/2.33.4" > Apr 05 20:34:22 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:22.753+0000 7f264fe00700 0 [prometheus INFO > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:22] "GET /metrics HTTP/1.1" 200 267240 "" > "Prometheus/2.33.4" > Apr 05 20:34:22 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:34:22.889+0000 7f2661600700 0 > log_channel(cluster) log [DBG] : pgmap v20: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 82 KiB/s rd, 45 KiB/s wr, 102 op/s > > Thank you > -jeremy > >> On Saturday, Apr 05, 2025 at 1:32 PM, Eugen Block <eblock(a)nde.ag >> (mailto:eblock@nde.ag)> wrote: >> The mgr logs should contain a stack trace, can you check again? >> >> >> Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: >> >> > No. Same issue unfortunately. >> > >> > > On Saturday, Apr 05, 2025 at 1:27 PM, Eugen Block <eblock(a)nde.ag >> > > (mailto:eblock@nde.ag)> wrote: >> > > I don't see the module error in the logs you provided. Has it cleared? >> > > A mgr failover is often helpful in such cases, basically it's the >> > > first thing I suggest since two or three years. >> > > >> > > Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: >> > > >> > > > Thank you so much for the detailed instructions. Here’s logs from >> > > > the failover to a new node. >> > > > >> > > > Apr 05 20:06:08 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 >> - >> > > "" >> > > > "Prometheus/2.33.4" >> > > > Apr 05 20:06:08 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:08.568+0000 7efcbd800700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 >> - >> > > "" >> > > > "Prometheus/2.33.4" >> > > > Apr 05 20:06:10 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 >> - >> > > "" >> > > > "Prometheus/2.33.4" >> > > > Apr 05 20:06:10 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:10.702+0000 7efcb4c00700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 >> - >> > > "" >> > > > "Prometheus/2.33.4" >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.027+0000 7efd95200700 1 >> mgr >> > > handle_mgr_map >> > > > Activating! >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.028+0000 7efd95200700 1 mgr handle_mgr_map I >> am >> > > now >> > > > activating >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO >> root] >> > > Stopping >> > > > engine... >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO >> root] >> > > Stopped >> > > > engine... >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO >> root] >> > > Stopping >> > > > engine... >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO >> root] >> > > Stopped >> > > > engine >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] >> ENGINE >> > > Bus >> > > > STOPPING >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] >> ENGINE >> > > Bus >> > > > STOPPING >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.053+0000 7efc3a400700 0 [dashboard INFO >> root] >> > > Engine >> > > > stopped. >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', >> 9283)) >> > > shut >> > > > down >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', >> 9283)) >> > > shut >> > > > down >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] >> ENGINE >> > > Bus >> > > > STOPPED >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] >> ENGINE >> > > Bus >> > > > STOPPED >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO >> root] >> > > Engine >> > > > stopped. >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.056+0000 7efc27e00700 0 [alerts DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 0 [alerts >> INFO >> > > root] >> > > > Init >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > alerts >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 0 [balancer DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > balancer >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efcfac00700 0 [alerts >> INFO >> > > root] >> > > > Starting >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efc27e00700 0 [cephadm DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efcb4c00700 0 [balancer >> INFO >> > > root] >> > > > Starting >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] >> Optimize >> > > plan >> > > > auto_2025-04-05_20:06:12 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Mode upmap, >> max >> > > misplaced >> > > > 0.050000 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Some PGs (1.000000) are unknown; >> try >> > > again >> > > > later >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.212+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > cephadm >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 0 [crash DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > crash >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 0 [dashboard DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > dashboard >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.216+0000 7efc23800700 0 [dashboard INFO access_control] Loading user >> roles >> > > DB >> > > > version=2 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO sso] Loading >> SSO >> > > DB >> > > > version=1 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO root] server: >> ssl=yes >> > > host=:: >> > > > port=8443 >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 0 [devicehealth DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > devicehealth >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 0 [insights DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > insights >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc20600700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:06:12 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc21a00700 0 [devicehealth >> INFO >> > > root] >> > > > Starting >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 0 [iostat DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > iostat >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.224+0000 7efc27e00700 0 [mirroring DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.227+0000 7efc21a00700 0 [devicehealth INFO >> root] >> > > Check >> > > > health >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.235+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > mirroring >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 0 [nfs DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > nfs >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 0 [orchestrator DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > orchestrator >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 0 [pg_autoscaler DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > pg_autoscaler >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc10200700 0 [pg_autoscaler >> INFO >> > > root] >> > > > _maybe_adjust >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 0 [progress DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > progress >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.247+0000 7efc27e00700 0 [prometheus DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.248+0000 7efc27e00700 1 mgr load Constructed class >> from >> > > module: >> > > > prometheus >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc0ee00700 0 [progress >> INFO >> > > root] >> > > > Loading... >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc27e00700 0 [rbd_support DEBUG root] setting log level based on >> debug_mgr: >> > > INFO >> > > > (2/5) >> > > > Apr 05 20:06:12 cn02.ceph.xyz.corp >> > > > >> > > >> ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7efcf2127710>, <progress.module.GhostEvent object at 0x7efcf2127cc0>, <progress.module.GhostEvent object at 0x7efcf2127208>, <progress.module.GhostEvent object at 0x7efcf2127f28>, <progress.module.GhostEvent object at 0x7efcf21270f0>, <progress.module.GhostEvent object at 0x7efcf2127eb8>, <progress.module.GhostEvent object at 0x7efcf2127b38>, <progress.module.GhostEvent object at 0x7efcf2127a20>, <progress.module.GhostEvent object at 0x7efcf2127da0>, <progress.module.GhostEvent object at 0x7efcf2127f98>, <progress.module.GhostEvent object at 0x7efcf2127780>, <progress.module.GhostEvent object at 0x7efcf2127320>, <progress.module.GhostEvent object at 0x7efcf2127588>, <progress.module.GhostEvent object at 0x7efcf2127518>, <progress.module.GhostEvent object at 0x7efcf2127c50>, <progress.module.GhostEvent object at 0x7efcf21279e8>, <progress.module.GhostEvent object at 0x7efcf2127d68>, <progress.module.GhostEvent object at 0x7efcf2127470>, <progress.module.GhostEvent object at >> 0x7efcf > [Text gekürzt]

2 6

Re: Cephadm upgrade from 16.2.15 -> 17.2.0
by Eugen Block 05 Apr '25

05 Apr '25

Please don't drop the list from your response. The orchestrator might be down, but you should have an active mgr (ceph -s). On the host running the mgr, you can just run 'cephadm logs --name mgr.<your_mgr>', or 'journalctl -u ceph-<ceph_fsid>@mgr.<your_mgr>'. To get fresh logs, you can just run 'ceph mgr fail', look on which host a different mgr takes over and gather logs from there. Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: > Thanks for the clarification on version jumps. I interpreted the > docs to mean that I had to incrementally step through each release. > This will save a lot of time. > > Since the orchestrator is down and I’m not sure how to trigger a > failure, I don’t know how to gather those logs. I apologize. I > appreciate your assistance. > > -jeremy > >> On Saturday, Apr 05, 2025 at 4:01 AM, Eugen Block <eblock(a)nde.ag >> (mailto:eblock@nde.ag)> wrote: >> It might be a different issue, can you paste the entire stack trace >> from the mgr when it's failing? >> >> Also, you could go directly from Pacific to Reef, there's no need to >> upgrade to Quincy (which is EOL). And I would also recommend to >> upgrade to a latest major version, e.g. 17.2.8, not .0, otherwise you >> might hit bugs which have already been fixed in later releases. >> >> Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: >> >> > I ran in to the “Error ENOENT: Module not found” issue with the >> > orchestrator. I see the note in the cephadm upgrade docs but I don’t >> > quite know what action to take to fix this: >> > >> > ceph versions >> > { >> > "mon": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 3 >> > }, >> > "mgr": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 1, >> > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) >> > quincy (stable)": 2 >> > }, >> > "osd": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 15 >> > }, >> > "mds": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 3 >> > }, >> > "overall": { >> > "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) >> > pacific (stable)": 22, >> > "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) >> > quincy (stable)": 2 >> > } >> > } >> > >> > ceph orch upgrade status >> > Error ENOENT: Module not found >> > >> > ceph config-key get mgr/cephadm/osd_remove_queue >> > [] >> > >> > Thanks. I’m trying to upgrade this cluster all the way to current. >> > >> > -jeremy >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 5

Jump to page: