Would I follow this process to downgrade? https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-... Thank you

On Saturday, Apr 05, 2025 at 2:04 PM, Jeremy Hansen <jeremy@skidrow.la (mailto:jeremy@skidrow.la)> wrote: ceph -s claims things are healthy:

ceph -s cluster: id: 95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1 health: HEALTH_OK

services: mon: 3 daemons, quorum cn01,cn03,cn02 (age 20h) mgr: cn03.negzvb(active, since 26m), standbys: cn01.tjmtph, cn02.ceph.xyz.corp.ggixgj mds: 1/1 daemons up, 2 standby osd: 15 osds: 15 up (since 19h), 15 in (since 14M)

data: volumes: 1/1 healthy pools: 6 pools, 610 pgs objects: 284.59k objects, 1.1 TiB usage: 3.3 TiB used, 106 TiB / 109 TiB avail pgs: 610 active+clean

io: client: 255 B/s rd, 1.2 MiB/s wr, 10 op/s rd, 16 op/s wr

— How do I downgrade if the orch is down?

Thank you -jeremy

...

On Saturday, Apr 05, 2025 at 1:56 PM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: It would help if you only pasted the relevant parts. Anyway, these two sections stand out:

---snip--- Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again later Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed to construct class in 'cephadm' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade = CephadmUpgrade(self) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: return cls(**c) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: TypeError: __init__() got an unexpected keyword argument 'daemon_types' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('cephadm')

Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed to construct class in 'snap_schedule' Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.client = SnapSchedClient(self) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 158, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: with self.get_schedule_db(fs_name) as conn_mgr: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 192, in get_schedule_db Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: db.executescript(dump) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: sqlite3.OperationalError: table schedules already exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('snap_schedule') ---snip---

Your cluster seems to be in an error state (ceph -s) because of an unknown PG. It's recommended to have a healthy cluster before attemping an upgrade. It's possible that these errors come from the not upgraded MGR, I'm not sure.

Since the upgrade was only successful for two MGRs, I am thinking about downgrading both MGRs back to 16.2.15, then retry an upgrade to a newer version, either 17.2.8 or 18.2.4. I haven't checked the snap_schedule error yet, though. Maybe someone else knows that already.

Zitat von Jeremy Hansen <jeremy@skidrow.la>:

...

This is a small three node cluster. Here’s logs from all three nodes during the failover. The current manager was cn02 and it failed over to cn03.

cn01 —— Apr 05 20:33:35 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:35.823+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v821: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 22 KiB/s wr, 5 op/s Apr 05 20:33:37 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:37.170+0000 7f2aaec00700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:33:37 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:33:37 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:33:37 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:37.823+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v822: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 22 KiB/s wr, 5 op/s Apr 05 20:33:39 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:39] "GET /metrics HTTP/1.1" 200 267241 "" "Prometheus/2.33.4" Apr 05 20:33:39 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:39.825+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v823: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 97 KiB/s wr, 12 op/s Apr 05 20:33:41 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:41.826+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v824: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 85 KiB/s wr, 10 op/s Apr 05 20:33:42 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:42.170+0000 7f2aaec00700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:33:42 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:33:42 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:33:42 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:42] "GET /metrics HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" Apr 05 20:33:43 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" Apr 05 20:33:43 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:43.827+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v825: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 511 B/s rd, 133 KiB/s wr, 18 op/s Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 -1 mgr handle_mgr_map I was active but no longer am Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn e: '/usr/bin/ceph-mgr' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 0: '/usr/bin/ceph-mgr' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 1: '-n' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 2: 'mgr.cn01.tjmtph' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 3: '-f' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 4: '--setuser' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 5: 'ceph' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 6: '--setgroup' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 7: 'ceph' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 8: '--default-log-to-file=false' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 9: '--default-log-to-stderr=true' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 10: '--default-log-stderr-prefix=debug ' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn respawning with exe /usr/bin/ceph-mgr Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn exe_path /proc/self/exe Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ignoring --setuser ceph since I am not root Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ignoring --setgroup ceph since I am not root Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mgr, pid 2 Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 pidfile_write: ignore empty --pid-file Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.477+0000 7fb814447000 1 mgr[py] Loading python module 'alerts' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.612+0000 7fb814447000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.612+0000 7fb814447000 1 mgr[py] Loading python module 'balancer' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.726+0000 7fb814447000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.726+0000 7fb814447000 1 mgr[py] Loading python module 'cephadm' Apr 05 20:33:46 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:46.417+0000 7fb814447000 1 mgr[py] Loading python module 'crash' Apr 05 20:33:46 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:46.540+0000 7fb814447000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member Apr 05 20:33:46 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:46.540+0000 7fb814447000 1 mgr[py] Loading python module 'dashboard' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.210+0000 7fb814447000 1 mgr[py] Loading python module 'devicehealth' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.331+0000 7fb814447000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.331+0000 7fb814447000 1 mgr[py] Loading python module 'diskprediction_local' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.653+0000 7fb814447000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.654+0000 7fb814447000 1 mgr[py] Loading python module 'influx' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.769+0000 7fb814447000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.769+0000 7fb814447000 1 mgr[py] Loading python module 'insights' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.884+0000 7fb814447000 1 mgr[py] Loading python module 'iostat' Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.082+0000 7fb814447000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.082+0000 7fb814447000 1 mgr[py] Loading python module 'k8sevents' Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.679+0000 7fb814447000 1 mgr[py] Loading python module 'localpool' Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.796+0000 7fb814447000 1 mgr[py] Loading python module 'mds_autoscaler' Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.983+0000 7fb814447000 1 mgr[py] Loading python module 'mirroring' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.118+0000 7fb814447000 1 mgr[py] Loading python module 'nfs' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.315+0000 7fb814447000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.315+0000 7fb814447000 1 mgr[py] Loading python module 'orchestrator' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.607+0000 7fb814447000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.607+0000 7fb814447000 1 mgr[py] Loading python module 'osd_perf_query' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.723+0000 7fb814447000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.723+0000 7fb814447000 1 mgr[py] Loading python module 'osd_support' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.837+0000 7fb814447000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.837+0000 7fb814447000 1 mgr[py] Loading python module 'pg_autoscaler' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.970+0000 7fb814447000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.970+0000 7fb814447000 1 mgr[py] Loading python module 'progress' Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.082+0000 7fb814447000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.082+0000 7fb814447000 1 mgr[py] Loading python module 'prometheus' Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.617+0000 7fb814447000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.617+0000 7fb814447000 1 mgr[py] Loading python module 'rbd_support' Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.761+0000 7fb814447000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.761+0000 7fb814447000 1 mgr[py] Loading python module 'restful' Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.023+0000 7fb814447000 1 mgr[py] Loading python module 'rook' Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.788+0000 7fb814447000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.788+0000 7fb814447000 1 mgr[py] Loading python module 'selftest' Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.912+0000 7fb814447000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.912+0000 7fb814447000 1 mgr[py] Loading python module 'snap_schedule' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.038+0000 7fb814447000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.038+0000 7fb814447000 1 mgr[py] Loading python module 'stats' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.155+0000 7fb814447000 1 mgr[py] Loading python module 'status' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.280+0000 7fb814447000 -1 mgr[py] Module status has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.280+0000 7fb814447000 1 mgr[py] Loading python module 'telegraf' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.398+0000 7fb814447000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.398+0000 7fb814447000 1 mgr[py] Loading python module 'telemetry' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.594+0000 7fb814447000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.594+0000 7fb814447000 1 mgr[py] Loading python module 'test_orchestrator' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.771+0000 7fb814447000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.771+0000 7fb814447000 1 mgr[py] Loading python module 'volumes' Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.225+0000 7fb814447000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.225+0000 7fb814447000 1 mgr[py] Loading python module 'zabbix' Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.342+0000 7fb814447000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.344+0000 7fb807c00700 0 ms_deliver_dispatch: unhandled message 0x555c78fe1080 mon_map magic: 0 v1 from mon.0 v2:192.168.47.71:3300/0 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.344+0000 7fb808600700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load Constructed class from module: dashboard Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.345+0000 7fb808600700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load Constructed class from module: prometheus Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.345+0000 7fb6abe00700 0 [dashboard INFO root] server: ssl=yes host=:: port=8443 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO root] server_addr: :: server_port: 9283 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO root] Starting engine... Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: [05/Apr/2025:20:33:53] ENGINE Bus STARTING Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTING Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO root] Configured CherryPy, starting engine... Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO root] Starting engine... Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: CherryPy Checker: Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: The Application mounted at '' has an empty config. Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: [05/Apr/2025:20:33:53] ENGINE Serving on http://:::9283 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Serving on http://:::9283 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: [05/Apr/2025:20:33:53] ENGINE Bus STARTED Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTED Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.451+0000 7fb6b4a00700 0 [prometheus INFO root] Engine started. Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.489+0000 7fb6abe00700 0 [dashboard INFO root] Engine started... Apr 05 20:33:59 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:59 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:59.603+0000 7fb76c600700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:02 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:02 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:02.476+0000 7fb763a00700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:03 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:03 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:03.061+0000 7fb75ae00700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:09 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:09 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:09.602+0000 7fb752200700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:12 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:12 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:12.476+0000 7fb749600700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:13 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:13 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:13.060+0000 7fb740a00700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.493+0000 7fb714e00700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.494+0000 7fb714e00700 0 [dashboard INFO request] [::ffff:192.168.10.101:52368] [GET] [303] [0.003s] [96.0B] [0031fbdd-947f-4c6c-943d-55cf256edfba] /api/summary Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.512+0000 7fb70c200700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.513+0000 7fb70c200700 0 [dashboard INFO request] [::ffff:192.168.10.101:52369] [GET] [303] [0.001s] [96.0B] [76c9a6ab-aae0-4ef9-815e-dbd0ac8f8b49] /api/feature_toggles Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.527+0000 7fb703600700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.528+0000 7fb703600700 0 [dashboard INFO request] [::ffff:192.168.10.101:52370] [GET] [303] [0.001s] [96.0B] [db017da3-e6f6-45fc-9d99-7b9f3b3859f2] /api/health/minimal Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.541+0000 7fb6faa00700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.542+0000 7fb6faa00700 0 [dashboard INFO request] [::ffff:192.168.10.101:52371] [GET] [303] [0.001s] [96.0B] [564ba322-b99c-49de-a3a3-e0ed422bfae7] /api/prometheus Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.616+0000 7fb6f1e00700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.617+0000 7fb6f1e00700 0 [dashboard INFO request] [::ffff:192.168.10.101:52372] [GET] [303] [0.001s] [96.0B] [1f349e52-9a10-4fda-b13f-621785208ea9] /api/prometheus/rules Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO request] [::ffff:192.168.10.101:52368] [GET] [303] [0.001s] [96.0B] [30d749d7-c972-44f0-88c7-b70078e2385c] /api/prometheus/notifications Apr 05 20:34:19 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:19 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:19.601+0000 7fb737e00700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4"

cn02 —— Apr 05 20:33:40 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:40 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:40.701+0000 7ff710400700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:43 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:43 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:43.287+0000 7ff707800700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.370+0000 7ff7e7c00700 1 mgr handle_mgr_map Activating! Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.371+0000 7ff7e7c00700 1 mgr handle_mgr_map I am now activating Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root] Stopping engine... Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root] Stopped engine... Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root] Stopping engine... Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root] Stopped engine Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus STOPPING Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.383+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus STOPPING Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.394+0000 7ff68d000700 0 [dashboard INFO root] Engine stopped. Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.395+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus STOPPED Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus STOPPED Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO root] Engine stopped. Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts INFO root] Init Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 1 mgr load Constructed class from module: alerts Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 0 [balancer DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 1 mgr load Constructed class from module: balancer Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff74d800700 0 [alerts INFO root] Starting Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.401+0000 7ff67aa00700 0 [cephadm DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.402+0000 7ff710400700 0 [balancer INFO root] Starting Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Optimize plan auto_2025-04-05_20:33:45 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Mode upmap, max misplaced 0.050000 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.409+0000 7ff710400700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again later Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.551+0000 7ff67aa00700 1 mgr load Constructed class from module: cephadm Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 0 [crash DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 1 mgr load Constructed class from module: crash Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 1 mgr load Constructed class from module: dashboard Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 1 mgr load Constructed class from module: devicehealth Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO access_control] Loading user roles DB version=2 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO sso] Loading SSO DB version=1 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO root] server: ssl=yes host=:: port=8443 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.557+0000 7ff674600700 0 [devicehealth INFO root] Starting Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.558+0000 7ff67aa00700 0 [insights DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.559+0000 7ff67aa00700 1 mgr load Constructed class from module: insights Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.561+0000 7ff674600700 0 [devicehealth INFO root] Check health Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.566+0000 7ff671e00700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 0 [iostat DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 1 mgr load Constructed class from module: iostat Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.572+0000 7ff67aa00700 0 [mirroring DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.579+0000 7ff67aa00700 1 mgr load Constructed class from module: mirroring Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 0 [nfs DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 1 mgr load Constructed class from module: nfs Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 1 mgr load Constructed class from module: orchestrator Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 1 mgr load Constructed class from module: pg_autoscaler Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.625+0000 7ff662a00700 0 [pg_autoscaler INFO root] _maybe_adjust Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 0 [progress DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 1 mgr load Constructed class from module: progress Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.72' (ECDSA) to the list of known hosts. Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.73' (ECDSA) to the list of known hosts. Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 1 mgr load Constructed class from module: prometheus Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff661600700 0 [progress INFO root] Loading... Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] server_addr: :: server_port: 9283 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] Cache enabled Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff65ee00700 0 [prometheus INFO root] starting metric collection thread Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.640+0000 7ff67aa00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7ff7443993c8>, <progress.module.GhostEvent object at 0x7ff744399400>, <progress.module.GhostEvent object at 0x7ff744399438>, <progress.module.GhostEvent object at 0x7ff744399470>, <progress.module.GhostEvent object at 0x7ff7443994a8>, <progress.module.GhostEvent object at 0x7ff7443994e0>, <progress.module.GhostEvent object at 0x7ff744399518>, <progress.module.GhostEvent object at 0x7ff744399550>, <progress.module.GhostEvent object at 0x7ff744399588>, <progress.module.GhostEvent object at 0x7ff7443995c0>, <progress.module.GhostEvent object at 0x7ff7443995f8>, <progress.module.GhostEvent object at 0x7ff744399630>, <progress.module.GhostEvent object at 0x7ff744399668>, <progress.module.GhostEvent object at 0x7ff7443996a0>, <progress.module.GhostEvent object at 0x7ff7443996d8>, <progress.module.GhostEvent object at 0x7ff744399710>, <progress.module.GhostEvent object at 0x7ff744399748>, <progress.module.GhostEvent object at 0x7ff744399780>, <progress.module.GhostEvent object at 0x7ff7443997b8>, <progress.module.GhostEvent object at 0x7ff7443997f0>, <progress.module.GhostEvent object at 0x7ff744399828>, <progress.module.GhostEvent object at 0x7ff744399860>, <progress.module.GhostEvent object at 0x7ff744399898>, <progress.module.GhostEvent object at 0x7ff7443998d0>, <progress.module.GhostEvent object at 0x7ff744399908>, <progress.module.GhostEvent object at 0x7ff744399940>, <progress.module.GhostEvent object at 0x7ff744399978>, <progress.module.GhostEvent object at 0x7ff7443999b0>, <progress.module.GhostEvent object at 0x7ff7443999e8>, <progress.module.GhostEvent object at 0x7ff744399a20>, <progress.module.GhostEvent object at 0x7ff744399a58>, <progress.module.GhostEvent object at 0x7ff744399a90>, <progress.module.GhostEvent object at 0x7ff744399ac8>, <progress.module.GhostEvent object at 0x7ff744399b00>, <progress.module.GhostEvent object at 0x7ff744399b38>, <progress.module.GhostEvent object at 0x7ff744399b70>, <progress.module.GhostEvent object at 0x7ff744399ba8>, <progress.module.GhostEvent object at 0x7ff744399be0>, <progress.module.GhostEvent object at 0x7ff744399c18>, <progress.module.GhostEvent object at 0x7ff744399c50>, <progress.module.GhostEvent object at 0x7ff744399c88>, <progress.module.GhostEvent object at 0x7ff744399cc0>, <progress.module.GhostEvent object at 0x7ff744399cf8>, <progress.module.GhostEvent object at 0x7ff744399d30>, <progress.module.GhostEvent object at 0x7ff744399d68>, <progress.module.GhostEvent object at 0x7ff744399da0>, <progress.module.GhostEvent object at 0x7ff744399dd8>, <progress.module.GhostEvent object at 0x7ff744399e10>, <progress.module.GhostEvent object at 0x7ff744399e48>, <progress.module.GhostEvent object at 0x7ff744399e80>] historic events Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded OSDMap, ready. Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Processing OSDMap change 27185..27185 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff676400700 0 [dashboard INFO root] Configured CherryPy, starting engine... Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff660200700 0 [prometheus INFO root] Starting engine... Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus STARTING Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.647+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus STARTING Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] recovery thread starting Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] starting setup Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff67aa00700 1 mgr load Constructed class from module: rbd_support Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.71' (ECDSA) to the list of known hosts. Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 0 [restful DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 1 mgr load Constructed class from module: restful Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.660+0000 7ff651c00700 0 [restful INFO root] server_addr: :: server_port: 8003 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.661+0000 7ff654e00700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.697+0000 7ff651c00700 0 [restful WARNING root] server not running: no certificate configured Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.705+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules: csxyz, start_after= Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.706+0000 7ff67aa00700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.707+0000 7ff64f400700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: starting Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.708+0000 7ff64ea00700 0 [rbd_support INFO root] PerfHandler: starting Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.709+0000 7ff654e00700 0 [rbd_support INFO root] load_task_task: csxyz, start_after= Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.718+0000 7ff649a00700 0 [rbd_support INFO root] TaskHandler: starting Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.719+0000 7ff654e00700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.721+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules: csxyz, start_after= Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff649000700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: starting Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff654e00700 0 [rbd_support INFO root] setup complete Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.726+0000 7ff67aa00700 1 mgr load Constructed class from module: snap_schedule Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 0 [status DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 1 mgr load Constructed class from module: status Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.728+0000 7ff67aa00700 0 [telemetry DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 1 mgr load Constructed class from module: telemetry Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 0 [volumes DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume 'cssnfsxyz' Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume 'cssnfsxyz' Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 1 mgr load Constructed class from module: volumes Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy Checker: Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty config. Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Serving on http://:::9283 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Serving on http://:::9283 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus STARTED Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus STARTED Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO root] Engine started. Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.785+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS -> /api/cephfs Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.787+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi -> /ui-api/cephfs Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi -> /ui-api/iscsi Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi -> /api/iscsi Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget -> /api/iscsi/target Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster -> /api/nfs-ganesha/cluster Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports -> /api/nfs-ganesha/export Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi -> /ui-api/nfs-ganesha Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd -> /api/block/image Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdStatus -> /ui-api/block/rbd Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot -> /api/block/image/{image_spec}/snap Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.796+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash -> /api/block/image/trash Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace -> /api/block/pool/{pool_name}/namespace Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool -> /api/pool Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.799+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi -> /ui-api/pool Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.800+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RBDPool -> /api/pool Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator -> /ui-api/orchestrator Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service -> /api/service Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring -> /api/block/mirroring Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary -> /api/block/mirroring/summary Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode -> /api/block/mirroring/pool Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap -> /api/block/mirroring/pool/{pool_name}/bootstrap Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer -> /api/block/mirroring/pool/{pool_name}/peer Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.804+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringStatus -> /ui-api/block/mirroring Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw -> /ui-api/rgw Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon -> /api/rgw/daemon Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite -> /api/rgw/site Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket -> /api/rgw/bucket Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.807+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser -> /api/rgw/user Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.810+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth -> /api/auth Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster -> /api/cluster Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration -> /api/cluster_conf Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.813+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule -> /api/crush_rule Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi -> /ui-api/crush_rule Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon -> /api/daemon Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs -> /docs Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile -> /api/erasure_code_profile Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.816+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi -> /ui-api/erasure_code_profile Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging -> /ui-api/logging Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana -> /api/grafana Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host -> /api/host Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.819+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi -> /ui-api/host Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health -> /api/health Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController -> / Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.822+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController -> /ui-api/langs Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LoginController -> /ui-api/login Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs -> /api/logs Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules -> /api/mgr/module Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor -> /api/monitor Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd -> /api/osd Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.828+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdUi -> /ui-api/osd Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.830+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController -> /api/osd/flags Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter -> /api/perf_counters/mds Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter -> /api/perf_counters/mon Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter -> /api/perf_counters/osd Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter -> /api/perf_counters/rgw Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter -> /api/perf_counters/rbd-mirror Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter -> /api/perf_counters/mgr Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter -> /api/perf_counters/tcmu-runner Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters -> /api/perf_counters Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver -> /api/prometheus_receiver Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus -> /api/prometheus Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications -> /api/prometheus/notifications Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role -> /api/role Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope -> /ui-api/scope Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2 -> /auth/saml2 Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings -> /api/settings Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings -> /ui-api/standard_settings Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary -> /api/summary Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task -> /api/task Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry -> /api/telemetry Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.842+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User -> /api/user Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy -> /api/user Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword -> /api/user/{username} Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint -> /api/feature_toggles Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.844+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay -> /ui-api/motd Apr 05 20:33:46 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.050+0000 7ff676400700 0 [dashboard INFO dashboard.module] Engine started. Apr 05 20:33:46 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.378+0000 7ff67a000700 0 log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail Apr 05 20:33:47 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:47.379+0000 7ff678c00700 0 log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104 "" "Prometheus/2.33.4" Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.568+0000 7ff65b200700 0 [prometheus INFO cherrypy.access.140699978276536] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104 "" "Prometheus/2.33.4" Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 [cephadm INFO cephadm.serve] Updating cn01.ceph.xyz.corp:/etc/ceph/ceph.conf Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 log_channel(cephadm) log [INF] : Updating cn01.ceph.xyz.corp:/etc/ceph/ceph.conf Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 [cephadm INFO cephadm.serve] Updating cn02.ceph.xyz.corp:/etc/ceph/ceph.conf Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve] Updating cn03.ceph.xyz.corp:/etc/ceph/ceph.conf Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 log_channel(cephadm) log [INF] : Updating cn02.ceph.xyz.corp:/etc/ceph/ceph.conf Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] : Updating cn03.ceph.xyz.corp:/etc/ceph/ceph.conf Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 [cephadm INFO cephadm.serve] Updating cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 log_channel(cephadm) log [INF] : Updating cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 [cephadm INFO cephadm.serve] Updating cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 log_channel(cephadm) log [INF] : Updating cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve] Updating cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] : Updating cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.812+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 824b8fde-5a37-4436-af45-a7d8255e2c66 does not exist Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 621cb660-d584-45e4-8033-4a476e804bba does not exist Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 007a4dd2-46d3-4655-9c93-c02f97e555cb does not exist Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 6b0fbff2-881e-4203-9149-f0977bc13eab does not exist Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 8a1a90b8-1f2a-48c8-924f-6e74032b6319 does not exist Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.815+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3c322d26-9edb-4d26-b4bc-97c330f5a1d8 does not exist Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev f3e5a517-d643-4fa5-b127-57716264e4e3 does not exist Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3fb020c7-92e8-4eb1-a884-2d04e5bca889 does not exist Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.840+0000 7ff6a7400700 0 [progress WARNING root] complete: ev bf9e0705-075e-4134-b77f-aa64fe92112a does not exist Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade myself (mgr.cn02.ceph.xyz.corp.ggixgj) Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade myself (mgr.cn02.ceph.xyz.corp.ggixgj) Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [progress INFO root] update: starting ev e7e1a809-558d-43a7-842a-c6229fdc57af (Upgrade to 17.2.0) Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 [cephadm INFO cephadm.services.cephadmservice] Failing over to other MGR Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Failing over to other MGR Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 -1 mgr handle_mgr_map I was active but no longer am Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn e: '/usr/bin/ceph-mgr' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 0: '/usr/bin/ceph-mgr' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 1: '-n' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 2: 'mgr.cn02.ceph.xyz.corp.ggixgj' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 3: '-f' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 4: '--setuser' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 5: 'ceph' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 6: '--setgroup' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 7: 'ceph' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 8: '--default-log-to-file=false' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 9: '--default-log-to-stderr=true' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 10: '--default-log-stderr-prefix=debug ' Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn respawning with exe /usr/bin/ceph-mgr Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn exe_path /proc/self/exe Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setuser ceph since I am not root Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setgroup ceph since I am not root Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable), process ceph-mgr, pid 2 Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 pidfile_write: ignore empty --pid-file Apr 05 20:33:48 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.976+0000 7f5a66492680 1 mgr[py] Loading python module 'alerts' Apr 05 20:33:49 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member Apr 05 20:33:49 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 1 mgr[py] Loading python module 'balancer' Apr 05 20:33:49 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member Apr 05 20:33:49 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 1 mgr[py] Loading python module 'cephadm' Apr 05 20:33:49 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.568+0000 7f5a66492680 1 mgr[py] Loading python module 'crash' Apr 05 20:33:49 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member Apr 05 20:33:49 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 1 mgr[py] Loading python module 'dashboard' Apr 05 20:33:50 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.276+0000 7f5a66492680 1 mgr[py] Loading python module 'devicehealth' Apr 05 20:33:50 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.415+0000 7f5a66492680 1 mgr[py] Loading python module 'diskprediction_local' Apr 05 20:33:50 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member Apr 05 20:33:50 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 1 mgr[py] Loading python module 'influx' Apr 05 20:33:50 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member Apr 05 20:33:50 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 1 mgr[py] Loading python module 'insights' Apr 05 20:33:50 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.919+0000 7f5a66492680 1 mgr[py] Loading python module 'iostat' Apr 05 20:33:51 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member Apr 05 20:33:51 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 1 mgr[py] Loading python module 'k8sevents' Apr 05 20:33:51 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.533+0000 7f5a66492680 1 mgr[py] Loading python module 'localpool' Apr 05 20:33:51 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.634+0000 7f5a66492680 1 mgr[py] Loading python module 'mds_autoscaler' Apr 05 20:33:51 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.894+0000 7f5a66492680 1 mgr[py] Loading python module 'mirroring' Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.019+0000 7f5a66492680 1 mgr[py] Loading python module 'nfs' Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 1 mgr[py] Loading python module 'orchestrator' Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 1 mgr[py] Loading python module 'osd_support' Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 1 mgr[py] Loading python module 'pg_autoscaler' Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 1 mgr[py] Loading python module 'progress' Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member Apr 05 20:33:52 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 1 mgr[py] Loading python module 'prometheus' Apr 05 20:33:53 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member Apr 05 20:33:53 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 1 mgr[py] Loading python module 'rbd_support' Apr 05 20:33:53 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member Apr 05 20:33:53 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 1 mgr[py] Loading python module 'restful' Apr 05 20:33:53 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.799+0000 7f5a66492680 1 mgr[py] Loading python module 'rook' Apr 05 20:33:54 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member Apr 05 20:33:54 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 1 mgr[py] Loading python module 'selftest' Apr 05 20:33:54 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member Apr 05 20:33:54 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 1 mgr[py] Loading python module 'snap_schedule' Apr 05 20:33:54 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member Apr 05 20:33:54 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 1 mgr[py] Loading python module 'stats' Apr 05 20:33:54 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.660+0000 7f5a66492680 1 mgr[py] Loading python module 'status' Apr 05 20:33:54 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 -1 mgr[py] Module status has missing NOTIFY_TYPES member Apr 05 20:33:54 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 1 mgr[py] Loading python module 'telegraf' Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 1 mgr[py] Loading python module 'telemetry' Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 1 mgr[py] Loading python module 'test_orchestrator' Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 1 mgr[py] Loading python module 'volumes' Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 1 mgr[py] Loading python module 'zabbix' Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.792+0000 7f5a66492680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.793+0000 7f5a52600700 0 ms_deliver_dispatch: unhandled message 0x556eb85f26e0 mon_map magic: 0 v1 from mon.2 v2:192.168.47.72:3300/0 Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from module: dashboard Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from module: prometheus Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f58f7800700 0 [dashboard INFO root] server: ssl=yes host=:: port=8443 Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] server_addr: :: server_port: 9283 Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] Starting engine... Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Bus STARTING Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Bus STARTING Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.797+0000 7f5a52600700 0 client.0 ms_handle_reset on v2:192.168.47.73:6842/917991990 Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.825+0000 7f58f7800700 0 [dashboard INFO root] Configured CherryPy, starting engine... Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.826+0000 7f58f7800700 0 [dashboard INFO root] Starting engine... Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy Checker: Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty config. Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Serving on http://:::9283 Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Serving on http://:::9283 Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Bus STARTED Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Bus STARTED Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO root] Engine started. Apr 05 20:33:55 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.939+0000 7f58f7800700 0 [dashboard INFO root] Engine started... Apr 05 20:33:58 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:58 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:58.569+0000 7f59b8000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:00 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:00 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:00.702+0000 7f59af400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:03 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:03 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:03.286+0000 7f59a6800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:08 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:08 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:08.569+0000 7f599dc00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:10 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:10 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:10.702+0000 7f5995000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:13 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:13 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:13.286+0000 7f598c400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:18 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:18 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:18.569+0000 7f5983800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:20 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:20 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:20.702+0000 7f597ac00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4"

cn03 —— Apr 05 20:33:42 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:42 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:42.753+0000 7f2724a00700 0 [prometheus INFO cherrypy.access.139806526902960] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:43 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:43 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:43.772+0000 7f271be00700 0 [prometheus INFO cherrypy.access.139806526902960] ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:47 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:47 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:47.037+0000 7f2713200700 0 [prometheus INFO cherrypy.access.139806526902960] ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.867+0000 7f27d1a00700 1 mgr handle_mgr_map Activating! Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.868+0000 7f27d1a00700 1 mgr handle_mgr_map I am now activating Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO root] Stopping engine... Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO root] Stopped engine... Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO root] Stopping engine... Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO root] Stopped engine Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: [05/Apr/2025:20:33:48] ENGINE Bus STOPPING Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.879+0000 7f267e600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPING Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.890+0000 7f2675a00700 0 [dashboard INFO root] Engine stopped. Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: [05/Apr/2025:20:33:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.891+0000 7f267e600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: [05/Apr/2025:20:33:48] ENGINE Bus STOPPED Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPED Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO root] Engine stopped. Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts INFO root] Init Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.898+0000 7f2663400700 1 mgr load Constructed class from module: alerts Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.899+0000 7f2663400700 0 [balancer DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.899+0000 7f2663400700 1 mgr load Constructed class from module: balancer Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.899+0000 7f26f8e00700 0 [alerts INFO root] Starting Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.901+0000 7f2663400700 0 [cephadm DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.901+0000 7f26f0200700 0 [balancer INFO root] Starting Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO root] Optimize plan auto_2025-04-05_20:33:48 Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO root] Mode upmap, max misplaced 0.050000 Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again later Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed to construct class in 'cephadm' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade = CephadmUpgrade(self) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: return cls(**c) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: TypeError: __init__() got an unexpected keyword argument 'daemon_types' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('cephadm') Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [crash DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load Constructed class from module: crash Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load Constructed class from module: dashboard Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.920+0000 7f2660c00700 0 [dashboard INFO access_control] Loading user roles DB version=2 Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO sso] Loading SSO DB version=1 Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO root] server: ssl=yes host=:: port=8443 Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load Constructed class from module: devicehealth Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [insights DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load Constructed class from module: insights Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.923+0000 7f2660200700 0 [devicehealth INFO root] Starting Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.925+0000 7f2663400700 0 [iostat DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.925+0000 7f2663400700 1 mgr load Constructed class from module: iostat Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.925+0000 7f265f800700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.926+0000 7f2663400700 0 [mirroring DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.931+0000 7f2663400700 1 mgr load Constructed class from module: mirroring Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [nfs DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load Constructed class from module: nfs Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load Constructed class from module: orchestrator Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.933+0000 7f2663400700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.933+0000 7f2663400700 1 mgr load Constructed class from module: pg_autoscaler Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.934+0000 7f2663400700 0 [progress DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.934+0000 7f2663400700 1 mgr load Constructed class from module: progress Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.934+0000 7f2657600700 0 [pg_autoscaler INFO root] _maybe_adjust Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.940+0000 7f2656c00700 0 [progress INFO root] Loading... Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.940+0000 7f2663400700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.944+0000 7f2663400700 1 mgr load Constructed class from module: prometheus Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO root] server_addr: :: server_port: 9283 Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO root] Cache enabled Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7f27363ac400>, <progress.module.GhostEvent object at 0x7f27363ac438>, <progress.module.GhostEvent object at 0x7f27363ac3c8>, <progress.module.GhostEvent object at 0x7f27363ac5c0>, <progress.module.GhostEvent object at 0x7f27363ac4e0>, <progress.module.GhostEvent object at 0x7f27363ac4a8>, <progress.module.GhostEvent object at 0x7f278d317e10>, <progress.module.GhostEvent object at 0x7f278d4397b8>, <progress.module.GhostEvent object at 0x7f278d439908>, <progress.module.GhostEvent object at 0x7f27363ed2e8>, <progress.module.GhostEvent object at 0x7f27363ed4e0>, <progress.module.GhostEvent object at 0x7f27363ed358>, <progress.module.GhostEvent object at 0x7f27363ed278>, <progress.module.GhostEvent object at 0x7f27363edeb8>, <progress.module.GhostEvent object at 0x7f27363ed438>, <progress.module.GhostEvent object at 0x7f27363ed400>, <progress.module.GhostEvent object at 0x7f27363edb00>, <progress.module.GhostEvent object at 0x7f27363ed780>, <progress.module.GhostEvent object at 0x7f27363ed240>, <progress.module.GhostEvent object at 0x7f27363edba8>, <progress.module.GhostEvent object at 0x7f27363ed2b0>, <progress.module.GhostEvent object at 0x7f27363ed0b8>, <progress.module.GhostEvent object at 0x7f27363edcf8>, <progress.module.GhostEvent object at 0x7f27363ed128>, <progress.module.GhostEvent object at 0x7f27363ed080>, <progress.module.GhostEvent object at 0x7f27363edc18>, <progress.module.GhostEvent object at 0x7f27363ed860>, <progress.module.GhostEvent object at 0x7f27363eda58>, <progress.module.GhostEvent object at 0x7f27363eda90>, <progress.module.GhostEvent object at 0x7f27363ed160>, <progress.module.GhostEvent object at 0x7f27363ed908>, <progress.module.GhostEvent object at 0x7f27363ed940>, <progress.module.GhostEvent object at 0x7f27363ed8d0>, <progress.module.GhostEvent object at 0x7f27363ed978>, <progress.module.GhostEvent object at 0x7f27363ed748>, <progress.module.GhostEvent object at 0x7f27363ed898>, <progress.module.GhostEvent object at 0x7f27363eda20>, <progress.module.GhostEvent object at 0x7f27363ed9b0>, <progress.module.GhostEvent object at 0x7f27363ed9e8>, <progress.module.GhostEvent object at 0x7f27363ed828>, <progress.module.GhostEvent object at 0x7f27363ed7f0>, <progress.module.GhostEvent object at 0x7f27363ed7b8>, <progress.module.GhostEvent object at 0x7f27363ed668>, <progress.module.GhostEvent object at 0x7f27363ed320>, <progress.module.GhostEvent object at 0x7f27363ede48>, <progress.module.GhostEvent object at 0x7f27363edc50>, <progress.module.GhostEvent object at 0x7f27363ede80>, <progress.module.GhostEvent object at 0x7f27363edf60>, <progress.module.GhostEvent object at 0x7f27363edd68>, <progress.module.GhostEvent object at 0x7f27363edda0>] historic events Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO root] Loaded OSDMap, ready. Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.946+0000 7f2663400700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.946+0000 7f2655800700 0 [prometheus INFO root] starting metric collection thread Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.947+0000 7f2656200700 0 [prometheus INFO root] Starting engine... Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: [05/Apr/2025:20:33:48] ENGINE Bus STARTING Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.948+0000 7f2656200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STARTING Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.948+0000 7f2660c00700 0 [dashboard INFO root] Configured CherryPy, starting engine... Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.953+0000 7f2663400700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.957+0000 7f2663400700 0 [rbd_support INFO root] load_schedules: csxyz, start_after= Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.989+0000 7f2649a00700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: starting Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.990+0000 7f2649000700 0 [rbd_support INFO root] PerfHandler: starting Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.991+0000 7f2663400700 0 [rbd_support INFO root] load_task_task: csxyz, start_after= Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.019+0000 7f2648600700 0 [rbd_support INFO root] TaskHandler: starting Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.020+0000 7f2663400700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.022+0000 7f2663400700 0 [rbd_support INFO root] load_schedules: csxyz, start_after= Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.024+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS -> /api/cephfs Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.026+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi -> /ui-api/cephfs Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi -> /ui-api/iscsi Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi -> /api/iscsi Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget -> /api/iscsi/target Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGanesha -> /api/nfs-ganesha Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster -> /api/nfs-ganesha/cluster Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports -> /api/nfs-ganesha/export Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi -> /ui-api/nfs-ganesha Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd -> /api/block/image Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.034+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot -> /api/block/image/{image_spec}/snap Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash -> /api/block/image/trash Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace -> /api/block/pool/{pool_name}/namespace Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring -> /api/block/mirroring Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary -> /api/block/mirroring/summary Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode -> /api/block/mirroring/pool Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap -> /api/block/mirroring/pool/{pool_name}/bootstrap Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer -> /api/block/mirroring/pool/{pool_name}/peer Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.038+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw -> /api/rgw Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon -> /api/rgw/daemon Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite -> /api/rgw/site Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.040+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket -> /api/rgw/bucket Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.041+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser -> /api/rgw/user Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth -> /api/auth Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster -> /api/cluster Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.045+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration -> /api/cluster_conf Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.046+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule -> /api/crush_rule Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.047+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi -> /ui-api/crush_rule Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator -> /api/orchestrator Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon -> /api/daemon Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs -> /docs Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile -> /api/erasure_code_profile Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi -> /ui-api/erasure_code_profile Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeedbackController -> /api/feedback Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.051+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeedbackApiController -> /api/feedback/api_key Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeedbackUiController -> /ui-api/feedback/api_key Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging -> /ui-api/logging Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana -> /api/grafana Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host -> /api/host Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi -> /ui-api/host Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health -> /api/health Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController -> / Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: [05/Apr/2025:20:33:49] ENGINE Serving on http://:::9283 Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Serving on http://:::9283 Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: [05/Apr/2025:20:33:49] ENGINE Bus STARTED Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Bus STARTED Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO root] Engine started. Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController -> /ui-api/langs Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs -> /api/logs Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules -> /api/mgr/module Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor -> /api/monitor Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd -> /api/osd Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.060+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController -> /api/osd/flags Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter -> /api/perf_counters/mds Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter -> /api/perf_counters/mon Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter -> /api/perf_counters/osd Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter -> /api/perf_counters/rgw Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter -> /api/perf_counters/rbd-mirror Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter -> /api/perf_counters/mgr Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter -> /api/perf_counters/tcmu-runner Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters -> /api/perf_counters Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.065+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool -> /api/pool Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.066+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi -> /ui-api/pool Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver -> /api/prometheus_receiver Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus -> /api/prometheus Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.068+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications -> /api/prometheus/notifications Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.069+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role -> /api/role Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope -> /ui-api/scope Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2 -> /auth/saml2 Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service -> /api/service Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.071+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings -> /api/settings Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings -> /ui-api/standard_settings Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary -> /api/summary Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task -> /api/task Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry -> /api/telemetry Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User -> /api/user Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy -> /api/user Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword -> /api/user/{username} Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint -> /api/feature_toggles Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay -> /ui-api/motd Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.090+0000 7f2647200700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: starting Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.096+0000 7f2663400700 1 mgr load Constructed class from module: rbd_support Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.102+0000 7f2663400700 0 [restful DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.102+0000 7f2663400700 1 mgr load Constructed class from module: restful Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.108+0000 7f2645e00700 0 [restful INFO root] server_addr: :: server_port: 8003 Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.109+0000 7f2663400700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.116+0000 7f2645e00700 0 [restful WARNING root] server not running: no certificate configured Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.257+0000 7f2660c00700 0 [dashboard INFO dashboard.module] Engine started. Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed to construct class in 'snap_schedule' Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.client = SnapSchedClient(self) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 158, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: with self.get_schedule_db(fs_name) as conn_mgr: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 192, in get_schedule_db Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: db.executescript(dump) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: sqlite3.OperationalError: table schedules already exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('snap_schedule') Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.274+0000 7f2663400700 0 [status DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.274+0000 7f2663400700 1 mgr load Constructed class from module: status Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.275+0000 7f2663400700 0 [telemetry DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.276+0000 7f2663400700 1 mgr load Constructed class from module: telemetry Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.276+0000 7f2663400700 0 [volumes DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.284+0000 7f2660200700 0 [devicehealth INFO root] Check health Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO volumes.fs.async_job] queuing job for volume 'cssnfsxyz' Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO volumes.fs.async_job] queuing job for volume 'cssnfsxyz' Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.290+0000 7f2663400700 1 mgr load Constructed class from module: volumes Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error registering admin socket command: (17) File exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.883+0000 7f2662a00700 0 log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail Apr 05 20:33:50 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:50.875+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail Apr 05 20:33:52 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:52] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:33:52 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:52.753+0000 7f264f400700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:52] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:33:52 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:52.876+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v5: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail Apr 05 20:33:53 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:53] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:33:53 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:53.772+0000 7f264ea00700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:53] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:33:53 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:53.870+0000 7f2663400700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:33:53 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:53.871+0000 7f265f800700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:33:53 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:53.871+0000 7f265f800700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:33:54 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:54.877+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v6: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 0 B/s rd, 12 KiB/s wr, 2 op/s Apr 05 20:33:55 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:55.796+0000 7f2662a00700 2 mgr.server handle_open ignoring open from mgr.cn02.ceph.xyz.corp.ggixgj 192.168.47.72:0/2580067884; not ready for session (expect reconnect) Apr 05 20:33:56 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:56.878+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v7: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 439 B/s rd, 27 KiB/s wr, 6 op/s Apr 05 20:33:57 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:57] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:33:57 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:57.038+0000 7f264e000700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:57] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:33:58 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:58.870+0000 7f2663400700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:33:58 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:58.878+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v8: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 341 B/s rd, 21 KiB/s wr, 4 op/s Apr 05 20:33:58 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:58.897+0000 7f265f800700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:33:58 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:58.897+0000 7f265f800700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:34:00 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:00.880+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v9: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 279 B/s rd, 24 KiB/s wr, 6 op/s Apr 05 20:34:02 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:34:02 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:02.753+0000 7f264d600700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:34:02 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:02.880+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v10: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 22 KiB/s wr, 5 op/s Apr 05 20:34:03 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:34:03 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:03.772+0000 7f264cc00700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" Apr 05 20:34:03 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:03.870+0000 7f2663400700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:34:03 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:03.873+0000 7f265f800700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:34:03 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:03.874+0000 7f265f800700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:34:04 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:04.881+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v11: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 59 KiB/s wr, 11 op/s Apr 05 20:34:06 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:06.882+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v12: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 511 B/s rd, 66 KiB/s wr, 13 op/s Apr 05 20:34:07 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:07] "GET /metrics HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" Apr 05 20:34:07 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:07.037+0000 7f264c200700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:07] "GET /metrics HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" Apr 05 20:34:08 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:08.871+0000 7f2663400700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:34:08 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:08.883+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v13: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 55 KiB/s wr, 11 op/s Apr 05 20:34:08 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:08.907+0000 7f265f800700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:34:08 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:08.907+0000 7f265f800700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:34:10 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:10.884+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v14: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 79 KiB/s wr, 16 op/s Apr 05 20:34:12 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" Apr 05 20:34:12 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:12.753+0000 7f264b800700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" Apr 05 20:34:12 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:12.884+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v15: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 73 KiB/s wr, 14 op/s Apr 05 20:34:13 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" Apr 05 20:34:13 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:13.772+0000 7f264ae00700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" Apr 05 20:34:13 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:13.871+0000 7f2663400700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:34:13 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:13.883+0000 7f265f800700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:34:13 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:13.883+0000 7f265f800700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:34:14 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:14.886+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v16: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 82 KiB/s rd, 103 KiB/s wr, 113 op/s Apr 05 20:34:15 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:15.754+0000 7f2644000700 0 [dashboard INFO request] [::ffff:192.168.10.101:52375] [GET] [200] [0.089s] [1.7K] [6c8ffcbf-e0d6-4f01-a24a-693d11b5fcdb] / Apr 05 20:34:15 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:15.769+0000 7f2643600700 0 [dashboard INFO request] [::ffff:192.168.10.101:52375] [GET] [200] [0.003s] [1.7K] [d20dd549-caaf-4b35-9784-465556871a83] / Apr 05 20:34:15 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:15.787+0000 7f2642c00700 0 [dashboard INFO request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] [2efb9ce6-0ec0-4525-afcf-5d04356c0f18] / Apr 05 20:34:15 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:15.803+0000 7f2642200700 0 [dashboard INFO request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] [2f0b6080-a99d-4fdf-b7f9-c013a4cf2b78] / Apr 05 20:34:15 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:15.825+0000 7f2641800700 0 [dashboard INFO request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] [5c67930f-511b-4e2a-993e-13bc0c73fe2c] / Apr 05 20:34:15 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:15.842+0000 7f2640e00700 0 [dashboard INFO request] [::ffff:192.168.10.101:52375] [GET] [200] [0.002s] [1.7K] [043cc2d4-1863-4072-a15e-91a04422c2d3] / Apr 05 20:34:16 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:16.887+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v17: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 82 KiB/s rd, 66 KiB/s wr, 106 op/s Apr 05 20:34:17 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:17] "GET /metrics HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" Apr 05 20:34:17 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:17.037+0000 7f264a400700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:17] "GET /metrics HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" Apr 05 20:34:18 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:18.871+0000 7f2663400700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:34:18 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:18.887+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v18: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 82 KiB/s rd, 54 KiB/s wr, 103 op/s Apr 05 20:34:18 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:18.916+0000 7f265f800700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:34:18 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:18.916+0000 7f265f800700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:34:18 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:18.931+0000 7f2659e00700 0 [mirroring INFO mgr_util] scanning for idle connections.. Apr 05 20:34:18 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:18.931+0000 7f2659e00700 0 [mirroring INFO mgr_util] cleaning up connections: [] Apr 05 20:34:19 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:19.109+0000 7f2645400700 0 [snap_schedule INFO mgr_util] scanning for idle connections.. Apr 05 20:34:19 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:19.109+0000 7f2645400700 0 [snap_schedule INFO mgr_util] cleaning up connections: [] Apr 05 20:34:19 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:19.277+0000 7f2632800700 0 [volumes INFO mgr_util] scanning for idle connections.. Apr 05 20:34:19 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:19.277+0000 7f2632800700 0 [volumes INFO mgr_util] cleaning up connections: [] Apr 05 20:34:19 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:19.282+0000 7f262d800700 0 [volumes INFO mgr_util] scanning for idle connections.. Apr 05 20:34:19 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:19.282+0000 7f262d800700 0 [volumes INFO mgr_util] cleaning up connections: [] Apr 05 20:34:19 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:19.287+0000 7f2629c00700 0 [volumes INFO mgr_util] scanning for idle connections.. Apr 05 20:34:19 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:19.287+0000 7f2629c00700 0 [volumes INFO mgr_util] cleaning up connections: [] Apr 05 20:34:20 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:20.889+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v19: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 82 KiB/s rd, 70 KiB/s wr, 107 op/s Apr 05 20:34:22 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:22] "GET /metrics HTTP/1.1" 200 267240 "" "Prometheus/2.33.4" Apr 05 20:34:22 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:22.753+0000 7f264fe00700 0 [prometheus INFO cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:22] "GET /metrics HTTP/1.1" 200 267240 "" "Prometheus/2.33.4" Apr 05 20:34:22 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:34:22.889+0000 7f2661600700 0 log_channel(cluster) log [DBG] : pgmap v20: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 82 KiB/s rd, 45 KiB/s wr, 102 op/s

Thank you -jeremy

...

On Saturday, Apr 05, 2025 at 1:32 PM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: The mgr logs should contain a stack trace, can you check again?

Zitat von Jeremy Hansen <jeremy@skidrow.la>:

...

...

"" > "Prometheus/2.33.4" > Apr 05 20:06:10 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:10.702+0000 7efcb4c00700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200

...

...

"" > "Prometheus/2.33.4" > Apr 05 20:06:10 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200

...

...

"" > "Prometheus/2.33.4" > Apr 05 20:06:08 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:08.568+0000 7efcbd800700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200

...

No. Same issue unfortunately.

...

On Saturday, Apr 05, 2025 at 1:27 PM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: I don't see the module error in the logs you provided. Has it cleared? A mgr failover is often helpful in such cases, basically it's the first thing I suggest since two or three years.

Zitat von Jeremy Hansen <jeremy@skidrow.la>:

> Thank you so much for the detailed instructions. Here’s logs from > the failover to a new node. > > Apr 05 20:06:08 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200

...

...

"" > "Prometheus/2.33.4" > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.027+0000 7efd95200700 1 mgr

...

handle_mgr_map > Activating! > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.028+0000 7efd95200700 1 mgr handle_mgr_map I am

...

now > activating > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root]

...

Stopping > engine... > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root]

...

Stopped > engine... > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root]

...

Stopping > engine... > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root]

...

Stopped > engine > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE

...

Bus > STOPPING > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE

...

Bus > STOPPING > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.053+0000 7efc3a400700 0 [dashboard INFO root]

...

Engine > stopped. > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283))

...

shut > down > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283))

...

shut > down > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE

...

Bus > STOPPED > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE

...

Bus > STOPPED > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO root]

...

Engine > stopped. > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.056+0000 7efc27e00700 0 [alerts DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 0 [alerts INFO

...

root] > Init > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > alerts > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 0 [balancer DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > balancer > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efcfac00700 0 [alerts INFO

...

root] > Starting > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efc27e00700 0 [cephadm DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efcb4c00700 0 [balancer INFO

...

root] > Starting > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Optimize

...

plan > auto_2025-04-05_20:06:12 > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Mode upmap, max

...

misplaced > 0.050000 > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try

...

again > later > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.212+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > cephadm > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 0 [crash DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > crash > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 0 [dashboard DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > dashboard > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.216+0000 7efc23800700 0 [dashboard INFO access_control] Loading user roles

...

DB > version=2 > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO sso] Loading SSO

...

DB > version=1 > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO root] server: ssl=yes

...

host=:: > port=8443 > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > devicehealth > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 0 [insights DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > insights > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc20600700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:06:12 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc21a00700 0 [devicehealth INFO

...

root] > Starting > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 0 [iostat DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > iostat > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.224+0000 7efc27e00700 0 [mirroring DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.227+0000 7efc21a00700 0 [devicehealth INFO root]

...

Check > health > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.235+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > mirroring > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 0 [nfs DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > nfs > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > orchestrator > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > pg_autoscaler > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc10200700 0 [pg_autoscaler INFO

...

root] > _maybe_adjust > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 0 [progress DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > progress > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.247+0000 7efc27e00700 0 [prometheus DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.248+0000 7efc27e00700 1 mgr load Constructed class from

...

module: > prometheus > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc0ee00700 0 [progress INFO

...

root] > Loading... > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc27e00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr:

...

INFO > (2/5) > Apr 05 20:06:12 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7efcf2127710>, <progress.module.GhostEvent object at 0x7efcf2127cc0>, <progress.module.GhostEvent object at 0x7efcf2127208>, <progress.module.GhostEvent object at 0x7efcf2127f28>, <progress.module.GhostEvent object at 0x7efcf21270f0>, <progress.module.GhostEvent object at 0x7efcf2127eb8>, <progress.module.GhostEvent object at 0x7efcf2127b38>, <progress.module.GhostEvent object at 0x7efcf2127a20>, <progress.module.GhostEvent object at 0x7efcf2127da0>, <progress.module.GhostEvent object at 0x7efcf2127f98>, <progress.module.GhostEvent object at 0x7efcf2127780>, <progress.module.GhostEvent object at 0x7efcf2127320>, <progress.module.GhostEvent object at 0x7efcf2127588>, <progress.module.GhostEvent object at 0x7efcf2127518>, <progress.module.GhostEvent object at 0x7efcf2127c50>, <progress.module.GhostEvent object at 0x7efcf21279e8>, <progress.module.GhostEvent object at 0x7efcf2127d68>, <progress.module.GhostEvent object at 0x7efcf2127470>, <progress.module.GhostEvent object at 0x7efcf [Text gekürzt]

Thanks. I’m trying to be extra careful since this cluster is actually in use. I’ll wait for your feedback. -jeremy

On Saturday, Apr 05, 2025 at 3:39 PM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: No, that's not necessary, just edit the unit.run file for the MGRs to use a different image. See Frédéric's instructions:

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/32APKOXKRAI...

But I'm not entirely sure if you need to clear some config-keys first in order to reset the upgrade state. If I have time, I'll try to check tomorrow, or on Monday.

Zitat von Jeremy Hansen <jeremy@skidrow.la>:

...

Would I follow this process to downgrade?

https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-...

Thank you

...

On Saturday, Apr 05, 2025 at 2:04 PM, Jeremy Hansen <jeremy@skidrow.la (mailto:jeremy@skidrow.la)> wrote: ceph -s claims things are healthy:

ceph -s cluster: id: 95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1 health: HEALTH_OK

services: mon: 3 daemons, quorum cn01,cn03,cn02 (age 20h) mgr: cn03.negzvb(active, since 26m), standbys: cn01.tjmtph, cn02.ceph.xyz.corp.ggixgj mds: 1/1 daemons up, 2 standby osd: 15 osds: 15 up (since 19h), 15 in (since 14M)

data: volumes: 1/1 healthy pools: 6 pools, 610 pgs objects: 284.59k objects, 1.1 TiB usage: 3.3 TiB used, 106 TiB / 109 TiB avail pgs: 610 active+clean

io: client: 255 B/s rd, 1.2 MiB/s wr, 10 op/s rd, 16 op/s wr

— How do I downgrade if the orch is down?

Thank you -jeremy

...

On Saturday, Apr 05, 2025 at 1:56 PM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: It would help if you only pasted the relevant parts. Anyway, these two sections stand out:

---snip--- Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again later Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed to construct class in 'cephadm' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade = CephadmUpgrade(self) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: return cls(**c) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: TypeError: __init__() got an unexpected keyword argument 'daemon_types' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('cephadm')

Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed to construct class in 'snap_schedule' Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.client = SnapSchedClient(self) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 158, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: with self.get_schedule_db(fs_name) as conn_mgr: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 192, in get_schedule_db Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: db.executescript(dump) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: sqlite3.OperationalError: table schedules already exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('snap_schedule') ---snip---

Your cluster seems to be in an error state (ceph -s) because of an unknown PG. It's recommended to have a healthy cluster before attemping an upgrade. It's possible that these errors come from the not upgraded MGR, I'm not sure.

Since the upgrade was only successful for two MGRs, I am thinking about downgrading both MGRs back to 16.2.15, then retry an upgrade to a newer version, either 17.2.8 or 18.2.4. I haven't checked the snap_schedule error yet, though. Maybe someone else knows that already.

Zitat von Jeremy Hansen <jeremy@skidrow.la>:

...

This is a small three node cluster. Here’s logs from all three nodes during the failover. The current manager was cn02 and it failed over to cn03.

cn01 —— Apr 05 20:33:35 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:35.823+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v821: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 22 KiB/s wr, 5 op/s Apr 05 20:33:37 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:37.170+0000 7f2aaec00700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:33:37 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:33:37 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:33:37 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:37.823+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v822: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 22 KiB/s wr, 5 op/s Apr 05 20:33:39 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:39] "GET /metrics HTTP/1.1" 200 267241 "" "Prometheus/2.33.4" Apr 05 20:33:39 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:39.825+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v823: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 97 KiB/s wr, 12 op/s Apr 05 20:33:41 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:41.826+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v824: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 255 B/s rd, 85 KiB/s wr, 10 op/s Apr 05 20:33:42 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:42.170+0000 7f2aaec00700 0 [insights INFO root] Received health check update 0 pending Apr 05 20:33:42 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count 0 Apr 05 20:33:42 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed False Apr 05 20:33:42 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:42] "GET /metrics HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" Apr 05 20:33:43 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" Apr 05 20:33:43 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:43.827+0000 7f2aace00700 0 log_channel(cluster) log [DBG] : pgmap v825: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 511 B/s rd, 133 KiB/s wr, 18 op/s Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 -1 mgr handle_mgr_map I was active but no longer am Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn e: '/usr/bin/ceph-mgr' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 0: '/usr/bin/ceph-mgr' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 1: '-n' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 2: 'mgr.cn01.tjmtph' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 3: '-f' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 4: '--setuser' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 5: 'ceph' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 6: '--setgroup' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 7: 'ceph' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 8: '--default-log-to-file=false' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 9: '--default-log-to-stderr=true' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 10: '--default-log-stderr-prefix=debug ' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn respawning with exe /usr/bin/ceph-mgr Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn exe_path /proc/self/exe Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ignoring --setuser ceph since I am not root Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ignoring --setgroup ceph since I am not root Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mgr, pid 2 Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 pidfile_write: ignore empty --pid-file Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.477+0000 7fb814447000 1 mgr[py] Loading python module 'alerts' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.612+0000 7fb814447000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.612+0000 7fb814447000 1 mgr[py] Loading python module 'balancer' Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.726+0000 7fb814447000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member Apr 05 20:33:45 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:45.726+0000 7fb814447000 1 mgr[py] Loading python module 'cephadm' Apr 05 20:33:46 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:46.417+0000 7fb814447000 1 mgr[py] Loading python module 'crash' Apr 05 20:33:46 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:46.540+0000 7fb814447000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member Apr 05 20:33:46 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:46.540+0000 7fb814447000 1 mgr[py] Loading python module 'dashboard' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.210+0000 7fb814447000 1 mgr[py] Loading python module 'devicehealth' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.331+0000 7fb814447000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.331+0000 7fb814447000 1 mgr[py] Loading python module 'diskprediction_local' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.653+0000 7fb814447000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.654+0000 7fb814447000 1 mgr[py] Loading python module 'influx' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.769+0000 7fb814447000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.769+0000 7fb814447000 1 mgr[py] Loading python module 'insights' Apr 05 20:33:47 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:47.884+0000 7fb814447000 1 mgr[py] Loading python module 'iostat' Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.082+0000 7fb814447000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.082+0000 7fb814447000 1 mgr[py] Loading python module 'k8sevents' Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.679+0000 7fb814447000 1 mgr[py] Loading python module 'localpool' Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.796+0000 7fb814447000 1 mgr[py] Loading python module 'mds_autoscaler' Apr 05 20:33:48 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:48.983+0000 7fb814447000 1 mgr[py] Loading python module 'mirroring' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.118+0000 7fb814447000 1 mgr[py] Loading python module 'nfs' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.315+0000 7fb814447000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.315+0000 7fb814447000 1 mgr[py] Loading python module 'orchestrator' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.607+0000 7fb814447000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.607+0000 7fb814447000 1 mgr[py] Loading python module 'osd_perf_query' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.723+0000 7fb814447000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.723+0000 7fb814447000 1 mgr[py] Loading python module 'osd_support' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.837+0000 7fb814447000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.837+0000 7fb814447000 1 mgr[py] Loading python module 'pg_autoscaler' Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.970+0000 7fb814447000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member Apr 05 20:33:49 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:49.970+0000 7fb814447000 1 mgr[py] Loading python module 'progress' Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.082+0000 7fb814447000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.082+0000 7fb814447000 1 mgr[py] Loading python module 'prometheus' Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.617+0000 7fb814447000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.617+0000 7fb814447000 1 mgr[py] Loading python module 'rbd_support' Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.761+0000 7fb814447000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member Apr 05 20:33:50 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:50.761+0000 7fb814447000 1 mgr[py] Loading python module 'restful' Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.023+0000 7fb814447000 1 mgr[py] Loading python module 'rook' Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.788+0000 7fb814447000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.788+0000 7fb814447000 1 mgr[py] Loading python module 'selftest' Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.912+0000 7fb814447000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member Apr 05 20:33:51 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:51.912+0000 7fb814447000 1 mgr[py] Loading python module 'snap_schedule' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.038+0000 7fb814447000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.038+0000 7fb814447000 1 mgr[py] Loading python module 'stats' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.155+0000 7fb814447000 1 mgr[py] Loading python module 'status' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.280+0000 7fb814447000 -1 mgr[py] Module status has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.280+0000 7fb814447000 1 mgr[py] Loading python module 'telegraf' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.398+0000 7fb814447000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.398+0000 7fb814447000 1 mgr[py] Loading python module 'telemetry' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.594+0000 7fb814447000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.594+0000 7fb814447000 1 mgr[py] Loading python module 'test_orchestrator' Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.771+0000 7fb814447000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member Apr 05 20:33:52 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:52.771+0000 7fb814447000 1 mgr[py] Loading python module 'volumes' Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.225+0000 7fb814447000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.225+0000 7fb814447000 1 mgr[py] Loading python module 'zabbix' Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.342+0000 7fb814447000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.344+0000 7fb807c00700 0 ms_deliver_dispatch: unhandled message 0x555c78fe1080 mon_map magic: 0 v1 from mon.0 v2:192.168.47.71:3300/0 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.344+0000 7fb808600700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load Constructed class from module: dashboard Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.345+0000 7fb808600700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5) Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load Constructed class from module: prometheus Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.345+0000 7fb6abe00700 0 [dashboard INFO root] server: ssl=yes host=:: port=8443 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO root] server_addr: :: server_port: 9283 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO root] Starting engine... Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: [05/Apr/2025:20:33:53] ENGINE Bus STARTING Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTING Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO root] Configured CherryPy, starting engine... Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO root] Starting engine... Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: CherryPy Checker: Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: The Application mounted at '' has an empty config. Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: [05/Apr/2025:20:33:53] ENGINE Serving on http://:::9283 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Serving on http://:::9283 Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: [05/Apr/2025:20:33:53] ENGINE Bus STARTED Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTED Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.451+0000 7fb6b4a00700 0 [prometheus INFO root] Engine started. Apr 05 20:33:53 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:53.489+0000 7fb6abe00700 0 [dashboard INFO root] Engine started... Apr 05 20:33:59 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:33:59 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:33:59.603+0000 7fb76c600700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:02 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:02 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:02.476+0000 7fb763a00700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:03 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:03 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:03.061+0000 7fb75ae00700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:09 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:09 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:09.602+0000 7fb752200700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:12 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:12 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:12.476+0000 7fb749600700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:13 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:13 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:13.060+0000 7fb740a00700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.493+0000 7fb714e00700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.494+0000 7fb714e00700 0 [dashboard INFO request] [::ffff:192.168.10.101:52368] [GET] [303] [0.003s] [96.0B] [0031fbdd-947f-4c6c-943d-55cf256edfba] /api/summary Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.512+0000 7fb70c200700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.513+0000 7fb70c200700 0 [dashboard INFO request] [::ffff:192.168.10.101:52369] [GET] [303] [0.001s] [96.0B] [76c9a6ab-aae0-4ef9-815e-dbd0ac8f8b49] /api/feature_toggles Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.527+0000 7fb703600700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.528+0000 7fb703600700 0 [dashboard INFO request] [::ffff:192.168.10.101:52370] [GET] [303] [0.001s] [96.0B] [db017da3-e6f6-45fc-9d99-7b9f3b3859f2] /api/health/minimal Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.541+0000 7fb6faa00700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.542+0000 7fb6faa00700 0 [dashboard INFO request] [::ffff:192.168.10.101:52371] [GET] [303] [0.001s] [96.0B] [564ba322-b99c-49de-a3a3-e0ed422bfae7] /api/prometheus Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.616+0000 7fb6f1e00700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.617+0000 7fb6f1e00700 0 [dashboard INFO request] [::ffff:192.168.10.101:52372] [GET] [303] [0.001s] [96.0B] [1f349e52-9a10-4fda-b13f-621785208ea9] /api/prometheus/rules Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO root] Redirecting to active 'https://192.168.47.73:8443/' Apr 05 20:34:15 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO request] [::ffff:192.168.10.101:52368] [GET] [303] [0.001s] [96.0B] [30d749d7-c972-44f0-88c7-b70078e2385c] /api/prometheus/notifications Apr 05 20:34:19 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" Apr 05 20:34:19 cn01.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: debug 2025-04-05T20:34:19.601+0000 7fb737e00700 0 [prometheus INFO cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4"

cn02 —— Apr 05 20:33:40 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200 - ""

...

"Prometheus/2.33.4" Apr 05 20:33:40 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:40.701+0000 7ff710400700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200 - ""

...

"Prometheus/2.33.4" Apr 05 20:33:43 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - ""

...

"Prometheus/2.33.4" Apr 05 20:33:43 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:43.287+0000 7ff707800700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - ""

...

"Prometheus/2.33.4" Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.370+0000 7ff7e7c00700 1 mgr handle_mgr_map

...

Activating! Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.371+0000 7ff7e7c00700 1 mgr handle_mgr_map I am now

...

activating Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root] Stopping

...

engine... Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root] Stopped

...

engine... Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root] Stopping

...

engine... Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root] Stopped

...

engine Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus

...

STOPPING Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.383+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus

...

STOPPING Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.394+0000 7ff68d000700 0 [dashboard INFO root] Engine

...

stopped. Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut

...

down Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.395+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut

...

down Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus

...

STOPPED Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus

...

STOPPED Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO root] Engine

...

stopped. Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts INFO root]

...

Init Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

alerts Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 0 [balancer DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

balancer Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff74d800700 0 [alerts INFO root]

...

Starting Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.401+0000 7ff67aa00700 0 [cephadm DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.402+0000 7ff710400700 0 [balancer INFO root]

...

Starting Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Optimize plan

...

auto_2025-04-05_20:33:45 Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Mode upmap, max misplaced

...

0.050000 Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.409+0000 7ff710400700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again

...

later Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.551+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

cephadm Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 0 [crash DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

crash Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

dashboard Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

devicehealth Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO access_control] Loading user roles DB

...

version=2 Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO sso] Loading SSO DB

...

version=1 Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO root] server: ssl=yes host=::

...

port=8443 Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.557+0000 7ff674600700 0 [devicehealth INFO root]

...

Starting Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.558+0000 7ff67aa00700 0 [insights DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.559+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

insights Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.561+0000 7ff674600700 0 [devicehealth INFO root] Check

...

health Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.566+0000 7ff671e00700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 0 [iostat DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

iostat Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.572+0000 7ff67aa00700 0 [mirroring DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.579+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

mirroring Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 0 [nfs DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

nfs Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

orchestrator Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

pg_autoscaler Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.625+0000 7ff662a00700 0 [pg_autoscaler INFO root]

...

_maybe_adjust Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 0 [progress DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

progress Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.72' (ECDSA) to the list of known

...

hosts. Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.73' (ECDSA) to the list of known

...

hosts. Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO

...

(2/5) Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 1 mgr load Constructed class from module:

...

prometheus Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff661600700 0 [progress INFO root]

...

Loading... Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] server_addr: :: server_port:

...

9283 Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] Cache

...

enabled Apr 05 20:33:45 cn02.ceph.xyz.corp

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff65ee00700 0 [prometheus INFO root] starting metric

collection > > thread > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.640+0000 7ff67aa00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded [&lt;progress.module.GhostEvent object at 0x7ff7443993c8>, &lt;progress.module.GhostEvent object at 0x7ff744399400>, &lt;progress.module.GhostEvent object at 0x7ff744399438>, &lt;progress.module.GhostEvent object at 0x7ff744399470>, &lt;progress.module.GhostEvent object at 0x7ff7443994a8>, &lt;progress.module.GhostEvent object at 0x7ff7443994e0>, &lt;progress.module.GhostEvent object at 0x7ff744399518>, &lt;progress.module.GhostEvent object at 0x7ff744399550>, &lt;progress.module.GhostEvent object at 0x7ff744399588>, &lt;progress.module.GhostEvent object at 0x7ff7443995c0>, &lt;progress.module.GhostEvent object at 0x7ff7443995f8>, &lt;progress.module.GhostEvent object at 0x7ff744399630>, &lt;progress.module.GhostEvent object at 0x7ff744399668>, &lt;progress.module.GhostEvent object at 0x7ff7443996a0>, &lt;progress.module.GhostEvent object at 0x7ff7443996d8>, &lt;progress.module.GhostEvent object at 0x7ff744399710>, &lt;progress.module.GhostEvent object at 0x7ff744399748>, &lt;progress.module.GhostEvent object at 0x7ff744399780>, &lt;progress.module.GhostEvent object at 0x7ff7443997b8>, &lt;progress.module.GhostEvent object at 0x7ff7443997f0>, &lt;progress.module.GhostEvent object at 0x7ff744399828>, &lt;progress.module.GhostEvent object at 0x7ff744399860>, &lt;progress.module.GhostEvent object at 0x7ff744399898>, &lt;progress.module.GhostEvent object at 0x7ff7443998d0>, &lt;progress.module.GhostEvent object at 0x7ff744399908>, &lt;progress.module.GhostEvent object at 0x7ff744399940>, &lt;progress.module.GhostEvent object at 0x7ff744399978>, &lt;progress.module.GhostEvent object at 0x7ff7443999b0>, &lt;progress.module.GhostEvent object at 0x7ff7443999e8>, &lt;progress.module.GhostEvent object at 0x7ff744399a20>, &lt;progress.module.GhostEvent object at 0x7ff744399a58>, &lt;progress.module.GhostEvent object at 0x7ff744399a90>, &lt;progress.module.GhostEvent object at 0x7ff744399ac8>, &lt;progress.module.GhostEvent object at 0x7ff744399b00>, &lt;progress.module.GhostEvent object at 0x7ff744399b38>, &lt;progress.module.GhostEvent object at 0x7ff744399b70>, &lt;progress.module.GhostEvent object at 0x7ff744399ba8>, &lt;progress.module.GhostEvent object at 0x7ff744399be0>, &lt;progress.module.GhostEvent object at 0x7ff744399c18>, &lt;progress.module.GhostEvent object at 0x7ff744399c50>, &lt;progress.module.GhostEvent object at 0x7ff744399c88>, &lt;progress.module.GhostEvent object at 0x7ff744399cc0>, &lt;progress.module.GhostEvent object at 0x7ff744399cf8>, &lt;progress.module.GhostEvent object at 0x7ff744399d30>, &lt;progress.module.GhostEvent object at 0x7ff744399d68>, &lt;progress.module.GhostEvent object at 0x7ff744399da0>, &lt;progress.module.GhostEvent object at 0x7ff744399dd8>, &lt;progress.module.GhostEvent object at 0x7ff744399e10>, &lt;progress.module.GhostEvent object at 0x7ff744399e48>, &lt;progress.module.GhostEvent object at 0x7ff744399e80>] historic > > events > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded OSDMap, > > ready. > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Processing OSDMap change > > 27185..27185 > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff676400700 0 [dashboard INFO root] Configured CherryPy, starting > > engine... > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff660200700 0 [prometheus INFO root] Starting > > engine... > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > > STARTING > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.647+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > > STARTING > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] recovery thread > > starting > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] starting > > setup > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff67aa00700 1 mgr load Constructed class from module: > > rbd_support > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.71' (ECDSA) to the list of known > > hosts. > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 0 [restful DEBUG root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 1 mgr load Constructed class from module: > > restful > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.660+0000 7ff651c00700 0 [restful INFO root] server_addr: :: server_port: > > 8003 > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.661+0000 7ff654e00700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: > > load_schedules > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.697+0000 7ff651c00700 0 [restful WARNING root] server not running: no certificate > > configured > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.705+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules: csxyz, > > start_after= > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.706+0000 7ff67aa00700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.707+0000 7ff64f400700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: > > starting > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.708+0000 7ff64ea00700 0 [rbd_support INFO root] PerfHandler: > > starting > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.709+0000 7ff654e00700 0 [rbd_support INFO root] load_task_task: csxyz, > > start_after= > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.718+0000 7ff649a00700 0 [rbd_support INFO root] TaskHandler: > > starting > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.719+0000 7ff654e00700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: > > load_schedules > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.721+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules: csxyz, > > start_after= > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff649000700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: > > starting > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff654e00700 0 [rbd_support INFO root] setup > > complete > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.726+0000 7ff67aa00700 1 mgr load Constructed class from module: > > snap_schedule > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 0 [status DEBUG root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 1 mgr load Constructed class from module: > > status > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.728+0000 7ff67aa00700 0 [telemetry DEBUG root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 1 mgr load Constructed class from module: > > telemetry > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 0 [volumes DEBUG root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume > > 'cssnfsxyz' > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume > > 'cssnfsxyz' > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 1 mgr load Constructed class from module: > > volumes > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17) File > > exists > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy > > Checker: > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty > > config. > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Serving on > > http://:::9283 > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Serving on > > http://:::9283 > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Bus > > STARTED > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Bus > > STARTED > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO root] Engine > > started. > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.785+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS -> > > /api/cephfs > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.787+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi -> > > /ui-api/cephfs > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi -> > > /ui-api/iscsi > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi -> > > /api/iscsi > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget -> > > /api/iscsi/target > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster -> > > /api/nfs-ganesha/cluster > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports -> > > /api/nfs-ganesha/export > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi -> > > /ui-api/nfs-ganesha > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd -> > > /api/block/image > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdStatus -> > > /ui-api/block/rbd > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot -> > > /api/block/image/{image_spec}/snap > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.796+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash -> > > /api/block/image/trash > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace -> > > /api/block/pool/{pool_name}/namespace > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool -> > > /api/pool > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.799+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi -> > > /ui-api/pool > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.800+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RBDPool -> > > /api/pool > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator -> > > /ui-api/orchestrator > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service -> > > /api/service > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring -> > > /api/block/mirroring > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary -> > > /api/block/mirroring/summary > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode -> > > /api/block/mirroring/pool > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap -> > > /api/block/mirroring/pool/{pool_name}/bootstrap > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer -> > > /api/block/mirroring/pool/{pool_name}/peer > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.804+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringStatus -> > > /ui-api/block/mirroring > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw -> > > /ui-api/rgw > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon -> > > /api/rgw/daemon > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite -> > > /api/rgw/site > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket -> > > /api/rgw/bucket > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.807+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser -> > > /api/rgw/user > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.810+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth -> > > /api/auth > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster -> > > /api/cluster > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration -> > > /api/cluster_conf > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.813+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule -> > > /api/crush_rule > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi -> > > /ui-api/crush_rule > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon -> > > /api/daemon > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs -> > > /docs > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile -> > > /api/erasure_code_profile > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.816+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi -> > > /ui-api/erasure_code_profile > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging -> > > /ui-api/logging > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana -> > > /api/grafana > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host -> > > /api/host > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.819+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi -> > > /ui-api/host > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health -> > > /api/health > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController -> > > / > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.822+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController -> > > /ui-api/langs > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LoginController -> > > /ui-api/login > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs -> > > /api/logs > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules -> > > /api/mgr/module > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor -> > > /api/monitor > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd -> > > /api/osd > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.828+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdUi -> > > /ui-api/osd > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.830+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController -> > > /api/osd/flags > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter -> > > /api/perf_counters/mds > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter -> > > /api/perf_counters/mon > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter -> > > /api/perf_counters/osd > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter -> > > /api/perf_counters/rgw > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter -> > > /api/perf_counters/rbd-mirror > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter -> > > /api/perf_counters/mgr > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter -> > > /api/perf_counters/tcmu-runner > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters -> > > /api/perf_counters > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver -> > > /api/prometheus_receiver > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus -> > > /api/prometheus > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications -> > > /api/prometheus/notifications > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role -> > > /api/role > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope -> > > /ui-api/scope > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2 -> > > /auth/saml2 > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings -> > > /api/settings > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings -> > > /ui-api/standard_settings > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary -> > > /api/summary > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task -> > > /api/task > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry -> > > /api/telemetry > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.842+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User -> > > /api/user > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy -> > > /api/user > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword -> > > /api/user/{username} > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint -> > > /api/feature_toggles > > Apr 05 20:33:45 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.844+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay -> > > /ui-api/motd > > Apr 05 20:33:46 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.050+0000 7ff676400700 0 [dashboard INFO dashboard.module] Engine > > started. > > Apr 05 20:33:46 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.378+0000 7ff67a000700 0 log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB > > avail > > Apr 05 20:33:47 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:47.379+0000 7ff678c00700 0 log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB > > avail > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104 "" > > "Prometheus/2.33.4" > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.568+0000 7ff65b200700 0 [prometheus INFO cherrypy.access.140699978276536] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104 "" > > "Prometheus/2.33.4" > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 [cephadm INFO cephadm.serve] Updating > > cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 log_channel(cephadm) log [INF] : Updating > > cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 [cephadm INFO cephadm.serve] Updating > > cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve] Updating > > cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 log_channel(cephadm) log [INF] : Updating > > cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] : Updating > > cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 [cephadm INFO cephadm.serve] Updating > > cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 log_channel(cephadm) log [INF] : Updating > > cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 [cephadm INFO cephadm.serve] Updating > > cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 log_channel(cephadm) log [INF] : Updating > > cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve] Updating > > cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] : Updating > > cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.812+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 824b8fde-5a37-4436-af45-a7d8255e2c66 does not > > exist > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 621cb660-d584-45e4-8033-4a476e804bba does not > > exist > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 007a4dd2-46d3-4655-9c93-c02f97e555cb does not > > exist > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 6b0fbff2-881e-4203-9149-f0977bc13eab does not > > exist > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 8a1a90b8-1f2a-48c8-924f-6e74032b6319 does not > > exist > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.815+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3c322d26-9edb-4d26-b4bc-97c330f5a1d8 does not > > exist > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev f3e5a517-d643-4fa5-b127-57716264e4e3 does not > > exist > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3fb020c7-92e8-4eb1-a884-2d04e5bca889 does not > > exist > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.840+0000 7ff6a7400700 0 [progress WARNING root] complete: ev bf9e0705-075e-4134-b77f-aa64fe92112a does not > > exist > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade myself > > (mgr.cn02.ceph.xyz.corp.ggixgj) > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade myself > > (mgr.cn02.ceph.xyz.corp.ggixgj) > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [progress INFO root] update: starting ev e7e1a809-558d-43a7-842a-c6229fdc57af (Upgrade to > > 17.2.0) > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 [cephadm INFO cephadm.services.cephadmservice] Failing over to other > > MGR > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Failing over to other > > MGR > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 -1 mgr handle_mgr_map I was active but no longer > > am > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn e: > > '/usr/bin/ceph-mgr' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 0: > > '/usr/bin/ceph-mgr' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 1: > > '-n' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 2: > > 'mgr.cn02.ceph.xyz.corp.ggixgj' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 3: > > '-f' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 4: > > '--setuser' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 5: > > 'ceph' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 6: > > '--setgroup' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 7: > > 'ceph' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 8: > > '--default-log-to-file=false' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 9: > > '--default-log-to-stderr=true' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 10: '--default-log-stderr-prefix=debug > > ' > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn respawning with exe > > /usr/bin/ceph-mgr > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn exe_path > > /proc/self/exe > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setuser ceph since I am not > > root > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setgroup ceph since I am not > > root > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable), process ceph-mgr, pid > > 2 > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 pidfile_write: ignore empty > > --pid-file > > Apr 05 20:33:48 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.976+0000 7f5a66492680 1 mgr[py] Loading python module > > 'alerts' > > Apr 05 20:33:49 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES > > member > > Apr 05 20:33:49 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 1 mgr[py] Loading python module > > 'balancer' > > Apr 05 20:33:49 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES > > member > > Apr 05 20:33:49 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 1 mgr[py] Loading python module > > 'cephadm' > > Apr 05 20:33:49 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.568+0000 7f5a66492680 1 mgr[py] Loading python module > > 'crash' > > Apr 05 20:33:49 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 -1 mgr[py] Module crash has missing NOTIFY_TYPES > > member > > Apr 05 20:33:49 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 1 mgr[py] Loading python module > > 'dashboard' > > Apr 05 20:33:50 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.276+0000 7f5a66492680 1 mgr[py] Loading python module > > 'devicehealth' > > Apr 05 20:33:50 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.415+0000 7f5a66492680 1 mgr[py] Loading python module > > 'diskprediction_local' > > Apr 05 20:33:50 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES > > member > > Apr 05 20:33:50 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 1 mgr[py] Loading python module > > 'influx' > > Apr 05 20:33:50 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 -1 mgr[py] Module influx has missing NOTIFY_TYPES > > member > > Apr 05 20:33:50 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 1 mgr[py] Loading python module > > 'insights' > > Apr 05 20:33:50 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.919+0000 7f5a66492680 1 mgr[py] Loading python module > > 'iostat' > > Apr 05 20:33:51 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES > > member > > Apr 05 20:33:51 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 1 mgr[py] Loading python module > > 'k8sevents' > > Apr 05 20:33:51 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.533+0000 7f5a66492680 1 mgr[py] Loading python module > > 'localpool' > > Apr 05 20:33:51 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.634+0000 7f5a66492680 1 mgr[py] Loading python module > > 'mds_autoscaler' > > Apr 05 20:33:51 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.894+0000 7f5a66492680 1 mgr[py] Loading python module > > 'mirroring' > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.019+0000 7f5a66492680 1 mgr[py] Loading python module > > 'nfs' > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES > > member > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 1 mgr[py] Loading python module > > 'orchestrator' > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES > > member > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 1 mgr[py] Loading python module > > 'osd_support' > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES > > member > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 1 mgr[py] Loading python module > > 'pg_autoscaler' > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES > > member > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 1 mgr[py] Loading python module > > 'progress' > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 -1 mgr[py] Module progress has missing NOTIFY_TYPES > > member > > Apr 05 20:33:52 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 1 mgr[py] Loading python module > > 'prometheus' > > Apr 05 20:33:53 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES > > member > > Apr 05 20:33:53 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 1 mgr[py] Loading python module > > 'rbd_support' > > Apr 05 20:33:53 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES > > member > > Apr 05 20:33:53 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 1 mgr[py] Loading python module > > 'restful' > > Apr 05 20:33:53 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.799+0000 7f5a66492680 1 mgr[py] Loading python module > > 'rook' > > Apr 05 20:33:54 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 -1 mgr[py] Module rook has missing NOTIFY_TYPES > > member > > Apr 05 20:33:54 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 1 mgr[py] Loading python module > > 'selftest' > > Apr 05 20:33:54 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES > > member > > Apr 05 20:33:54 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 1 mgr[py] Loading python module > > 'snap_schedule' > > Apr 05 20:33:54 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES > > member > > Apr 05 20:33:54 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 1 mgr[py] Loading python module > > 'stats' > > Apr 05 20:33:54 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.660+0000 7f5a66492680 1 mgr[py] Loading python module > > 'status' > > Apr 05 20:33:54 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 -1 mgr[py] Module status has missing NOTIFY_TYPES > > member > > Apr 05 20:33:54 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 1 mgr[py] Loading python module > > 'telegraf' > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES > > member > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 1 mgr[py] Loading python module > > 'telemetry' > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES > > member > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 1 mgr[py] Loading python module > > 'test_orchestrator' > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES > > member > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 1 mgr[py] Loading python module > > 'volumes' > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES > > member > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 1 mgr[py] Loading python module > > 'zabbix' > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.792+0000 7f5a66492680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES > > member > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.793+0000 7f5a52600700 0 ms_deliver_dispatch: unhandled message 0x556eb85f26e0 mon_map magic: 0 v1 from mon.2 > > v2:192.168.47.72:3300/0 > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from module: > > dashboard > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO > > (2/5) > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from module: > > prometheus > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f58f7800700 0 [dashboard INFO root] server: ssl=yes host=:: > > port=8443 > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] server_addr: :: server_port: > > 9283 > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] Starting > > engine... > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Bus > > STARTING > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Bus > > STARTING > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.797+0000 7f5a52600700 0 client.0 ms_handle_reset on > > v2:192.168.47.73:6842/917991990 > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.825+0000 7f58f7800700 0 [dashboard INFO root] Configured CherryPy, starting > > engine... > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.826+0000 7f58f7800700 0 [dashboard INFO root] Starting > > engine... > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy > > Checker: > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty > > config. > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Serving on > > http://:::9283 > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Serving on > > http://:::9283 > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Bus > > STARTED > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Bus > > STARTED > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO root] Engine > > started. > > Apr 05 20:33:55 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.939+0000 7f58f7800700 0 [dashboard INFO root] Engine > > started... > > Apr 05 20:33:58 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:33:58 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:58.569+0000 7f59b8000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:00 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:00 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:00.702+0000 7f59af400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:03 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:03 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:03.286+0000 7f59a6800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:08 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:08 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:08.569+0000 7f599dc00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:10 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:10 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:10.702+0000 7f5995000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:13 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:13 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:13.286+0000 7f598c400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:18 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:18 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:18.569+0000 7f5983800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:20 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:34:20 cn02.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:20.702+0000 7f597ac00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > > > cn03 > > —— > > Apr 05 20:33:42 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:42] "GET /metrics > > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > > Apr 05 20:33:42 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:42.753+0000 7f2724a00700 0 [prometheus INFO > > cherrypy.access.139806526902960] ::ffff:192.168.47.72 - - > > [05/Apr/2025:20:33:42] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:33:43 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:43] "GET /metrics > > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > > Apr 05 20:33:43 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:43.772+0000 7f271be00700 0 [prometheus INFO > > cherrypy.access.139806526902960] ::ffff:192.168.47.71 - - > > [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:33:47 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:47] "GET /metrics > > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > > Apr 05 20:33:47 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:47.037+0000 7f2713200700 0 [prometheus INFO > > cherrypy.access.139806526902960] ::ffff:192.168.47.73 - - > > [05/Apr/2025:20:33:47] "GET /metrics HTTP/1.1" 200 - "" > > "Prometheus/2.33.4" > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.867+0000 7f27d1a00700 1 mgr handle_mgr_map > > Activating! > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.868+0000 7f27d1a00700 1 mgr handle_mgr_map > > I am now activating > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > > root] Stopping engine... > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > > root] Stopped engine... > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > > root] Stopping engine... > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > > root] Stopped engine > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.879+0000 7f267e600700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.890+0000 7f2675a00700 0 [dashboard INFO > > root] Engine stopped. > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > [05/Apr/2025:20:33:48] ENGINE HTTP Server > > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.891+0000 7f267e600700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE HTTP Server > > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > > root] Engine stopped. > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts INFO root] > > Init > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.898+0000 7f2663400700 1 mgr load > > Constructed class from module: alerts > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.899+0000 7f2663400700 0 [balancer DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.899+0000 7f2663400700 1 mgr load > > Constructed class from module: balancer > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.899+0000 7f26f8e00700 0 [alerts INFO root] > > Starting > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.901+0000 7f2663400700 0 [cephadm DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.901+0000 7f26f0200700 0 [balancer INFO > > root] Starting > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > > root] Optimize plan auto_2025-04-05_20:33:48 > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > > root] Mode upmap, max misplaced 0.050000 > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO > > root] Some PGs (1.000000) are unknown; try again later > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed > > to construct class in 'cephadm' > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load > > Traceback (most recent call last): > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > self.upgrade = CephadmUpgrade(self) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > self.upgrade_state: Optional[UpgradeState] = > > UpgradeState.from_json(json.loads(t)) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > return cls(**c) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > TypeError: __init__() got an unexpected keyword argument > > 'daemon_types' > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() > > Failed to run module in active mode ('cephadm') > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [crash DEBUG root] > > setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > > Constructed class from module: crash > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [dashboard DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > > Constructed class from module: dashboard > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.920+0000 7f2660c00700 0 [dashboard INFO > > access_control] Loading user roles DB version=2 > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > > sso] Loading SSO DB version=1 > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > > root] server: ssl=yes host=:: port=8443 > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [devicehealth > > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > > Constructed class from module: devicehealth > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [insights DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > > Constructed class from module: insights > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.923+0000 7f2660200700 0 [devicehealth INFO > > root] Starting > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.925+0000 7f2663400700 0 [iostat DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.925+0000 7f2663400700 1 mgr load > > Constructed class from module: iostat > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.925+0000 7f265f800700 0 [insights INFO > > root] Reset curr health slot key health_history/2025-04-05_20 next > > flush None checks check count 0 > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.926+0000 7f2663400700 0 [mirroring DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.931+0000 7f2663400700 1 mgr load > > Constructed class from module: mirroring > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [nfs DEBUG root] > > setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > > Constructed class from module: nfs > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [orchestrator > > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > > Constructed class from module: orchestrator > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.933+0000 7f2663400700 0 [pg_autoscaler > > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.933+0000 7f2663400700 1 mgr load > > Constructed class from module: pg_autoscaler > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.934+0000 7f2663400700 0 [progress DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.934+0000 7f2663400700 1 mgr load > > Constructed class from module: progress > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.934+0000 7f2657600700 0 [pg_autoscaler > > INFO root] _maybe_adjust > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.940+0000 7f2656c00700 0 [progress INFO > > root] Loading... > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.940+0000 7f2663400700 0 [prometheus DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.944+0000 7f2663400700 1 mgr load > > Constructed class from module: prometheus > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > > root] server_addr: :: server_port: 9283 > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > > root] Cache enabled > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > > root] Loaded [&lt;progress.module.GhostEvent object at 0x7f27363ac400>, > > &lt;progress.module.GhostEvent object at 0x7f27363ac438>, > > &lt;progress.module.GhostEvent object at 0x7f27363ac3c8>, > > &lt;progress.module.GhostEvent object at 0x7f27363ac5c0>, > > &lt;progress.module.GhostEvent object at 0x7f27363ac4e0>, > > &lt;progress.module.GhostEvent object at 0x7f27363ac4a8>, > > &lt;progress.module.GhostEvent object at 0x7f278d317e10>, > > &lt;progress.module.GhostEvent object at 0x7f278d4397b8>, > > &lt;progress.module.GhostEvent object at 0x7f278d439908>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed2e8>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed4e0>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed358>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed278>, > > &lt;progress.module.GhostEvent object at 0x7f27363edeb8>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed438>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed400>, > > &lt;progress.module.GhostEvent object at 0x7f27363edb00>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed780>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed240>, > > &lt;progress.module.GhostEvent object at 0x7f27363edba8>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed2b0>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed0b8>, > > &lt;progress.module.GhostEvent object at 0x7f27363edcf8>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed128>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed080>, > > &lt;progress.module.GhostEvent object at 0x7f27363edc18>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed860>, > > &lt;progress.module.GhostEvent object at 0x7f27363eda58>, > > &lt;progress.module.GhostEvent object at 0x7f27363eda90>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed160>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed908>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed940>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed8d0>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed978>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed748>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed898>, > > &lt;progress.module.GhostEvent object at 0x7f27363eda20>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed9b0>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed9e8>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed828>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed7f0>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed7b8>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed668>, > > &lt;progress.module.GhostEvent object at 0x7f27363ed320>, > > &lt;progress.module.GhostEvent object at 0x7f27363ede48>, > > &lt;progress.module.GhostEvent object at 0x7f27363edc50>, > > &lt;progress.module.GhostEvent object at 0x7f27363ede80>, > > &lt;progress.module.GhostEvent object at 0x7f27363edf60>, > > &lt;progress.module.GhostEvent object at 0x7f27363edd68>, > > &lt;progress.module.GhostEvent object at 0x7f27363edda0>] historic events > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > > root] Loaded OSDMap, ready. > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.946+0000 7f2663400700 0 [rbd_support DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.946+0000 7f2655800700 0 [prometheus INFO > > root] starting metric collection thread > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.947+0000 7f2656200700 0 [prometheus INFO > > root] Starting engine... > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > [05/Apr/2025:20:33:48] ENGINE Bus STARTING > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.948+0000 7f2656200700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STARTING > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.948+0000 7f2660c00700 0 [dashboard INFO > > root] Configured CherryPy, starting engine... > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.953+0000 7f2663400700 0 [rbd_support INFO > > root] MirrorSnapshotScheduleHandler: load_schedules > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.957+0000 7f2663400700 0 [rbd_support INFO > > root] load_schedules: csxyz, start_after= > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.989+0000 7f2649a00700 0 [rbd_support INFO > > root] MirrorSnapshotScheduleHandler: starting > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.990+0000 7f2649000700 0 [rbd_support INFO > > root] PerfHandler: starting > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:48.991+0000 7f2663400700 0 [rbd_support INFO > > root] load_task_task: csxyz, start_after= > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.019+0000 7f2648600700 0 [rbd_support INFO > > root] TaskHandler: starting > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.020+0000 7f2663400700 0 [rbd_support INFO > > root] TrashPurgeScheduleHandler: load_schedules > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.022+0000 7f2663400700 0 [rbd_support INFO > > root] load_schedules: csxyz, start_after= > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.024+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > CephFS -> /api/cephfs > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.026+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > CephFsUi -> /ui-api/cephfs > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > IscsiUi -> /ui-api/iscsi > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Iscsi -> /api/iscsi > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > IscsiTarget -> /api/iscsi/target > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > NFSGanesha -> /api/nfs-ganesha > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > NFSGaneshaCluster -> /api/nfs-ganesha/cluster > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > NFSGaneshaExports -> /api/nfs-ganesha/export > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > NFSGaneshaUi -> /ui-api/nfs-ganesha > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Rbd > > -> /api/block/image > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.034+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdSnapshot -> /api/block/image/{image_spec}/snap > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdTrash -> /api/block/image/trash > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdNamespace -> /api/block/pool/{pool_name}/namespace > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroring -> /api/block/mirroring > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroringSummary -> /api/block/mirroring/summary > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroringPoolMode -> /api/block/mirroring/pool > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroringPoolBootstrap -> > > /api/block/mirroring/pool/{pool_name}/bootstrap > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirroringPoolPeer -> /api/block/mirroring/pool/{pool_name}/peer > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.038+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Rgw > > -> /api/rgw > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RgwDaemon -> /api/rgw/daemon > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RgwSite -> /api/rgw/site > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.040+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RgwBucket -> /api/rgw/bucket > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.041+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RgwUser -> /api/rgw/user > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Auth -> /api/auth > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Cluster -> /api/cluster > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.045+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > ClusterConfiguration -> /api/cluster_conf > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.046+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > CrushRule -> /api/crush_rule > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.047+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > CrushRuleUi -> /ui-api/crush_rule > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Orchestrator -> /api/orchestrator > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Daemon -> /api/daemon > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Docs -> /docs > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > ErasureCodeProfile -> /api/erasure_code_profile > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > ErasureCodeProfileUi -> /ui-api/erasure_code_profile > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > FeedbackController -> /api/feedback > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.051+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > FeedbackApiController -> /api/feedback/api_key > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > FeedbackUiController -> /ui-api/feedback/api_key > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > FrontendLogging -> /ui-api/logging > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Grafana -> /api/grafana > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Host -> /api/host > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > HostUi -> /ui-api/host > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Health -> /api/health > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > HomeController -> / > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > [05/Apr/2025:20:33:49] ENGINE Serving on http://:::9283 > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Serving on > > http://:::9283 > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > [05/Apr/2025:20:33:49] ENGINE Bus STARTED > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Bus STARTED > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > > root] Engine started. > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > LangsController -> /ui-api/langs > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Logs -> /api/logs > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > MgrModules -> /api/mgr/module > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Monitor -> /api/monitor > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: Osd > > -> /api/osd > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.060+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > OsdFlagsController -> /api/osd/flags > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > MdsPerfCounter -> /api/perf_counters/mds > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > MonPerfCounter -> /api/perf_counters/mon > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > OsdPerfCounter -> /api/perf_counters/osd > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RgwPerfCounter -> /api/perf_counters/rgw > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > RbdMirrorPerfCounter -> /api/perf_counters/rbd-mirror > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > MgrPerfCounter -> /api/perf_counters/mgr > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > TcmuRunnerPerfCounter -> /api/perf_counters/tcmu-runner > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > PerfCounters -> /api/perf_counters > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.065+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Pool -> /api/pool > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.066+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > PoolUi -> /ui-api/pool > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > PrometheusReceiver -> /api/prometheus_receiver > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Prometheus -> /api/prometheus > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.068+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > PrometheusNotifications -> /api/prometheus/notifications > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.069+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Role -> /api/role > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Scope -> /ui-api/scope > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Saml2 -> /auth/saml2 > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Service -> /api/service > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.071+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Settings -> /api/settings > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > StandardSettings -> /ui-api/standard_settings > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Summary -> /api/summary > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Task -> /api/task > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > Telemetry -> /api/telemetry > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > User -> /api/user > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > UserPasswordPolicy -> /api/user > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > UserChangePassword -> /api/user/{username} > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > FeatureTogglesEndpoint -> /api/feature_toggles > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > > dashboard.controllers._base_controller] Initializing controller: > > MessageOfTheDay -> /ui-api/motd > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.090+0000 7f2647200700 0 [rbd_support INFO > > root] TrashPurgeScheduleHandler: starting > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.096+0000 7f2663400700 1 mgr load > > Constructed class from module: rbd_support > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.102+0000 7f2663400700 0 [restful DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.102+0000 7f2663400700 1 mgr load > > Constructed class from module: restful > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.108+0000 7f2645e00700 0 [restful INFO > > root] server_addr: :: server_port: 8003 > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.109+0000 7f2663400700 0 [snap_schedule > > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.116+0000 7f2645e00700 0 [restful WARNING > > root] server not running: no certificate configured > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.257+0000 7f2660c00700 0 [dashboard INFO > > dashboard.module] Engine started. > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed > > to construct class in 'snap_schedule' > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load > > Traceback (most recent call last): > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in > > __init__ > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > self.client = SnapSchedClient(self) > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > > 158, in __init__ > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > with self.get_schedule_db(fs_name) as conn_mgr: > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > > 192, in get_schedule_db > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > db.executescript(dump) > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > sqlite3.OperationalError: table schedules already exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() > > Failed to run module in active mode ('snap_schedule') > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.274+0000 7f2663400700 0 [status DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.274+0000 7f2663400700 1 mgr load > > Constructed class from module: status > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.275+0000 7f2663400700 0 [telemetry DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.276+0000 7f2663400700 1 mgr load > > Constructed class from module: telemetry > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.276+0000 7f2663400700 0 [volumes DEBUG > > root] setting log level based on debug_mgr: INFO (2/5) > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.284+0000 7f2660200700 0 [devicehealth INFO > > root] Check health > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.290+0000 7f2663400700 1 mgr load > > Constructed class from module: volumes > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.295+0000 7f262b000700 -1 client.0 error > > registering admin socket command: (17) File exists > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:49.883+0000 7f2662a00700 0 > > log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail > > Apr 05 20:33:50 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:50.875+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail > > Apr 05 20:33:52 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:52] "GET /metrics > > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > > Apr 05 20:33:52 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:52.753+0000 7f264f400700 0 [prometheus INFO > > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > > [05/Apr/2025:20:33:52] "GET /metrics HTTP/1.1" 200 33671 "" > > "Prometheus/2.33.4" > > Apr 05 20:33:52 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:52.876+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v5: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail > > Apr 05 20:33:53 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:53] "GET /metrics > > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > > Apr 05 20:33:53 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:53.772+0000 7f264ea00700 0 [prometheus INFO > > cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - > > [05/Apr/2025:20:33:53] "GET /metrics HTTP/1.1" 200 33671 "" > > "Prometheus/2.33.4" > > Apr 05 20:33:53 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:53.870+0000 7f2663400700 0 [insights INFO > > root] Received health check update 0 pending > > Apr 05 20:33:53 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:53.871+0000 7f265f800700 0 [insights INFO > > root] Applying 1 health updates to slot key > > health_history/2025-04-05_20 next flush None checks check count 0 > > Apr 05 20:33:53 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:53.871+0000 7f265f800700 0 [insights INFO > > root] Maybe flushing slot key health_history/2025-04-05_20 next > > flush None checks check count 0 needed False > > Apr 05 20:33:54 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:54.877+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v6: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; 0 > > B/s rd, 12 KiB/s wr, 2 op/s > > Apr 05 20:33:55 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:55.796+0000 7f2662a00700 2 mgr.server > > handle_open ignoring open from mgr.cn02.ceph.xyz.corp.ggixgj > > 192.168.47.72:0/2580067884; not ready for session (expect reconnect) > > Apr 05 20:33:56 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:56.878+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v7: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 439 B/s rd, 27 KiB/s wr, 6 op/s > > Apr 05 20:33:57 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:57] "GET /metrics > > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > > Apr 05 20:33:57 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:57.038+0000 7f264e000700 0 [prometheus INFO > > cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - > > [05/Apr/2025:20:33:57] "GET /metrics HTTP/1.1" 200 33671 "" > > "Prometheus/2.33.4" > > Apr 05 20:33:58 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:58.870+0000 7f2663400700 0 [insights INFO > > root] Received health check update 0 pending > > Apr 05 20:33:58 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:58.878+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v8: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 341 B/s rd, 21 KiB/s wr, 4 op/s > > Apr 05 20:33:58 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:58.897+0000 7f265f800700 0 [insights INFO > > root] Applying 1 health updates to slot key > > health_history/2025-04-05_20 next flush None checks check count 0 > > Apr 05 20:33:58 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:33:58.897+0000 7f265f800700 0 [insights INFO > > root] Maybe flushing slot key health_history/2025-04-05_20 next > > flush None checks check count 0 needed False > > Apr 05 20:34:00 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:00.880+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v9: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 279 B/s rd, 24 KiB/s wr, 6 op/s > > Apr 05 20:34:02 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:02] "GET /metrics > > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > > Apr 05 20:34:02 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:02.753+0000 7f264d600700 0 [prometheus INFO > > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > > [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 33671 "" > > "Prometheus/2.33.4" > > Apr 05 20:34:02 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:02.880+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v10: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 255 B/s rd, 22 KiB/s wr, 5 op/s > > Apr 05 20:34:03 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:03] "GET /metrics > > HTTP/1.1" 200 33671 "" "Prometheus/2.33.4" > > Apr 05 20:34:03 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:03.772+0000 7f264cc00700 0 [prometheus INFO > > cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - > > [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 33671 "" > > "Prometheus/2.33.4" > > Apr 05 20:34:03 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:03.870+0000 7f2663400700 0 [insights INFO > > root] Received health check update 0 pending > > Apr 05 20:34:03 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:03.873+0000 7f265f800700 0 [insights INFO > > root] Applying 1 health updates to slot key > > health_history/2025-04-05_20 next flush None checks check count 0 > > Apr 05 20:34:03 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:03.874+0000 7f265f800700 0 [insights INFO > > root] Maybe flushing slot key health_history/2025-04-05_20 next > > flush None checks check count 0 needed False > > Apr 05 20:34:04 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:04.881+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v11: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 255 B/s rd, 59 KiB/s wr, 11 op/s > > Apr 05 20:34:06 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:06.882+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v12: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 511 B/s rd, 66 KiB/s wr, 13 op/s > > Apr 05 20:34:07 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:07] "GET /metrics > > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > > Apr 05 20:34:07 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:07.037+0000 7f264c200700 0 [prometheus INFO > > cherrypy.access.139806389938328] ::ffff:192.168.47.73 - - > > [05/Apr/2025:20:34:07] "GET /metrics HTTP/1.1" 200 267239 "" > > "Prometheus/2.33.4" > > Apr 05 20:34:08 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:08.871+0000 7f2663400700 0 [insights INFO > > root] Received health check update 0 pending > > Apr 05 20:34:08 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:08.883+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v13: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 255 B/s rd, 55 KiB/s wr, 11 op/s > > Apr 05 20:34:08 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:08.907+0000 7f265f800700 0 [insights INFO > > root] Applying 1 health updates to slot key > > health_history/2025-04-05_20 next flush None checks check count 0 > > Apr 05 20:34:08 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:08.907+0000 7f265f800700 0 [insights INFO > > root] Maybe flushing slot key health_history/2025-04-05_20 next > > flush None checks check count 0 needed False > > Apr 05 20:34:10 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:10.884+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v14: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 255 B/s rd, 79 KiB/s wr, 16 op/s > > Apr 05 20:34:12 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:12] "GET /metrics > > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > > Apr 05 20:34:12 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:12.753+0000 7f264b800700 0 [prometheus INFO > > cherrypy.access.139806389938328] ::ffff:192.168.47.72 - - > > [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 267239 "" > > "Prometheus/2.33.4" > > Apr 05 20:34:12 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:12.884+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v15: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 255 B/s rd, 73 KiB/s wr, 14 op/s > > Apr 05 20:34:13 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:13] "GET /metrics > > HTTP/1.1" 200 267239 "" "Prometheus/2.33.4" > > Apr 05 20:34:13 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:13.772+0000 7f264ae00700 0 [prometheus INFO > > cherrypy.access.139806389938328] ::ffff:192.168.47.71 - - > > [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 267239 "" > > "Prometheus/2.33.4" > > Apr 05 20:34:13 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:13.871+0000 7f2663400700 0 [insights INFO > > root] Received health check update 0 pending > > Apr 05 20:34:13 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:13.883+0000 7f265f800700 0 [insights INFO > > root] Applying 1 health updates to slot key > > health_history/2025-04-05_20 next flush None checks check count 0 > > Apr 05 20:34:13 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:13.883+0000 7f265f800700 0 [insights INFO > > root] Maybe flushing slot key health_history/2025-04-05_20 next > > flush None checks check count 0 needed False > > Apr 05 20:34:14 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:14.886+0000 7f2661600700 0 > > log_channel(cluster) log [DBG] : pgmap v16: 610 pgs: 610 > > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > > 82 KiB/s rd, 103 KiB/s wr, 113 op/s > > Apr 05 20:34:15 cn03.ceph.xyz.corp > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > debug 2025-04-05T20:34:15.754+0000 7f2644000700 0 [dashboard I [Text gekürzt]

ceph config-key get mgr/cephadm/upgrade_state {"target_name": "quay.io/ceph/ceph:v17.2.0", "progress_id": "e7e1a809-558d-43a7-842a-c6229fdc57af", "target_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "target_digests": ["quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667"], "target_version": "17.2.0", "fs_original_max_mds": null, "fs_original_allow_standby_replay": null, "error": null, "paused": false, "daemon_types": null, "hosts": null, "services": null, "total_count": null, "remaining_count": null} What should I do next? Thank you! -jeremy

On Sunday, Apr 06, 2025 at 1:38 AM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: Can you check if you have this config-key?

ceph config-key get mgr/cephadm/upgrade_state

If you reset the MGRs, it might be necessary to clear this key, otherwise you might end up in some inconsistency. Just to be sure.

Zitat von Jeremy Hansen <jeremy@skidrow.la>:

...

Thanks. I’m trying to be extra careful since this cluster is actually in use. I’ll wait for your feedback.

-jeremy

...

On Saturday, Apr 05, 2025 at 3:39 PM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: No, that's not necessary, just edit the unit.run file for the MGRs to use a different image. See Frédéric's instructions:

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/32APKOXKRAI...

But I'm not entirely sure if you need to clear some config-keys first in order to reset the upgrade state. If I have time, I'll try to check tomorrow, or on Monday.

Zitat von Jeremy Hansen <jeremy@skidrow.la>:

...

Would I follow this process to downgrade?

https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-...

...

Thank you

...

On Saturday, Apr 05, 2025 at 2:04 PM, Jeremy Hansen <jeremy@skidrow.la (mailto:jeremy@skidrow.la)> wrote: ceph -s claims things are healthy:

ceph -s cluster: id: 95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1 health: HEALTH_OK

services: mon: 3 daemons, quorum cn01,cn03,cn02 (age 20h) mgr: cn03.negzvb(active, since 26m), standbys: cn01.tjmtph, cn02.ceph.xyz.corp.ggixgj mds: 1/1 daemons up, 2 standby osd: 15 osds: 15 up (since 19h), 15 in (since 14M)

data: volumes: 1/1 healthy pools: 6 pools, 610 pgs objects: 284.59k objects, 1.1 TiB usage: 3.3 TiB used, 106 TiB / 109 TiB avail pgs: 610 active+clean

io: client: 255 B/s rd, 1.2 MiB/s wr, 10 op/s rd, 16 op/s wr

— How do I downgrade if the orch is down?

Thank you -jeremy

...

On Saturday, Apr 05, 2025 at 1:56 PM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: It would help if you only pasted the relevant parts. Anyway, these two sections stand out:

---snip--- Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again later Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed to construct class in 'cephadm' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade = CephadmUpgrade(self) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: return cls(**c) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: TypeError: __init__() got an unexpected keyword argument

...

...

...

Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('cephadm')

Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed to construct class in 'snap_schedule' Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.client = SnapSchedClient(self) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 158, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: with self.get_schedule_db(fs_name) as conn_mgr: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 192, in get_schedule_db Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: db.executescript(dump) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: sqlite3.OperationalError: table schedules already exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('snap_schedule') ---snip---

Your cluster seems to be in an error state (ceph -s) because of an unknown PG. It's recommended to have a healthy cluster before attemping an upgrade. It's possible that these errors come from the not upgraded MGR, I'm not sure.

Since the upgrade was only successful for two MGRs, I am thinking about downgrading both MGRs back to 16.2.15, then retry an upgrade to a newer version, either 17.2.8 or 18.2.4. I haven't checked the snap_schedule error yet, though. Maybe someone else knows

...

...

...

Zitat von Jeremy Hansen <jeremy@skidrow.la>:

> This is a small three node cluster. Here’s logs from all three nodes > during the failover. The current manager was cn02 and it failed over > to cn03. > > cn01 > —— > Apr 05 20:33:35 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:35.823+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v821: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.170+0000 7f2aaec00700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.823+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v822: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:33:39 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:39] "GET /metrics > HTTP/1.1" 200 267241 "" "Prometheus/2.33.4" > Apr 05 20:33:39 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:39.825+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v823: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 97 KiB/s wr, 12 op/s > Apr 05 20:33:41 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:41.826+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v824: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 85 KiB/s wr, 10 op/s > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.170+0000 7f2aaec00700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:42] "GET /metrics > HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics > HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:43.827+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v825: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 511 B/s rd, 133 KiB/s wr, 18 op/s > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 -1 mgr > handle_mgr_map I was active but no longer am > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn e: > '/usr/bin/ceph-mgr' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 0: > '/usr/bin/ceph-mgr' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr

respawn 1: '-n'

...

> Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 2: > 'mgr.cn01.tjmtph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 3: '-f' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 4: > '--setuser' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 5: > 'ceph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 6: > '--setgroup' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 7: > 'ceph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 8: > '--default-log-to-file=false' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 9: > '--default-log-to-stderr=true' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 10: > '--default-log-stderr-prefix=debug ' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn > respawning with exe /usr/bin/ceph-mgr > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn > exe_path /proc/self/exe > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ignoring --setuser ceph since I am not root > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ignoring --setgroup ceph since I am not root > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 ceph version > 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), > process ceph-mgr, pid 2 > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 pidfile_write: > ignore empty --pid-file > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.477+0000 7fb814447000 1 mgr[py] Loading > python module 'alerts' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.612+0000 7fb814447000 -1 mgr[py] Module > alerts has missing NOTIFY_TYPES member > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.612+0000 7fb814447000 1 mgr[py] Loading > python module 'balancer' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.726+0000 7fb814447000 -1 mgr[py] Module > balancer has missing NOTIFY_TYPES member > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.726+0000 7fb814447000 1 mgr[py] Loading > python module 'cephadm' > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.417+0000 7fb814447000 1 mgr[py] Loading > python module 'crash' > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.540+0000 7fb814447000 -1 mgr[py] Module > crash has missing NOTIFY_TYPES member > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.540+0000 7fb814447000 1 mgr[py] Loading > python module 'dashboard' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.210+0000 7fb814447000 1 mgr[py] Loading > python module 'devicehealth' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.331+0000 7fb814447000 -1 mgr[py] Module > devicehealth has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.331+0000 7fb814447000 1 mgr[py] Loading > python module 'diskprediction_local' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.653+0000 7fb814447000 -1 mgr[py] Module > diskprediction_local has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.654+0000 7fb814447000 1 mgr[py] Loading > python module 'influx' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.769+0000 7fb814447000 -1 mgr[py] Module > influx has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.769+0000 7fb814447000 1 mgr[py] Loading > python module 'insights' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.884+0000 7fb814447000 1 mgr[py] Loading > python module 'iostat' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.082+0000 7fb814447000 -1 mgr[py] Module > iostat has missing NOTIFY_TYPES member > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.082+0000 7fb814447000 1 mgr[py] Loading > python module 'k8sevents' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.679+0000 7fb814447000 1 mgr[py] Loading > python module 'localpool' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.796+0000 7fb814447000 1 mgr[py] Loading > python module 'mds_autoscaler' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.983+0000 7fb814447000 1 mgr[py] Loading > python module 'mirroring' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.118+0000 7fb814447000 1 mgr[py] Loading > python module 'nfs' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.315+0000 7fb814447000 -1 mgr[py] Module > nfs has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.315+0000 7fb814447000 1 mgr[py] Loading > python module 'orchestrator' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.607+0000 7fb814447000 -1 mgr[py] Module > orchestrator has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.607+0000 7fb814447000 1 mgr[py] Loading > python module 'osd_perf_query' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.723+0000 7fb814447000 -1 mgr[py] Module > osd_perf_query has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.723+0000 7fb814447000 1 mgr[py] Loading > python module 'osd_support' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.837+0000 7fb814447000 -1 mgr[py] Module > osd_support has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.837+0000 7fb814447000 1 mgr[py] Loading > python module 'pg_autoscaler' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.970+0000 7fb814447000 -1 mgr[py] Module > pg_autoscaler has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.970+0000 7fb814447000 1 mgr[py] Loading > python module 'progress' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.082+0000 7fb814447000 -1 mgr[py] Module > progress has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.082+0000 7fb814447000 1 mgr[py] Loading > python module 'prometheus' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.617+0000 7fb814447000 -1 mgr[py] Module > prometheus has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.617+0000 7fb814447000 1 mgr[py] Loading > python module 'rbd_support' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.761+0000 7fb814447000 -1 mgr[py] Module > rbd_support has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.761+0000 7fb814447000 1 mgr[py] Loading > python module 'restful' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.023+0000 7fb814447000 1 mgr[py] Loading > python module 'rook' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.788+0000 7fb814447000 -1 mgr[py] Module > rook has missing NOTIFY_TYPES member > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.788+0000 7fb814447000 1 mgr[py] Loading > python module 'selftest' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.912+0000 7fb814447000 -1 mgr[py] Module > selftest has missing NOTIFY_TYPES member > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.912+0000 7fb814447000 1 mgr[py] Loading > python module 'snap_schedule' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.038+0000 7fb814447000 -1 mgr[py] Module > snap_schedule has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.038+0000 7fb814447000 1 mgr[py] Loading > python module 'stats' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.155+0000 7fb814447000 1 mgr[py] Loading > python module 'status' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.280+0000 7fb814447000 -1 mgr[py] Module > status has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.280+0000 7fb814447000 1 mgr[py] Loading > python module 'telegraf' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.398+0000 7fb814447000 -1 mgr[py] Module > telegraf has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.398+0000 7fb814447000 1 mgr[py] Loading > python module 'telemetry' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.594+0000 7fb814447000 -1 mgr[py] Module > telemetry has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.594+0000 7fb814447000 1 mgr[py] Loading > python module 'test_orchestrator' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.771+0000 7fb814447000 -1 mgr[py] Module > test_orchestrator has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.771+0000 7fb814447000 1 mgr[py] Loading > python module 'volumes' > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.225+0000 7fb814447000 -1 mgr[py] Module > volumes has missing NOTIFY_TYPES member > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.225+0000 7fb814447000 1 mgr[py] Loading > python module 'zabbix' > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.342+0000 7fb814447000 -1 mgr[py] Module > zabbix has missing NOTIFY_TYPES member > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.344+0000 7fb807c00700 0 > ms_deliver_dispatch: unhandled message 0x555c78fe1080 mon_map magic: > 0 v1 from mon.0 v2:192.168.47.71:3300/0 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.344+0000 7fb808600700 0 [dashboard DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load > Constructed class from module: dashboard > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 0 [prometheus DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load > Constructed class from module: prometheus > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb6abe00700 0 [dashboard INFO > root] server: ssl=yes host=:: port=8443 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > root] server_addr: :: server_port: 9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > root] Starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Bus STARTING > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTING > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO > root] Configured CherryPy, starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO > root] Starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > CherryPy Checker: > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > The Application mounted at '' has an empty config. > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Serving on http://:::9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Serving on > http://:::9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Bus STARTED > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTED > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.451+0000 7fb6b4a00700 0 [prometheus INFO > root] Engine started. > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.489+0000 7fb6abe00700 0 [dashboard INFO > root] Engine started... > Apr 05 20:33:59 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:59] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:59 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:59.603+0000 7fb76c600700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:59] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:02 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:02] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:02 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:02.476+0000 7fb763a00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:03 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:03.061+0000 7fb75ae00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:09 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:09] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:09 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:09.602+0000 7fb752200700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:09] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:12 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:12] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:12 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:12.476+0000 7fb749600700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:13 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:13.060+0000 7fb740a00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.493+0000 7fb714e00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.494+0000 7fb714e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52368] [GET] [303] [0.003s] [96.0B] > [0031fbdd-947f-4c6c-943d-55cf256edfba] /api/summary > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.512+0000 7fb70c200700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.513+0000 7fb70c200700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52369] [GET] [303] [0.001s] [96.0B] > [76c9a6ab-aae0-4ef9-815e-dbd0ac8f8b49] /api/feature_toggles > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.527+0000 7fb703600700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.528+0000 7fb703600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52370] [GET] [303] [0.001s] [96.0B] > [db017da3-e6f6-45fc-9d99-7b9f3b3859f2] /api/health/minimal > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.541+0000 7fb6faa00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.542+0000 7fb6faa00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52371] [GET] [303] [0.001s] [96.0B] > [564ba322-b99c-49de-a3a3-e0ed422bfae7] /api/prometheus > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.616+0000 7fb6f1e00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.617+0000 7fb6f1e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52372] [GET] [303] [0.001s] [96.0B] > [1f349e52-9a10-4fda-b13f-621785208ea9] /api/prometheus/rules > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52368] [GET] [303] [0.001s] [96.0B] > [30d749d7-c972-44f0-88c7-b70078e2385c] /api/prometheus/notifications > Apr 05 20:34:19 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:19] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:19 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:19.601+0000 7fb737e00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:19] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > > cn02 > —— > Apr 05 20:33:40 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:33:40 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:40.701+0000 7ff710400700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:33:43 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:33:43 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:43.287+0000 7ff707800700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.370+0000 7ff7e7c00700 1 mgr

...

handle_mgr_map

...

> Activating! > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.371+0000 7ff7e7c00700 1 mgr handle_mgr_map I am

...

now

...

> activating > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root]

...

Stopping

...

> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root]

...

Stopped

...

> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root]

...

Stopping

...

> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root]

...

Stopped

...

> engine > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE

...

Bus

...

> STOPPING > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.383+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE

...

Bus

...

> STOPPING > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.394+0000 7ff68d000700 0 [dashboard INFO root]

...

Engine

...

> stopped. > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283))

...

shut

...

> down > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.395+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283))

...

shut

...

> down > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE

...

Bus

...

> STOPPED > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE

...

Bus

...

> STOPPED > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO root]

...

Engine

...

> stopped. > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts INFO

...

root]

...

> Init > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> alerts > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 0 [balancer DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> balancer > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff74d800700 0 [alerts INFO

...

root]

...

> Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.401+0000 7ff67aa00700 0 [cephadm DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.402+0000 7ff710400700 0 [balancer INFO

...

root]

...

> Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Optimize

...

plan

...

> auto_2025-04-05_20:33:45 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Mode upmap, max

...

misplaced

...

> 0.050000 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.409+0000 7ff710400700 0 [balancer INFO root] Some PGs (1.000000) are unknown;

...

...

again

...

> later > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.551+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> cephadm > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 0 [crash DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> crash > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 0 [dashboard DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> dashboard > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> devicehealth > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO access_control] Loading user roles

...

DB

...

> version=2 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO sso] Loading SSO

...

DB

...

> version=1 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO root] server: ssl=yes

...

host=::

...

> port=8443 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.557+0000 7ff674600700 0 [devicehealth INFO

...

root]

...

> Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.558+0000 7ff67aa00700 0 [insights DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.559+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> insights > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.561+0000 7ff674600700 0 [devicehealth INFO root]

...

Check

...

> health > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.566+0000 7ff671e00700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 0 [iostat DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> iostat > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.572+0000 7ff67aa00700 0 [mirroring DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.579+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 0 [nfs DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> nfs > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> orchestrator > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> pg_autoscaler > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.625+0000 7ff662a00700 0 [pg_autoscaler INFO

...

root]

...

> _maybe_adjust > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 0 [progress DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> progress > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.72' (ECDSA) to the list of

...

known

...

> hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.73' (ECDSA) to the list of

...

known

...

> hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 0 [prometheus DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> prometheus > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff661600700 0 [progress INFO

...

root]

...

> Loading... > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] server_addr: ::

...

server_port:

...

> 9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root]

...

Cache

...

> enabled > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff65ee00700 0 [prometheus INFO root] starting metric

...

collection

...

> thread > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.640+0000 7ff67aa00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7ff7443993c8>, <progress.module.GhostEvent object at 0x7ff744399400>, <progress.module.GhostEvent object at 0x7ff744399438>, <progress.module.GhostEvent object at 0x7ff744399470>, <progress.module.GhostEvent object at 0x7ff7443994a8>, <progress.module.GhostEvent object at 0x7ff7443994e0>, <progress.module.GhostEvent object at 0x7ff744399518>, <progress.module.GhostEvent object at 0x7ff744399550>, <progress.module.GhostEvent object at 0x7ff744399588>, <progress.module.GhostEvent object at 0x7ff7443995c0>, <progress.module.GhostEvent object at 0x7ff7443995f8>, <progress.module.GhostEvent object at 0x7ff744399630>, <progress.module.GhostEvent object at 0x7ff744399668>, <progress.module.GhostEvent object at 0x7ff7443996a0>, <progress.module.GhostEvent object at 0x7ff7443996d8>, <progress.module.GhostEvent object at 0x7ff744399710>, <progress.module.GhostEvent object at 0x7ff744399748>, <progress.module.GhostEvent object at 0x7ff744399780>, <progress.module.GhostEvent object at 0x7ff7443997b8>, <progress.module.GhostEvent object at 0x7ff7443997f0>, <progress.module.GhostEvent object at 0x7ff744399828>, <progress.module.GhostEvent object at 0x7ff744399860>, <progress.module.GhostEvent object at 0x7ff744399898>, <progress.module.GhostEvent object at 0x7ff7443998d0>, <progress.module.GhostEvent object at 0x7ff744399908>, <progress.module.GhostEvent object at 0x7ff744399940>, <progress.module.GhostEvent object at 0x7ff744399978>, <progress.module.GhostEvent object at 0x7ff7443999b0>, <progress.module.GhostEvent object at 0x7ff7443999e8>, <progress.module.GhostEvent object at 0x7ff744399a20>, <progress.module.GhostEvent object at 0x7ff744399a58>, <progress.module.GhostEvent object at 0x7ff744399a90>, <progress.module.GhostEvent object at 0x7ff744399ac8>, <progress.module.GhostEvent object at 0x7ff744399b00>, <progress.module.GhostEvent object at 0x7ff744399b38>, <progress.module.GhostEvent object at 0x7ff744399b70>, <progress.module.GhostEvent object at 0x7ff744399ba8>, <progress.module.GhostEvent object at 0x7ff744399be0>, <progress.module.GhostEvent object at 0x7ff744399c18>, <progress.module.GhostEvent object at 0x7ff744399c50>, <progress.module.GhostEvent object at 0x7ff744399c88>, <progress.module.GhostEvent object at 0x7ff744399cc0>, <progress.module.GhostEvent object at 0x7ff744399cf8>, <progress.module.GhostEvent object at 0x7ff744399d30>, <progress.module.GhostEvent object at 0x7ff744399d68>, <progress.module.GhostEvent object at 0x7ff744399da0>, <progress.module.GhostEvent object at 0x7ff744399dd8>, <progress.module.GhostEvent object at 0x7ff744399e10>, <progress.module.GhostEvent object at 0x7ff744399e48>, <progress.module.GhostEvent object at 0x7ff744399e80>]

...

historic

...

> events > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded

...

OSDMap,

...

> ready. > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Processing OSDMap

...

change

...

> 27185..27185 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff676400700 0 [dashboard INFO root] Configured CherryPy,

...

starting

...

> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff660200700 0 [prometheus INFO root]

...

Starting

...

> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE

...

Bus

...

> STARTING > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.647+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE

...

Bus

...

> STARTING > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] recovery

...

thread

...

> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root]

...

starting

...

> setup > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> rbd_support > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.71' (ECDSA) to the list of

...

known

...

> hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 0 [restful DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> restful > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.660+0000 7ff651c00700 0 [restful INFO root] server_addr: ::

...

server_port:

...

> 8003 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.661+0000 7ff654e00700 0 [rbd_support INFO root]

...

MirrorSnapshotScheduleHandler:

...

> load_schedules > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.697+0000 7ff651c00700 0 [restful WARNING root] server not running: no

...

certificate

...

> configured > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.705+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules:

...

csxyz,

...

> start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.706+0000 7ff67aa00700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.707+0000 7ff64f400700 0 [rbd_support INFO root]

...

MirrorSnapshotScheduleHandler:

...

> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.708+0000 7ff64ea00700 0 [rbd_support INFO root]

...

PerfHandler:

...

> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.709+0000 7ff654e00700 0 [rbd_support INFO root] load_task_task:

...

csxyz,

...

> start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.718+0000 7ff649a00700 0 [rbd_support INFO root]

...

TaskHandler:

...

> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.719+0000 7ff654e00700 0 [rbd_support INFO root]

...

TrashPurgeScheduleHandler:

...

> load_schedules > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.721+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules:

...

csxyz,

...

> start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff649000700 0 [rbd_support INFO root]

...

TrashPurgeScheduleHandler:

...

> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff654e00700 0 [rbd_support INFO root]

...

setup

...

> complete > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.726+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> snap_schedule > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 0 [status DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> status > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.728+0000 7ff67aa00700 0 [telemetry DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> telemetry > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 0 [volumes DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for

...

volume

...

> 'cssnfsxyz' > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for

...

volume

...

> 'cssnfsxyz' > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 1 mgr load Constructed class from

...

module:

...

> volumes > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)

...

File

...

> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:

...

CherryPy

...

> Checker: > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an

...

empty

...

> config. > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:

...

...

> Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Serving

...

on

...

> http://:::9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Serving

...

on

...

> http://:::9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE

...

Bus

...

> STARTED > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE

...

Bus

...

> STARTED > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO root]

...

Engine

...

> started. > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.785+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS

...

->

...

> /api/cephfs > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.787+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi

...

->

...

> /ui-api/cephfs > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi

...

->

...

> /ui-api/iscsi > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi

...

->

...

> /api/iscsi > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget

...

->

...

> /api/iscsi/target > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster

...

->

...

> /api/nfs-ganesha/cluster > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports

...

->

...

> /api/nfs-ganesha/export > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi

...

->

...

> /ui-api/nfs-ganesha > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd

...

->

...

> /api/block/image > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdStatus

...

->

...

> /ui-api/block/rbd > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot

...

->

...

> /api/block/image/{image_spec}/snap > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.796+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash

...

->

...

> /api/block/image/trash > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace

...

->

...

> /api/block/pool/{pool_name}/namespace > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool

...

->

...

> /api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.799+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi

...

->

...

> /ui-api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.800+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RBDPool

...

->

...

> /api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator

...

->

...

> /ui-api/orchestrator > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service

...

->

...

> /api/service > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring

...

->

...

> /api/block/mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary

...

->

...

> /api/block/mirroring/summary > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode

...

->

...

> /api/block/mirroring/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap

...

->

...

> /api/block/mirroring/pool/{pool_name}/bootstrap > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer

...

->

...

> /api/block/mirroring/pool/{pool_name}/peer > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.804+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringStatus

...

->

...

> /ui-api/block/mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw

...

->

...

> /ui-api/rgw > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon

...

->

...

> /api/rgw/daemon > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite

...

->

...

> /api/rgw/site > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket

...

->

...

> /api/rgw/bucket > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.807+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser

...

->

...

> /api/rgw/user > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.810+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth

...

->

...

> /api/auth > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster

...

->

...

> /api/cluster > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration

...

->

...

> /api/cluster_conf > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.813+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule

...

->

...

> /api/crush_rule > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi

...

->

...

> /ui-api/crush_rule > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon

...

->

...

> /api/daemon > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs

...

->

...

> /docs > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile

...

->

...

> /api/erasure_code_profile > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.816+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi

...

->

...

> /ui-api/erasure_code_profile > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging

...

->

...

> /ui-api/logging > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana

...

->

...

> /api/grafana > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host

...

->

...

> /api/host > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.819+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi

...

->

...

> /ui-api/host > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health

...

->

...

> /api/health > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController

...

->

...

> / > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.822+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController

...

->

...

> /ui-api/langs > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LoginController

...

->

...

> /ui-api/login > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs

...

->

...

> /api/logs > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules

...

->

...

> /api/mgr/module > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor

...

->

...

> /api/monitor > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd

...

->

...

> /api/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.828+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdUi

...

->

...

> /ui-api/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.830+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController

...

->

...

> /api/osd/flags > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter

...

->

...

> /api/perf_counters/mds > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter

...

->

...

> /api/perf_counters/mon > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter

...

->

...

> /api/perf_counters/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter

...

->

...

> /api/perf_counters/rgw > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter

...

->

...

> /api/perf_counters/rbd-mirror > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter

...

->

...

> /api/perf_counters/mgr > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter

...

->

...

> /api/perf_counters/tcmu-runner > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters

...

->

...

> /api/perf_counters > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver

...

->

...

> /api/prometheus_receiver > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus

...

->

...

> /api/prometheus > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications

...

->

...

> /api/prometheus/notifications > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role

...

->

...

> /api/role > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope

...

->

...

> /ui-api/scope > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2

...

->

...

> /auth/saml2 > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings

...

->

...

> /api/settings > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings

...

->

...

> /ui-api/standard_settings > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary

...

->

...

> /api/summary > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task

...

->

...

> /api/task > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry

...

->

...

> /api/telemetry > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.842+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User

...

->

...

> /api/user > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy

...

->

...

> /api/user > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword

...

->

...

> /api/user/{username} > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint

...

->

...

> /api/feature_toggles > Apr 05 20:33:45 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.844+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay

...

->

...

> /ui-api/motd > Apr 05 20:33:46 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.050+0000 7ff676400700 0 [dashboard INFO dashboard.module]

...

Engine

...

> started. > Apr 05 20:33:46 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.378+0000 7ff67a000700 0 log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109

...

TiB

...

> avail > Apr 05 20:33:47 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:47.379+0000 7ff678c00700 0 log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109

...

TiB

...

> avail > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.568+0000 7ff65b200700 0 [prometheus INFO cherrypy.access.140699978276536] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 [cephadm INFO cephadm.serve]

...

Updating

...

> cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 log_channel(cephadm) log [INF] :

...

Updating

...

> cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 [cephadm INFO cephadm.serve]

...

Updating

...

> cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve]

...

Updating

...

> cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 log_channel(cephadm) log [INF] :

...

Updating

...

> cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] :

...

Updating

...

> cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 [cephadm INFO cephadm.serve]

...

Updating

...

> cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 log_channel(cephadm) log [INF] :

...

Updating

...

> cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 [cephadm INFO cephadm.serve]

...

Updating

...

> cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 log_channel(cephadm) log [INF] :

...

Updating

...

> cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve]

...

Updating

...

> cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] :

...

Updating

...

> cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.812+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 824b8fde-5a37-4436-af45-a7d8255e2c66 does

...

not

...

> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 621cb660-d584-45e4-8033-4a476e804bba does

...

not

...

> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 007a4dd2-46d3-4655-9c93-c02f97e555cb does

...

not

...

> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 6b0fbff2-881e-4203-9149-f0977bc13eab does

...

not

...

> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 8a1a90b8-1f2a-48c8-924f-6e74032b6319 does

...

not

...

> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.815+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3c322d26-9edb-4d26-b4bc-97c330f5a1d8 does

...

not

...

> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev f3e5a517-d643-4fa5-b127-57716264e4e3 does

...

not

...

> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3fb020c7-92e8-4eb1-a884-2d04e5bca889 does

...

not

...

> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.840+0000 7ff6a7400700 0 [progress WARNING root] complete: ev bf9e0705-075e-4134-b77f-aa64fe92112a does

...

not

...

> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade

...

myself

...

> (mgr.cn02.ceph.xyz.corp.ggixgj) > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade

...

myself

...

> (mgr.cn02.ceph.xyz.corp.ggixgj) > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [progress INFO root] update: starting ev e7e1a809-558d-43a7-842a-c6229fdc57af (Upgrade

...

to

...

> 17.2.0) > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 [cephadm INFO cephadm.services.cephadmservice] Failing over to

...

other

...

> MGR > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Failing over to

...

other

...

> MGR > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 -1 mgr handle_mgr_map I was active but no

...

longer

...

> am > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

e:

...

> '/usr/bin/ceph-mgr' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

0:

...

> '/usr/bin/ceph-mgr' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

1:

...

> '-n' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

2:

...

> 'mgr.cn02.ceph.xyz.corp.ggixgj' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

3:

...

> '-f' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

4:

...

> '--setuser' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

5:

...

> 'ceph' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

6:

...

> '--setgroup' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

7:

...

> 'ceph' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

8:

...

> '--default-log-to-file=false' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

9:

...

> '--default-log-to-stderr=true' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 10:

...

'--default-log-stderr-prefix=debug

...

> ' > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn respawning with

...

exe

...

> /usr/bin/ceph-mgr > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn

...

exe_path

...

> /proc/self/exe > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setuser ceph since I am

...

not

...

> root > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setgroup ceph since I am

...

not

...

> root > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable), process ceph-mgr,

...

pid

...

> 2 > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 pidfile_write: ignore

...

empty

...

> --pid-file > Apr 05 20:33:48 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.976+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'alerts' > Apr 05 20:33:49 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 -1 mgr[py] Module alerts has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:49 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'balancer' > Apr 05 20:33:49 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 -1 mgr[py] Module balancer has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:49 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'cephadm' > Apr 05 20:33:49 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.568+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'crash' > Apr 05 20:33:49 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 -1 mgr[py] Module crash has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:49 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'dashboard' > Apr 05 20:33:50 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.276+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'devicehealth' > Apr 05 20:33:50 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.415+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'diskprediction_local' > Apr 05 20:33:50 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 -1 mgr[py] Module diskprediction_local has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:50 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'influx' > Apr 05 20:33:50 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 -1 mgr[py] Module influx has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:50 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'insights' > Apr 05 20:33:50 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.919+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'iostat' > Apr 05 20:33:51 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 -1 mgr[py] Module iostat has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:51 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'k8sevents' > Apr 05 20:33:51 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.533+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'localpool' > Apr 05 20:33:51 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.634+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'mds_autoscaler' > Apr 05 20:33:51 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.894+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'mirroring' > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.019+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'nfs' > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 -1 mgr[py] Module nfs has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'orchestrator' > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 -1 mgr[py] Module orchestrator has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'osd_support' > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 -1 mgr[py] Module osd_support has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'pg_autoscaler' > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 -1 mgr[py] Module pg_autoscaler has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'progress' > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 -1 mgr[py] Module progress has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'prometheus' > Apr 05 20:33:53 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 -1 mgr[py] Module prometheus has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:53 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'rbd_support' > Apr 05 20:33:53 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 -1 mgr[py] Module rbd_support has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:53 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'restful' > Apr 05 20:33:53 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.799+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'rook' > Apr 05 20:33:54 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 -1 mgr[py] Module rook has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:54 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'selftest' > Apr 05 20:33:54 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 -1 mgr[py] Module selftest has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:54 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'snap_schedule' > Apr 05 20:33:54 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 -1 mgr[py] Module snap_schedule has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:54 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'stats' > Apr 05 20:33:54 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.660+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'status' > Apr 05 20:33:54 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 -1 mgr[py] Module status has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:54 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'telegraf' > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 -1 mgr[py] Module telegraf has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'telemetry' > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 -1 mgr[py] Module telemetry has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'test_orchestrator' > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 -1 mgr[py] Module test_orchestrator has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'volumes' > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 -1 mgr[py] Module volumes has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 1 mgr[py] Loading

...

...

module

...

> 'zabbix' > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.792+0000 7f5a66492680 -1 mgr[py] Module zabbix has missing

...

NOTIFY_TYPES

...

> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.793+0000 7f5a52600700 0 ms_deliver_dispatch: unhandled message 0x556eb85f26e0 mon_map magic: 0 v1 from

...

mon.2

...

> v2:192.168.47.72:3300/0 > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [dashboard DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from

...

module:

...

> dashboard > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [prometheus DEBUG root] setting log level based on debug_mgr:

...

INFO

...

> (2/5) > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from

...

module:

...

> prometheus > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f58f7800700 0 [dashboard INFO root] server: ssl=yes

...

host=::

...

> port=8443 > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] server_addr: ::

...

server_port:

...

> 9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root]

...

Starting

...

> engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE

...

Bus

...

> STARTING > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE

...

Bus

...

> STARTING > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.797+0000 7f5a52600700 0 client.0 ms_handle_reset

...

on

...

> v2:192.168.47.73:6842/917991990 > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.825+0000 7f58f7800700 0 [dashboard INFO root] Configured CherryPy,

...

starting

...

> engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.826+0000 7f58f7800700 0 [dashboard INFO root]

...

Starting

...

> engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:

...

CherryPy

...

> Checker: > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an

...

empty

...

> config. > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:

...

...

> Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Serving

...

on

...

> http://:::9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Serving

...

on

...

> http://:::9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE

...

Bus

...

> STARTED > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE

...

Bus

...

> STARTED > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO root]

...

Engine

...

> started. > Apr 05 20:33:55 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.939+0000 7f58f7800700 0 [dashboard INFO root]

...

Engine

...

> started... > Apr 05 20:33:58 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:33:58 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:58.569+0000 7f59b8000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:00 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:00 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:00.702+0000 7f59af400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:03 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:03 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:03.286+0000 7f59a6800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:08 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:08 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:08.569+0000 7f599dc00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:10 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:10 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:10.702+0000 7f5995000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:13 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:13 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:13.286+0000 7f598c400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:18 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:18 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:18.569+0000 7f5983800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:20 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200

...

...

""

...

> "Prometheus/2.33.4" > Apr 05 20:34:20 cn02.ceph.xyz.corp >

ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:20.702+0000 7f597ac00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200

'daemon_types' that already. - - - - try python python python python python python python python python python python python python python python python python python python python python python python python python python python python python python python python - - - - - - - - - - - - - - - -

...

...

""

...

> "Prometheus/2.33.4" > > cn03 > —— > Apr 05 20:33:42 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:42] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:42 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:42.753+0000 7f2724a00700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:42] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:43 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:43] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:43.772+0000 7f271be00700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:47 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:47] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:47 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:47.037+0000 7f2713200700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:33:47] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.867+0000 7f27d1a00700 1 mgr handle_mgr_map > Activating! > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.868+0000 7f27d1a00700 1 mgr handle_mgr_map > I am now activating > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > root] Stopping engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > root] Stopped engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > root] Stopping engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > root] Stopped engine > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.879+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.890+0000 7f2675a00700 0 [dashboard INFO > root] Engine stopped. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE HTTP Server > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.891+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE HTTP Server > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > root] Engine stopped. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts INFO root] > Init > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 1 mgr load > Constructed class from module: alerts > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f2663400700 0 [balancer DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f2663400700 1 mgr load > Constructed class from module: balancer > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f26f8e00700 0 [alerts INFO root] > Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.901+0000 7f2663400700 0 [cephadm DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.901+0000 7f26f0200700 0 [balancer INFO > root] Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > root] Optimize plan auto_2025-04-05_20:33:48 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > root] Mode upmap, max misplaced 0.050000 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO > root] Some PGs (1.000000) are unknown; try again later > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed > to construct class in 'cephadm' > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load > Traceback (most recent call last): > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.upgrade = CephadmUpgrade(self) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.upgrade_state: Optional[UpgradeState] = > UpgradeState.from_json(json.loads(t)) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > return cls(**c) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > TypeError: __init__() got an unexpected keyword argument > 'daemon_types' > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() > Failed to run module in active mode ('cephadm') > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [crash DEBUG root] > setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > Constructed class from module: crash > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [dashboard DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > Constructed class from module: dashboard > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.920+0000 7f2660c00700 0 [dashboard INFO > access_control] Loading user roles DB version=2 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > sso] Loading SSO DB version=1 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > root] server: ssl=yes host=:: port=8443 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [devicehealth > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > Constructed class from module: devicehealth > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [insights DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > Constructed class from module: insights > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.923+0000 7f2660200700 0 [devicehealth INFO > root] Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f2663400700 0 [iostat DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f2663400700 1 mgr load > Constructed class from module: iostat > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f265f800700 0 [insights INFO > root] Reset curr health slot key health_history/2025-04-05_20 next > flush None checks check count 0 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.926+0000 7f2663400700 0 [mirroring DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.931+0000 7f2663400700 1 mgr load > Constructed class from module: mirroring > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [nfs DEBUG root] > setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > Constructed class from module: nfs > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [orchestrator > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > Constructed class from module: orchestrator > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.933+0000 7f2663400700 0 [pg_autoscaler > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.933+0000 7f2663400700 1 mgr load > Constructed class from module: pg_autoscaler > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2663400700 0 [progress DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2663400700 1 mgr load > Constructed class from module: progress > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2657600700 0 [pg_autoscaler > INFO root] _maybe_adjust > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.940+0000 7f2656c00700 0 [progress INFO > root] Loading... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.940+0000 7f2663400700 0 [prometheus DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2663400700 1 mgr load > Constructed class from module: prometheus > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > root] server_addr: :: server_port: 9283 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > root] Cache enabled > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > root] Loaded [<progress.module.GhostEvent object at 0x7f27363ac400>, > <progress.module.GhostEvent object at 0x7f27363ac438>, > <progress.module.GhostEvent object at 0x7f27363ac3c8>, > <progress.module.GhostEvent object at 0x7f27363ac5c0>, > <progress.module.GhostEvent object at 0x7f27363ac4e0>, > <progress.module.GhostEvent object at 0x7f27363ac4a8>, > <progress.module.GhostEvent object at 0x7f278d317e10>, > <progress.module.GhostEvent object at 0x7f278d4397b8>, > <progress.module.GhostEvent object at 0x7f278d439908>, > <progress.module.GhostEvent object at 0x7f27363ed2e8>, > <progress.module.GhostEvent object at 0x7f27363ed4e0>, > <progress.module.GhostEvent object at 0x7f27363ed358>, > <progress.module.GhostEvent object at 0x7f27363ed278>, > <progress.module.GhostEvent object at 0x7f27363edeb8>, > <progress.module.GhostEvent object at 0x7f27363ed438>, > <progress.module.GhostEvent object at 0x7f27363ed400>, > <progress.module.GhostEvent object at 0x7f27363edb00>, > <progress.module.GhostEvent object at 0x7f27363ed780>, > <progress.module.GhostEvent object at 0x7f27363ed240>, > <progress.module.GhostEvent object at 0x7f27363edba8>, > <progress.module.GhostEvent object at 0x7f27363ed2b0>, > <progress.module.GhostEvent object at 0x7f27363ed0b8>, > <progress.module.GhostEvent object at 0x7f27363edcf8>, > <progress.module.GhostEvent object at 0x7f27363ed128>, > <progress.module.GhostEvent object at 0x7f27363ed080>, > <progress.module.GhostEvent object at 0x7f27363edc18>, > <progress.module.GhostEvent object at 0x7f27363ed860>, > <progress.module.GhostEvent object at 0x7f27363eda58>, > <progress.module.GhostEvent object at 0x7f27363eda90>, > <progress.module.GhostEvent object at 0x7f27363ed160>, > <progress.module.GhostEvent object at 0x7f27363ed908>, > <progress.module.GhostEvent object at 0x7f27363ed940>, > <progress.module.GhostEvent object at 0x7f27363ed8d0>, > <progress.module.GhostEvent object at 0x7f27363ed978>, > <progress.module.GhostEvent object at 0x7f27363ed748>, > <progress.module.GhostEvent object at 0x7f27363ed898>, > <progress.module.GhostEvent object at 0x7f27363eda20>, > <progress.module.GhostEvent object at 0x7f27363ed9b0>, > <progress.module.GhostEvent object at 0x7f27363ed9e8>, > <progress.module.GhostEvent object at 0x7f27363ed828>, > <progress.module.GhostEvent object at 0x7f27363ed7f0>, > <progress.module.GhostEvent object at 0x7f27363ed7b8>, > <progress.module.GhostEvent object at 0x7f27363ed668>, > <progress.module.GhostEvent object at 0x7f27363ed320>, > <progress.module.GhostEvent object at 0x7f27363ede48>, > <progress.module.GhostEvent object at 0x7f27363edc50>, > <progress.module.GhostEvent object at 0x7f27363ede80>, > <progress.module.GhostEvent object at 0x7f27363edf60>, > <progress.module.GhostEvent object at 0x7f27363edd68>, > <progress.module.GhostEvent object at 0x7f27363edda0>] historic events > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > root] Loaded OSDMap, ready. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.946+0000 7f2663400700 0 [rbd_support DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.946+0000 7f2655800700 0 [prometheus INFO > root] starting metric collection thread > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.947+0000 7f2656200700 0 [prometheus INFO > root] Starting engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STARTING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.948+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STARTING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.948+0000 7f2660c00700 0 [dashboard INFO > root] Configured CherryPy, starting engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.953+0000 7f2663400700 0 [rbd_support INFO > root] MirrorSnapshotScheduleHandler: load_schedules > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.957+0000 7f2663400700 0 [rbd_support INFO > root] load_schedules: csxyz, start_after= > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.989+0000 7f2649a00700 0 [rbd_support INFO > root] MirrorSnapshotScheduleHandler: starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.990+0000 7f2649000700 0 [rbd_support INFO > root] PerfHandler: starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.991+0000 7f2663400700 0 [rbd_support INFO > root] load_task_task: csxyz, start_after= > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.019+0000 7f2648600700 0 [rbd_support INFO > root] TaskHandler: starting > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.020+0000 7f2663400700 0 [rbd_support INFO > root] TrashPurgeScheduleHandler: load_schedules > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.022+0000 7f2663400700 0 [rbd_support INFO > root] load_schedules: csxyz, start_after= > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.024+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CephFS -> /api/cephfs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.026+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CephFsUi -> /ui-api/cephfs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > IscsiUi -> /ui-api/iscsi > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Iscsi -> /api/iscsi > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > IscsiTarget -> /api/iscsi/target > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGanesha -> /api/nfs-ganesha > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaCluster -> /api/nfs-ganesha/cluster > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaExports -> /api/nfs-ganesha/export > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaUi -> /ui-api/nfs-ganesha > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Rbd > -> /api/block/image > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.034+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdSnapshot -> /api/block/image/{image_spec}/snap > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdTrash -> /api/block/image/trash > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdNamespace -> /api/block/pool/{pool_name}/namespace > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroring -> /api/block/mirroring > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringSummary -> /api/block/mirroring/summary > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolMode -> /api/block/mirroring/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolBootstrap -> > /api/block/mirroring/pool/{pool_name}/bootstrap > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolPeer -> /api/block/mirroring/pool/{pool_name}/peer > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.038+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Rgw > -> /api/rgw > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwDaemon -> /api/rgw/daemon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwSite -> /api/rgw/site > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.040+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwBucket -> /api/rgw/bucket > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.041+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwUser -> /api/rgw/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Auth -> /api/auth > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Cluster -> /api/cluster > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.045+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ClusterConfiguration -> /api/cluster_conf > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.046+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CrushRule -> /api/crush_rule > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.047+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CrushRuleUi -> /ui-api/crush_rule > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Orchestrator -> /api/orchestrator > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Daemon -> /api/daemon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Docs -> /docs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ErasureCodeProfile -> /api/erasure_code_profile > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ErasureCodeProfileUi -> /ui-api/erasure_code_profile > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackController -> /api/feedback > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.051+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackApiController -> /api/feedback/api_key > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackUiController -> /ui-api/feedback/api_key > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FrontendLogging -> /ui-api/logging > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Grafana -> /api/grafana > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Host -> /api/host > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > HostUi -> /ui-api/host > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Health -> /api/health > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > HomeController -> / > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:49] ENGINE Serving on http://:::9283 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Serving on > http://:::9283 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:49] ENGINE Bus STARTED > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Bus STARTED > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > root] Engine started. > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > LangsController -> /ui-api/langs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Logs -> /api/logs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MgrModules -> /api/mgr/module > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Monitor -> /api/monitor > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Osd > -> /api/osd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.060+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > OsdFlagsController -> /api/osd/flags > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MdsPerfCounter -> /api/perf_counters/mds > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MonPerfCounter -> /api/perf_counters/mon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > OsdPerfCounter -> /api/perf_counters/osd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwPerfCounter -> /api/perf_counters/rgw > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirrorPerfCounter -> /api/perf_counters/rbd-mirror > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MgrPerfCounter -> /api/perf_counters/mgr > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > TcmuRunnerPerfCounter -> /api/perf_counters/tcmu-runner > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PerfCounters -> /api/perf_counters > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.065+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Pool -> /api/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.066+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PoolUi -> /ui-api/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PrometheusReceiver -> /api/prometheus_receiver > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Prometheus -> /api/prometheus > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.068+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PrometheusNotifications -> /api/prometheus/notifications > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.069+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Role -> /api/role > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Scope -> /ui-api/scope > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Saml2 -> /auth/saml2 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Service -> /api/service > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.071+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Settings -> /api/settings > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > StandardSettings -> /ui-api/standard_settings > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Summary -> /api/summary > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Task -> /api/task > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Telemetry -> /api/telemetry > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > User -> /api/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > UserPasswordPolicy -> /api/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > UserChangePassword -> /api/user/{username} > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeatureTogglesEndpoint -> /api/feature_toggles > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MessageOfTheDay -> /ui-api/motd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.090+0000 7f2647200700 0 [rbd_support INFO > root] TrashPurgeScheduleHandler: starting > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.096+0000 7f2663400700 1 mgr load > Constructed class from module: rbd_support > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.102+0000 7f2663400700 0 [restful DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.102+0000 7f2663400700 1 mgr load > Constructed class from module: restful > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.108+0000 7f2645e00700 0 [restful INFO > root] server_addr: :: server_port: 8003 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.109+0000 7f2663400700 0 [snap_schedule > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.116+0000 7f2645e00700 0 [restful WARNING > root] server not running: no certificate configured > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.257+0000 7f2660c00700 0 [dashboard INFO > dashboard.module] Engine started. > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed > to construct class in 'snap_schedule' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load > Traceback (most recent call last): > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in > __init__ > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.client = SnapSchedClient(self) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > 158, in __init__ > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > with self.get_schedule_db(fs_name) as conn_mgr: > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > 192, in get_schedule_db > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > db.executescript(dump) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > sqlite3.OperationalError: table schedules already exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() > Failed to run module in active mode ('snap_schedule') > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 0 [status DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 1 mgr load > Constructed class from module: status > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.275+0000 7f2663400700 0 [telemetry DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.276+0000 7f2663400700 1 mgr load > Constructed class from module: telemetry > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.276+0000 7f2663400700 0 [volumes DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.284+0000 7f2660200700 0 [devicehealth INFO > root] Check health > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 1 mgr load > Constructed class from module: volumes > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.co [Text gekürzt]