ceph config-key get mgr/cephadm/upgrade_state {"target_name": "quay.io/ceph/ceph:v17.2.0", "progress_id": "e7e1a809-558d-43a7-842a-c6229fdc57af", "target_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "target_digests": ["quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667"], "target_version": "17.2.0", "fs_original_max_mds": null, "fs_original_allow_standby_replay": null, "error": null, "paused": false, "daemon_types": null, "hosts": null, "services": null, "total_count": null, "remaining_count": null} What should I do next? Thank you! -jeremy
On Sunday, Apr 06, 2025 at 1:38 AM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: Can you check if you have this config-key?
ceph config-key get mgr/cephadm/upgrade_state
If you reset the MGRs, it might be necessary to clear this key, otherwise you might end up in some inconsistency. Just to be sure.
Zitat von Jeremy Hansen <jeremy@skidrow.la>:
Thanks. I’m trying to be extra careful since this cluster is actually in use. I’ll wait for your feedback.
-jeremy
On Saturday, Apr 05, 2025 at 3:39 PM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: No, that's not necessary, just edit the unit.run file for the MGRs to use a different image. See Frédéric's instructions:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/32APKOXKRAI...
But I'm not entirely sure if you need to clear some config-keys first in order to reset the upgrade state. If I have time, I'll try to check tomorrow, or on Monday.
Zitat von Jeremy Hansen <jeremy@skidrow.la>:
Would I follow this process to downgrade?
https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-...
Thank you
On Saturday, Apr 05, 2025 at 2:04 PM, Jeremy Hansen <jeremy@skidrow.la (mailto:jeremy@skidrow.la)> wrote: ceph -s claims things are healthy:
ceph -s cluster: id: 95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1 health: HEALTH_OK
services: mon: 3 daemons, quorum cn01,cn03,cn02 (age 20h) mgr: cn03.negzvb(active, since 26m), standbys: cn01.tjmtph, cn02.ceph.xyz.corp.ggixgj mds: 1/1 daemons up, 2 standby osd: 15 osds: 15 up (since 19h), 15 in (since 14M)
data: volumes: 1/1 healthy pools: 6 pools, 610 pgs objects: 284.59k objects, 1.1 TiB usage: 3.3 TiB used, 106 TiB / 109 TiB avail pgs: 610 active+clean
io: client: 255 B/s rd, 1.2 MiB/s wr, 10 op/s rd, 16 op/s wr
— How do I downgrade if the orch is down?
Thank you -jeremy
On Saturday, Apr 05, 2025 at 1:56 PM, Eugen Block <eblock@nde.ag (mailto:eblock@nde.ag)> wrote: It would help if you only pasted the relevant parts. Anyway, these two sections stand out:
---snip--- Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again later Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed to construct class in 'cephadm' Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade = CephadmUpgrade(self) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: return cls(**c) Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: TypeError: __init__() got an unexpected keyword argument
Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:48 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('cephadm')
Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed to construct class in 'snap_schedule' Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Traceback (most recent call last): Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: self.client = SnapSchedClient(self) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 158, in __init__ Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: with self.get_schedule_db(fs_name) as conn_mgr: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line 192, in get_schedule_db Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: db.executescript(dump) Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: sqlite3.OperationalError: table schedules already exists Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: Apr 05 20:33:49 cn03.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() Failed to run module in active mode ('snap_schedule') ---snip---
Your cluster seems to be in an error state (ceph -s) because of an unknown PG. It's recommended to have a healthy cluster before attemping an upgrade. It's possible that these errors come from the not upgraded MGR, I'm not sure.
Since the upgrade was only successful for two MGRs, I am thinking about downgrading both MGRs back to 16.2.15, then retry an upgrade to a newer version, either 17.2.8 or 18.2.4. I haven't checked the snap_schedule error yet, though. Maybe someone else knows
Zitat von Jeremy Hansen <jeremy@skidrow.la>:
> This is a small three node cluster. Here’s logs from all three nodes > during the failover. The current manager was cn02 and it failed over > to cn03. > > cn01 > —— > Apr 05 20:33:35 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:35.823+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v821: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.170+0000 7f2aaec00700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.215+0000 7f2aab000700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:37 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:37.823+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v822: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 22 KiB/s wr, 5 op/s > Apr 05 20:33:39 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:39] "GET /metrics > HTTP/1.1" 200 267241 "" "Prometheus/2.33.4" > Apr 05 20:33:39 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:39.825+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v823: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 97 KiB/s wr, 12 op/s > Apr 05 20:33:41 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:41.826+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v824: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 255 B/s rd, 85 KiB/s wr, 10 op/s > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.170+0000 7f2aaec00700 0 [insights INFO > root] Received health check update 0 pending > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO > root] Applying 1 health updates to slot key > health_history/2025-04-05_20 next flush None checks check count 0 > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:42.191+0000 7f2aab000700 0 [insights INFO > root] Maybe flushing slot key health_history/2025-04-05_20 next > flush None checks check count 0 needed False > Apr 05 20:33:42 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:42] "GET /metrics > HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics > HTTP/1.1" 200 267249 "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:43.827+0000 7f2aace00700 0 > log_channel(cluster) log [DBG] : pgmap v825: 610 pgs: 610 > active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB avail; > 511 B/s rd, 133 KiB/s wr, 18 op/s > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 -1 mgr > handle_mgr_map I was active but no longer am > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn e: > '/usr/bin/ceph-mgr' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 0: > '/usr/bin/ceph-mgr' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr
respawn 1: '-n'
> Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 2: > 'mgr.cn01.tjmtph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 3: '-f' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 4: > '--setuser' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 5: > 'ceph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 6: > '--setgroup' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 7: > 'ceph' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 8: > '--default-log-to-file=false' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 9: > '--default-log-to-stderr=true' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn 10: > '--default-log-stderr-prefix=debug ' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn > respawning with exe /usr/bin/ceph-mgr > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.364+0000 7f2c1d200700 1 mgr respawn > exe_path /proc/self/exe > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ignoring --setuser ceph since I am not root > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ignoring --setgroup ceph since I am not root > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 ceph version > 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), > process ceph-mgr, pid 2 > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.449+0000 7fb814447000 0 pidfile_write: > ignore empty --pid-file > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.477+0000 7fb814447000 1 mgr[py] Loading > python module 'alerts' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.612+0000 7fb814447000 -1 mgr[py] Module > alerts has missing NOTIFY_TYPES member > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.612+0000 7fb814447000 1 mgr[py] Loading > python module 'balancer' > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.726+0000 7fb814447000 -1 mgr[py] Module > balancer has missing NOTIFY_TYPES member > Apr 05 20:33:45 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:45.726+0000 7fb814447000 1 mgr[py] Loading > python module 'cephadm' > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.417+0000 7fb814447000 1 mgr[py] Loading > python module 'crash' > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.540+0000 7fb814447000 -1 mgr[py] Module > crash has missing NOTIFY_TYPES member > Apr 05 20:33:46 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:46.540+0000 7fb814447000 1 mgr[py] Loading > python module 'dashboard' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.210+0000 7fb814447000 1 mgr[py] Loading > python module 'devicehealth' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.331+0000 7fb814447000 -1 mgr[py] Module > devicehealth has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.331+0000 7fb814447000 1 mgr[py] Loading > python module 'diskprediction_local' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.653+0000 7fb814447000 -1 mgr[py] Module > diskprediction_local has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.654+0000 7fb814447000 1 mgr[py] Loading > python module 'influx' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.769+0000 7fb814447000 -1 mgr[py] Module > influx has missing NOTIFY_TYPES member > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.769+0000 7fb814447000 1 mgr[py] Loading > python module 'insights' > Apr 05 20:33:47 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:47.884+0000 7fb814447000 1 mgr[py] Loading > python module 'iostat' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.082+0000 7fb814447000 -1 mgr[py] Module > iostat has missing NOTIFY_TYPES member > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.082+0000 7fb814447000 1 mgr[py] Loading > python module 'k8sevents' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.679+0000 7fb814447000 1 mgr[py] Loading > python module 'localpool' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.796+0000 7fb814447000 1 mgr[py] Loading > python module 'mds_autoscaler' > Apr 05 20:33:48 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:48.983+0000 7fb814447000 1 mgr[py] Loading > python module 'mirroring' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.118+0000 7fb814447000 1 mgr[py] Loading > python module 'nfs' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.315+0000 7fb814447000 -1 mgr[py] Module > nfs has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.315+0000 7fb814447000 1 mgr[py] Loading > python module 'orchestrator' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.607+0000 7fb814447000 -1 mgr[py] Module > orchestrator has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.607+0000 7fb814447000 1 mgr[py] Loading > python module 'osd_perf_query' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.723+0000 7fb814447000 -1 mgr[py] Module > osd_perf_query has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.723+0000 7fb814447000 1 mgr[py] Loading > python module 'osd_support' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.837+0000 7fb814447000 -1 mgr[py] Module > osd_support has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.837+0000 7fb814447000 1 mgr[py] Loading > python module 'pg_autoscaler' > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.970+0000 7fb814447000 -1 mgr[py] Module > pg_autoscaler has missing NOTIFY_TYPES member > Apr 05 20:33:49 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:49.970+0000 7fb814447000 1 mgr[py] Loading > python module 'progress' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.082+0000 7fb814447000 -1 mgr[py] Module > progress has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.082+0000 7fb814447000 1 mgr[py] Loading > python module 'prometheus' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.617+0000 7fb814447000 -1 mgr[py] Module > prometheus has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.617+0000 7fb814447000 1 mgr[py] Loading > python module 'rbd_support' > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.761+0000 7fb814447000 -1 mgr[py] Module > rbd_support has missing NOTIFY_TYPES member > Apr 05 20:33:50 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:50.761+0000 7fb814447000 1 mgr[py] Loading > python module 'restful' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.023+0000 7fb814447000 1 mgr[py] Loading > python module 'rook' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.788+0000 7fb814447000 -1 mgr[py] Module > rook has missing NOTIFY_TYPES member > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.788+0000 7fb814447000 1 mgr[py] Loading > python module 'selftest' > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.912+0000 7fb814447000 -1 mgr[py] Module > selftest has missing NOTIFY_TYPES member > Apr 05 20:33:51 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:51.912+0000 7fb814447000 1 mgr[py] Loading > python module 'snap_schedule' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.038+0000 7fb814447000 -1 mgr[py] Module > snap_schedule has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.038+0000 7fb814447000 1 mgr[py] Loading > python module 'stats' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.155+0000 7fb814447000 1 mgr[py] Loading > python module 'status' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.280+0000 7fb814447000 -1 mgr[py] Module > status has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.280+0000 7fb814447000 1 mgr[py] Loading > python module 'telegraf' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.398+0000 7fb814447000 -1 mgr[py] Module > telegraf has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.398+0000 7fb814447000 1 mgr[py] Loading > python module 'telemetry' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.594+0000 7fb814447000 -1 mgr[py] Module > telemetry has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.594+0000 7fb814447000 1 mgr[py] Loading > python module 'test_orchestrator' > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.771+0000 7fb814447000 -1 mgr[py] Module > test_orchestrator has missing NOTIFY_TYPES member > Apr 05 20:33:52 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:52.771+0000 7fb814447000 1 mgr[py] Loading > python module 'volumes' > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.225+0000 7fb814447000 -1 mgr[py] Module > volumes has missing NOTIFY_TYPES member > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.225+0000 7fb814447000 1 mgr[py] Loading > python module 'zabbix' > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.342+0000 7fb814447000 -1 mgr[py] Module > zabbix has missing NOTIFY_TYPES member > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.344+0000 7fb807c00700 0 > ms_deliver_dispatch: unhandled message 0x555c78fe1080 mon_map magic: > 0 v1 from mon.0 v2:192.168.47.71:3300/0 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.344+0000 7fb808600700 0 [dashboard DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load > Constructed class from module: dashboard > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 0 [prometheus DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb808600700 1 mgr load > Constructed class from module: prometheus > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.345+0000 7fb6abe00700 0 [dashboard INFO > root] server: ssl=yes host=:: port=8443 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > root] server_addr: :: server_port: 9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > root] Starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Bus STARTING > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.346+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTING > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO > root] Configured CherryPy, starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.377+0000 7fb6abe00700 0 [dashboard INFO > root] Starting engine... > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > CherryPy Checker: > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > The Application mounted at '' has an empty config. > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Serving on http://:::9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Serving on > http://:::9283 > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > [05/Apr/2025:20:33:53] ENGINE Bus STARTED > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.450+0000 7fb6b4a00700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:53] ENGINE Bus STARTED > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.451+0000 7fb6b4a00700 0 [prometheus INFO > root] Engine started. > Apr 05 20:33:53 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:53.489+0000 7fb6abe00700 0 [dashboard INFO > root] Engine started... > Apr 05 20:33:59 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:59] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:59 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:33:59.603+0000 7fb76c600700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:59] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:02 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:02] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:02 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:02.476+0000 7fb763a00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:02] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:03 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:03 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:03.061+0000 7fb75ae00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:09 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:09] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:09 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:09.602+0000 7fb752200700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:09] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:12 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:12] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:12 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:12.476+0000 7fb749600700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:34:12] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:13 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:13 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:13.060+0000 7fb740a00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.493+0000 7fb714e00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.494+0000 7fb714e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52368] [GET] [303] [0.003s] [96.0B] > [0031fbdd-947f-4c6c-943d-55cf256edfba] /api/summary > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.512+0000 7fb70c200700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.513+0000 7fb70c200700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52369] [GET] [303] [0.001s] [96.0B] > [76c9a6ab-aae0-4ef9-815e-dbd0ac8f8b49] /api/feature_toggles > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.527+0000 7fb703600700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.528+0000 7fb703600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52370] [GET] [303] [0.001s] [96.0B] > [db017da3-e6f6-45fc-9d99-7b9f3b3859f2] /api/health/minimal > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.541+0000 7fb6faa00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.542+0000 7fb6faa00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52371] [GET] [303] [0.001s] [96.0B] > [564ba322-b99c-49de-a3a3-e0ed422bfae7] /api/prometheus > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.616+0000 7fb6f1e00700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.617+0000 7fb6f1e00700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52372] [GET] [303] [0.001s] [96.0B] > [1f349e52-9a10-4fda-b13f-621785208ea9] /api/prometheus/rules > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO > root] Redirecting to active 'https://192.168.47.73:8443/' > Apr 05 20:34:15 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:15.630+0000 7fb6e0600700 0 [dashboard INFO > request] [::ffff:192.168.10.101:52368] [GET] [303] [0.001s] [96.0B] > [30d749d7-c972-44f0-88c7-b70078e2385c] /api/prometheus/notifications > Apr 05 20:34:19 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:19] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:34:19 cn01.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn01-tjmtph[771594]: > debug 2025-04-05T20:34:19.601+0000 7fb737e00700 0 [prometheus INFO > cherrypy.access.140425911062880] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:34:19] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > > cn02 > —— > Apr 05 20:33:40 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:33:40 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:40.701+0000 7ff710400700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:40] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:33:43 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:33:43 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:43.287+0000 7ff707800700 0 [prometheus INFO cherrypy.access.140700134177256] ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.370+0000 7ff7e7c00700 1 mgr
handle_mgr_map
> Activating! > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.371+0000 7ff7e7c00700 1 mgr handle_mgr_map I am
now
> activating > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root]
Stopping
> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [dashboard INFO root]
Stopped
> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root]
Stopping
> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.382+0000 7ff67aa00700 0 [prometheus INFO root]
Stopped
> engine > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE
Bus
> STOPPING > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.383+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE
Bus
> STOPPING > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.394+0000 7ff68d000700 0 [dashboard INFO root]
Engine
> stopped. > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283))
shut
> down > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.395+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283))
shut
> down > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE
Bus
> STOPPED > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE
Bus
> STOPPED > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.396+0000 7ff695c00700 0 [prometheus INFO root]
Engine
> stopped. > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 0 [alerts INFO
root]
> Init > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.398+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> alerts > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 0 [balancer DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> balancer > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.399+0000 7ff74d800700 0 [alerts INFO
root]
> Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.401+0000 7ff67aa00700 0 [cephadm DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.402+0000 7ff710400700 0 [balancer INFO
root]
> Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Optimize
plan
> auto_2025-04-05_20:33:45 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.408+0000 7ff710400700 0 [balancer INFO root] Mode upmap, max
misplaced
> 0.050000 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.409+0000 7ff710400700 0 [balancer INFO root] Some PGs (1.000000) are unknown;
again
> later > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.551+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> cephadm > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 0 [crash DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.552+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> crash > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 0 [dashboard DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.554+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> dashboard > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.555+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> devicehealth > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO access_control] Loading user roles
DB
> version=2 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO sso] Loading SSO
DB
> version=1 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.556+0000 7ff676400700 0 [dashboard INFO root] server: ssl=yes
host=::
> port=8443 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.557+0000 7ff674600700 0 [devicehealth INFO
root]
> Starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.558+0000 7ff67aa00700 0 [insights DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.559+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> insights > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.561+0000 7ff674600700 0 [devicehealth INFO root]
Check
> health > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.566+0000 7ff671e00700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:33:45 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 0 [iostat DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.567+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> iostat > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.572+0000 7ff67aa00700 0 [mirroring DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.579+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 0 [nfs DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.604+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> nfs > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.622+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> orchestrator > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.623+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> pg_autoscaler > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.625+0000 7ff662a00700 0 [pg_autoscaler INFO
root]
> _maybe_adjust > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 0 [progress DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.631+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> progress > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.72' (ECDSA) to the list of
known
> hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.73' (ECDSA) to the list of
known
> hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 0 [prometheus DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.637+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> prometheus > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff661600700 0 [progress INFO
root]
> Loading... > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root] server_addr: ::
server_port:
> 9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff660200700 0 [prometheus INFO root]
Cache
> enabled > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.638+0000 7ff65ee00700 0 [prometheus INFO root] starting metric
collection
> thread > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.640+0000 7ff67aa00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7ff7443993c8>, <progress.module.GhostEvent object at 0x7ff744399400>, <progress.module.GhostEvent object at 0x7ff744399438>, <progress.module.GhostEvent object at 0x7ff744399470>, <progress.module.GhostEvent object at 0x7ff7443994a8>, <progress.module.GhostEvent object at 0x7ff7443994e0>, <progress.module.GhostEvent object at 0x7ff744399518>, <progress.module.GhostEvent object at 0x7ff744399550>, <progress.module.GhostEvent object at 0x7ff744399588>, <progress.module.GhostEvent object at 0x7ff7443995c0>, <progress.module.GhostEvent object at 0x7ff7443995f8>, <progress.module.GhostEvent object at 0x7ff744399630>, <progress.module.GhostEvent object at 0x7ff744399668>, <progress.module.GhostEvent object at 0x7ff7443996a0>, <progress.module.GhostEvent object at 0x7ff7443996d8>, <progress.module.GhostEvent object at 0x7ff744399710>, <progress.module.GhostEvent object at 0x7ff744399748>, <progress.module.GhostEvent object at 0x7ff744399780>, <progress.module.GhostEvent object at 0x7ff7443997b8>, <progress.module.GhostEvent object at 0x7ff7443997f0>, <progress.module.GhostEvent object at 0x7ff744399828>, <progress.module.GhostEvent object at 0x7ff744399860>, <progress.module.GhostEvent object at 0x7ff744399898>, <progress.module.GhostEvent object at 0x7ff7443998d0>, <progress.module.GhostEvent object at 0x7ff744399908>, <progress.module.GhostEvent object at 0x7ff744399940>, <progress.module.GhostEvent object at 0x7ff744399978>, <progress.module.GhostEvent object at 0x7ff7443999b0>, <progress.module.GhostEvent object at 0x7ff7443999e8>, <progress.module.GhostEvent object at 0x7ff744399a20>, <progress.module.GhostEvent object at 0x7ff744399a58>, <progress.module.GhostEvent object at 0x7ff744399a90>, <progress.module.GhostEvent object at 0x7ff744399ac8>, <progress.module.GhostEvent object at 0x7ff744399b00>, <progress.module.GhostEvent object at 0x7ff744399b38>, <progress.module.GhostEvent object at 0x7ff744399b70>, <progress.module.GhostEvent object at 0x7ff744399ba8>, <progress.module.GhostEvent object at 0x7ff744399be0>, <progress.module.GhostEvent object at 0x7ff744399c18>, <progress.module.GhostEvent object at 0x7ff744399c50>, <progress.module.GhostEvent object at 0x7ff744399c88>, <progress.module.GhostEvent object at 0x7ff744399cc0>, <progress.module.GhostEvent object at 0x7ff744399cf8>, <progress.module.GhostEvent object at 0x7ff744399d30>, <progress.module.GhostEvent object at 0x7ff744399d68>, <progress.module.GhostEvent object at 0x7ff744399da0>, <progress.module.GhostEvent object at 0x7ff744399dd8>, <progress.module.GhostEvent object at 0x7ff744399e10>, <progress.module.GhostEvent object at 0x7ff744399e48>, <progress.module.GhostEvent object at 0x7ff744399e80>]
historic
> events > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Loaded
OSDMap,
> ready. > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.642+0000 7ff661600700 0 [progress INFO root] Processing OSDMap
change
> 27185..27185 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff676400700 0 [dashboard INFO root] Configured CherryPy,
starting
> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.646+0000 7ff660200700 0 [prometheus INFO root]
Starting
> engine... > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE
Bus
> STARTING > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.647+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE
Bus
> STARTING > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root] recovery
thread
> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff654e00700 0 [rbd_support INFO root]
starting
> setup > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.656+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> rbd_support > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.71' (ECDSA) to the list of
known
> hosts. > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 0 [restful DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.659+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> restful > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.660+0000 7ff651c00700 0 [restful INFO root] server_addr: ::
server_port:
> 8003 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.661+0000 7ff654e00700 0 [rbd_support INFO root]
MirrorSnapshotScheduleHandler:
> load_schedules > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.697+0000 7ff651c00700 0 [restful WARNING root] server not running: no
certificate
> configured > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.705+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules:
csxyz,
> start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.706+0000 7ff67aa00700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.707+0000 7ff64f400700 0 [rbd_support INFO root]
MirrorSnapshotScheduleHandler:
> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.708+0000 7ff64ea00700 0 [rbd_support INFO root]
PerfHandler:
> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.709+0000 7ff654e00700 0 [rbd_support INFO root] load_task_task:
csxyz,
> start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.718+0000 7ff649a00700 0 [rbd_support INFO root]
TaskHandler:
> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.719+0000 7ff654e00700 0 [rbd_support INFO root]
TrashPurgeScheduleHandler:
> load_schedules > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.721+0000 7ff654e00700 0 [rbd_support INFO root] load_schedules:
csxyz,
> start_after= > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff649000700 0 [rbd_support INFO root]
TrashPurgeScheduleHandler:
> starting > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.723+0000 7ff654e00700 0 [rbd_support INFO root]
setup
> complete > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.726+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> snap_schedule > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 0 [status DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.727+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> status > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.728+0000 7ff67aa00700 0 [telemetry DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> telemetry > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.729+0000 7ff67aa00700 0 [volumes DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for
volume
> 'cssnfsxyz' > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 0 [volumes INFO volumes.fs.async_job] queuing job for
volume
> 'cssnfsxyz' > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.742+0000 7ff67aa00700 1 mgr load Constructed class from
module:
> volumes > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63aa00700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.747+0000 7ff63c800700 -1 client.0 error registering admin socket command: (17)
File
> exists > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:
CherryPy
> Checker: > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an
empty
> config. > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:
> Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE Serving
on
> http://:::9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE Serving
on
> http://:::9283 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:45] ENGINE
Bus
> STARTED > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:45] ENGINE
Bus
> STARTED > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.754+0000 7ff660200700 0 [prometheus INFO root]
Engine
> started. > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.785+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS
->
> /api/cephfs > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.787+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi
->
> /ui-api/cephfs > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi
->
> /ui-api/iscsi > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi
->
> /api/iscsi > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.789+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget
->
> /api/iscsi/target > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster
->
> /api/nfs-ganesha/cluster > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.791+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports
->
> /api/nfs-ganesha/export > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi
->
> /ui-api/nfs-ganesha > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.792+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd
->
> /api/block/image > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdStatus
->
> /ui-api/block/rbd > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.795+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot
->
> /api/block/image/{image_spec}/snap > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.796+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash
->
> /api/block/image/trash > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace
->
> /api/block/pool/{pool_name}/namespace > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.797+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool
->
> /api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.799+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi
->
> /ui-api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.800+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RBDPool
->
> /api/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator
->
> /ui-api/orchestrator > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.801+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service
->
> /api/service > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring
->
> /api/block/mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.802+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary
->
> /api/block/mirroring/summary > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode
->
> /api/block/mirroring/pool > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap
->
> /api/block/mirroring/pool/{pool_name}/bootstrap > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.803+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer
->
> /api/block/mirroring/pool/{pool_name}/peer > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.804+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringStatus
->
> /ui-api/block/mirroring > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw
->
> /ui-api/rgw > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.805+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon
->
> /api/rgw/daemon > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite
->
> /api/rgw/site > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.806+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket
->
> /api/rgw/bucket > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.807+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser
->
> /api/rgw/user > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.810+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth
->
> /api/auth > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster
->
> /api/cluster > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.811+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration
->
> /api/cluster_conf > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.813+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule
->
> /api/crush_rule > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi
->
> /ui-api/crush_rule > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.814+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon
->
> /api/daemon > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs
->
> /docs > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.815+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile
->
> /api/erasure_code_profile > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.816+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi
->
> /ui-api/erasure_code_profile > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging
->
> /ui-api/logging > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana
->
> /api/grafana > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.817+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host
->
> /api/host > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.819+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi
->
> /ui-api/host > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health
->
> /api/health > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.820+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController
->
> / > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.822+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController
->
> /ui-api/langs > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LoginController
->
> /ui-api/login > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs
->
> /api/logs > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.824+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules
->
> /api/mgr/module > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor
->
> /api/monitor > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.826+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd
->
> /api/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.828+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdUi
->
> /ui-api/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.830+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController
->
> /api/osd/flags > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter
->
> /api/perf_counters/mds > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.831+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter
->
> /api/perf_counters/mon > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter
->
> /api/perf_counters/osd > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.832+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter
->
> /api/perf_counters/rgw > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter
->
> /api/perf_counters/rbd-mirror > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.833+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter
->
> /api/perf_counters/mgr > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter
->
> /api/perf_counters/tcmu-runner > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.834+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters
->
> /api/perf_counters > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver
->
> /api/prometheus_receiver > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.835+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus
->
> /api/prometheus > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications
->
> /api/prometheus/notifications > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.836+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role
->
> /api/role > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope
->
> /ui-api/scope > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2
->
> /auth/saml2 > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.838+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings
->
> /api/settings > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings
->
> /ui-api/standard_settings > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.840+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary
->
> /api/summary > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task
->
> /api/task > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.841+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry
->
> /api/telemetry > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.842+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User
->
> /api/user > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy
->
> /api/user > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword
->
> /api/user/{username} > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.843+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint
->
> /api/feature_toggles > Apr 05 20:33:45 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:45.844+0000 7ff676400700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay
->
> /ui-api/motd > Apr 05 20:33:46 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.050+0000 7ff676400700 0 [dashboard INFO dashboard.module]
Engine
> started. > Apr 05 20:33:46 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:46.378+0000 7ff67a000700 0 log_channel(cluster) log [DBG] : pgmap v3: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109
TiB
> avail > Apr 05 20:33:47 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:47.379+0000 7ff678c00700 0 log_channel(cluster) log [DBG] : pgmap v4: 610 pgs: 610 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109
TiB
> avail > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104
""
> "Prometheus/2.33.4" > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.568+0000 7ff65b200700 0 [prometheus INFO cherrypy.access.140699978276536] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:48] "GET /metrics HTTP/1.1" 200 42104
""
> "Prometheus/2.33.4" > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 [cephadm INFO cephadm.serve]
Updating
> cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff721c00700 0 log_channel(cephadm) log [INF] :
Updating
> cn01.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 [cephadm INFO cephadm.serve]
Updating
> cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve]
Updating
> cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff719000700 0 log_channel(cephadm) log [INF] :
Updating
> cn02.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.765+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] :
Updating
> cn03.ceph.xyz.corp:/etc/ceph/ceph.conf > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 [cephadm INFO cephadm.serve]
Updating
> cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff721c00700 0 log_channel(cephadm) log [INF] :
Updating
> cn01.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 [cephadm INFO cephadm.serve]
Updating
> cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff719000700 0 log_channel(cephadm) log [INF] :
Updating
> cn02.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 [cephadm INFO cephadm.serve]
Updating
> cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.768+0000 7ff6fec00700 0 log_channel(cephadm) log [INF] :
Updating
> cn03.ceph.xyz.corp:/etc/ceph/ceph.client.admin.keyring > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.812+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 824b8fde-5a37-4436-af45-a7d8255e2c66 does
not
> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 621cb660-d584-45e4-8033-4a476e804bba does
not
> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.813+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 007a4dd2-46d3-4655-9c93-c02f97e555cb does
not
> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 6b0fbff2-881e-4203-9149-f0977bc13eab does
not
> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.814+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 8a1a90b8-1f2a-48c8-924f-6e74032b6319 does
not
> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.815+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3c322d26-9edb-4d26-b4bc-97c330f5a1d8 does
not
> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev f3e5a517-d643-4fa5-b127-57716264e4e3 does
not
> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.825+0000 7ff6a7400700 0 [progress WARNING root] complete: ev 3fb020c7-92e8-4eb1-a884-2d04e5bca889 does
not
> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.840+0000 7ff6a7400700 0 [progress WARNING root] complete: ev bf9e0705-075e-4134-b77f-aa64fe92112a does
not
> exist > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade
myself
> (mgr.cn02.ceph.xyz.corp.ggixgj) > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade
myself
> (mgr.cn02.ceph.xyz.corp.ggixgj) > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.848+0000 7ff6a7400700 0 [progress INFO root] update: starting ev e7e1a809-558d-43a7-842a-c6229fdc57af (Upgrade
to
> 17.2.0) > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 [cephadm INFO cephadm.services.cephadmservice] Failing over to
other
> MGR > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.851+0000 7ff6a7400700 0 log_channel(cephadm) log [INF] : Failing over to
other
> MGR > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 -1 mgr handle_mgr_map I was active but no
longer
> am > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
e:
> '/usr/bin/ceph-mgr' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
0:
> '/usr/bin/ceph-mgr' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
1:
> '-n' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
2:
> 'mgr.cn02.ceph.xyz.corp.ggixgj' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
3:
> '-f' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
4:
> '--setuser' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
5:
> 'ceph' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
6:
> '--setgroup' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
7:
> 'ceph' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
8:
> '--default-log-to-file=false' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
9:
> '--default-log-to-stderr=true' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn 10:
'--default-log-stderr-prefix=debug
> ' > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn respawning with
exe
> /usr/bin/ceph-mgr > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.869+0000 7ff7e7c00700 1 mgr respawn
exe_path
> /proc/self/exe > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setuser ceph since I am
not
> root > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ignoring --setgroup ceph since I am
not
> root > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable), process ceph-mgr,
pid
> 2 > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.949+0000 7f5a66492680 0 pidfile_write: ignore
empty
> --pid-file > Apr 05 20:33:48 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:48.976+0000 7f5a66492680 1 mgr[py] Loading
module
> 'alerts' > Apr 05 20:33:49 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 -1 mgr[py] Module alerts has missing
NOTIFY_TYPES
> member > Apr 05 20:33:49 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.105+0000 7f5a66492680 1 mgr[py] Loading
module
> 'balancer' > Apr 05 20:33:49 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 -1 mgr[py] Module balancer has missing
NOTIFY_TYPES
> member > Apr 05 20:33:49 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.212+0000 7f5a66492680 1 mgr[py] Loading
module
> 'cephadm' > Apr 05 20:33:49 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.568+0000 7f5a66492680 1 mgr[py] Loading
module
> 'crash' > Apr 05 20:33:49 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 -1 mgr[py] Module crash has missing
NOTIFY_TYPES
> member > Apr 05 20:33:49 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:49.677+0000 7f5a66492680 1 mgr[py] Loading
module
> 'dashboard' > Apr 05 20:33:50 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.276+0000 7f5a66492680 1 mgr[py] Loading
module
> 'devicehealth' > Apr 05 20:33:50 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.415+0000 7f5a66492680 1 mgr[py] Loading
module
> 'diskprediction_local' > Apr 05 20:33:50 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 -1 mgr[py] Module diskprediction_local has missing
NOTIFY_TYPES
> member > Apr 05 20:33:50 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.640+0000 7f5a66492680 1 mgr[py] Loading
module
> 'influx' > Apr 05 20:33:50 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 -1 mgr[py] Module influx has missing
NOTIFY_TYPES
> member > Apr 05 20:33:50 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.750+0000 7f5a66492680 1 mgr[py] Loading
module
> 'insights' > Apr 05 20:33:50 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:50.919+0000 7f5a66492680 1 mgr[py] Loading
module
> 'iostat' > Apr 05 20:33:51 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 -1 mgr[py] Module iostat has missing
NOTIFY_TYPES
> member > Apr 05 20:33:51 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.020+0000 7f5a66492680 1 mgr[py] Loading
module
> 'k8sevents' > Apr 05 20:33:51 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.533+0000 7f5a66492680 1 mgr[py] Loading
module
> 'localpool' > Apr 05 20:33:51 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.634+0000 7f5a66492680 1 mgr[py] Loading
module
> 'mds_autoscaler' > Apr 05 20:33:51 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:51.894+0000 7f5a66492680 1 mgr[py] Loading
module
> 'mirroring' > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.019+0000 7f5a66492680 1 mgr[py] Loading
module
> 'nfs' > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 -1 mgr[py] Module nfs has missing
NOTIFY_TYPES
> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.209+0000 7f5a66492680 1 mgr[py] Loading
module
> 'orchestrator' > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 -1 mgr[py] Module orchestrator has missing
NOTIFY_TYPES
> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.386+0000 7f5a66492680 1 mgr[py] Loading
module
> 'osd_support' > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 -1 mgr[py] Module osd_support has missing
NOTIFY_TYPES
> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.494+0000 7f5a66492680 1 mgr[py] Loading
module
> 'pg_autoscaler' > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 -1 mgr[py] Module pg_autoscaler has missing
NOTIFY_TYPES
> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.760+0000 7f5a66492680 1 mgr[py] Loading
module
> 'progress' > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 -1 mgr[py] Module progress has missing
NOTIFY_TYPES
> member > Apr 05 20:33:52 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:52.870+0000 7f5a66492680 1 mgr[py] Loading
module
> 'prometheus' > Apr 05 20:33:53 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 -1 mgr[py] Module prometheus has missing
NOTIFY_TYPES
> member > Apr 05 20:33:53 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.257+0000 7f5a66492680 1 mgr[py] Loading
module
> 'rbd_support' > Apr 05 20:33:53 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 -1 mgr[py] Module rbd_support has missing
NOTIFY_TYPES
> member > Apr 05 20:33:53 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.388+0000 7f5a66492680 1 mgr[py] Loading
module
> 'restful' > Apr 05 20:33:53 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:53.799+0000 7f5a66492680 1 mgr[py] Loading
module
> 'rook' > Apr 05 20:33:54 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 -1 mgr[py] Module rook has missing
NOTIFY_TYPES
> member > Apr 05 20:33:54 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.318+0000 7f5a66492680 1 mgr[py] Loading
module
> 'selftest' > Apr 05 20:33:54 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 -1 mgr[py] Module selftest has missing
NOTIFY_TYPES
> member > Apr 05 20:33:54 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.428+0000 7f5a66492680 1 mgr[py] Loading
module
> 'snap_schedule' > Apr 05 20:33:54 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 -1 mgr[py] Module snap_schedule has missing
NOTIFY_TYPES
> member > Apr 05 20:33:54 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.549+0000 7f5a66492680 1 mgr[py] Loading
module
> 'stats' > Apr 05 20:33:54 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.660+0000 7f5a66492680 1 mgr[py] Loading
module
> 'status' > Apr 05 20:33:54 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 -1 mgr[py] Module status has missing
NOTIFY_TYPES
> member > Apr 05 20:33:54 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:54.972+0000 7f5a66492680 1 mgr[py] Loading
module
> 'telegraf' > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 -1 mgr[py] Module telegraf has missing
NOTIFY_TYPES
> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.089+0000 7f5a66492680 1 mgr[py] Loading
module
> 'telemetry' > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 -1 mgr[py] Module telemetry has missing
NOTIFY_TYPES
> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.268+0000 7f5a66492680 1 mgr[py] Loading
module
> 'test_orchestrator' > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 -1 mgr[py] Module test_orchestrator has missing
NOTIFY_TYPES
> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.448+0000 7f5a66492680 1 mgr[py] Loading
module
> 'volumes' > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 -1 mgr[py] Module volumes has missing
NOTIFY_TYPES
> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.676+0000 7f5a66492680 1 mgr[py] Loading
module
> 'zabbix' > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.792+0000 7f5a66492680 -1 mgr[py] Module zabbix has missing
NOTIFY_TYPES
> member > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.793+0000 7f5a52600700 0 ms_deliver_dispatch: unhandled message 0x556eb85f26e0 mon_map magic: 0 v1 from
mon.2
> v2:192.168.47.72:3300/0 > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [dashboard DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from
module:
> dashboard > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 0 [prometheus DEBUG root] setting log level based on debug_mgr:
INFO
> (2/5) > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f5a53000700 1 mgr load Constructed class from
module:
> prometheus > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.794+0000 7f58f7800700 0 [dashboard INFO root] server: ssl=yes
host=::
> port=8443 > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root] server_addr: ::
server_port:
> 9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO root]
Starting
> engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE
Bus
> STARTING > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.795+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE
Bus
> STARTING > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.797+0000 7f5a52600700 0 client.0 ms_handle_reset
on
> v2:192.168.47.73:6842/917991990 > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.825+0000 7f58f7800700 0 [dashboard INFO root] Configured CherryPy,
starting
> engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.826+0000 7f58f7800700 0 [dashboard INFO root]
Starting
> engine... > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:
CherryPy
> Checker: > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an
empty
> config. > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:
> Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE Serving
on
> http://:::9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE Serving
on
> http://:::9283 > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:33:55] ENGINE
Bus
> STARTED > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:33:55] ENGINE
Bus
> STARTED > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.900+0000 7f5900400700 0 [prometheus INFO root]
Engine
> started. > Apr 05 20:33:55 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:55.939+0000 7f58f7800700 0 [dashboard INFO root]
Engine
> started... > Apr 05 20:33:58 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:33:58 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:33:58.569+0000 7f59b8000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:58] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:00 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:00 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:00.702+0000 7f59af400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:00] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:03 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:03 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:03.286+0000 7f59a6800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:03] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:08 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:08 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:08.569+0000 7f599dc00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:08] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:10 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:10 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:10.702+0000 7f5995000700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:10] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:13 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:13 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:13.286+0000 7f598c400700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.73 - - [05/Apr/2025:20:34:13] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:18 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:18 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:18.569+0000 7f5983800700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.72 - - [05/Apr/2025:20:34:18] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:20 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200
""
> "Prometheus/2.33.4" > Apr 05 20:34:20 cn02.ceph.xyz.corp >
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:34:20.702+0000 7f597ac00700 0 [prometheus INFO cherrypy.access.140023317690912] ::ffff:192.168.47.71 - - [05/Apr/2025:20:34:20] "GET /metrics HTTP/1.1" 200
'daemon_types' that already. - - - - try python python python python python python python python python python python python python python python python python python python python python python python python python python python python python python python python - - - - - - - - - - - - - - - -
""
> "Prometheus/2.33.4" > > cn03 > —— > Apr 05 20:33:42 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.72 - - [05/Apr/2025:20:33:42] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:42 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:42.753+0000 7f2724a00700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.72 - - > [05/Apr/2025:20:33:42] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:43 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.71 - - [05/Apr/2025:20:33:43] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:43 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:43.772+0000 7f271be00700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.71 - - > [05/Apr/2025:20:33:43] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:47 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > ::ffff:192.168.47.73 - - [05/Apr/2025:20:33:47] "GET /metrics > HTTP/1.1" 200 - "" "Prometheus/2.33.4" > Apr 05 20:33:47 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:47.037+0000 7f2713200700 0 [prometheus INFO > cherrypy.access.139806526902960] ::ffff:192.168.47.73 - - > [05/Apr/2025:20:33:47] "GET /metrics HTTP/1.1" 200 - "" > "Prometheus/2.33.4" > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.867+0000 7f27d1a00700 1 mgr handle_mgr_map > Activating! > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.868+0000 7f27d1a00700 1 mgr handle_mgr_map > I am now activating > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > root] Stopping engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [dashboard INFO > root] Stopped engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > root] Stopping engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.878+0000 7f2663400700 0 [prometheus INFO > root] Stopped engine > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.879+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.890+0000 7f2675a00700 0 [dashboard INFO > root] Engine stopped. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE HTTP Server > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.891+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE HTTP Server > cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STOPPED > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.892+0000 7f267e600700 0 [prometheus INFO > root] Engine stopped. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 0 [alerts INFO root] > Init > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.898+0000 7f2663400700 1 mgr load > Constructed class from module: alerts > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f2663400700 0 [balancer DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f2663400700 1 mgr load > Constructed class from module: balancer > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.899+0000 7f26f8e00700 0 [alerts INFO root] > Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.901+0000 7f2663400700 0 [cephadm DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.901+0000 7f26f0200700 0 [balancer INFO > root] Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > root] Optimize plan auto_2025-04-05_20:33:48 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.908+0000 7f26f0200700 0 [balancer INFO > root] Mode upmap, max misplaced 0.050000 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO > root] Some PGs (1.000000) are unknown; try again later > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed > to construct class in 'cephadm' > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load > Traceback (most recent call last): > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.upgrade = CephadmUpgrade(self) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.upgrade_state: Optional[UpgradeState] = > UpgradeState.from_json(json.loads(t)) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > return cls(**c) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > TypeError: __init__() got an unexpected keyword argument > 'daemon_types' > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() > Failed to run module in active mode ('cephadm') > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [crash DEBUG root] > setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > Constructed class from module: crash > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 0 [dashboard DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.919+0000 7f2663400700 1 mgr load > Constructed class from module: dashboard > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.920+0000 7f2660c00700 0 [dashboard INFO > access_control] Loading user roles DB version=2 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > sso] Loading SSO DB version=1 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.921+0000 7f2660c00700 0 [dashboard INFO > root] server: ssl=yes host=:: port=8443 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [devicehealth > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > Constructed class from module: devicehealth > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 0 [insights DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.922+0000 7f2663400700 1 mgr load > Constructed class from module: insights > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.923+0000 7f2660200700 0 [devicehealth INFO > root] Starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f2663400700 0 [iostat DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f2663400700 1 mgr load > Constructed class from module: iostat > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.925+0000 7f265f800700 0 [insights INFO > root] Reset curr health slot key health_history/2025-04-05_20 next > flush None checks check count 0 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.926+0000 7f2663400700 0 [mirroring DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.931+0000 7f2663400700 1 mgr load > Constructed class from module: mirroring > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [nfs DEBUG root] > setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > Constructed class from module: nfs > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 0 [orchestrator > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.932+0000 7f2663400700 1 mgr load > Constructed class from module: orchestrator > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.933+0000 7f2663400700 0 [pg_autoscaler > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.933+0000 7f2663400700 1 mgr load > Constructed class from module: pg_autoscaler > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2663400700 0 [progress DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2663400700 1 mgr load > Constructed class from module: progress > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.934+0000 7f2657600700 0 [pg_autoscaler > INFO root] _maybe_adjust > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.940+0000 7f2656c00700 0 [progress INFO > root] Loading... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.940+0000 7f2663400700 0 [prometheus DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2663400700 1 mgr load > Constructed class from module: prometheus > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > root] server_addr: :: server_port: 9283 > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.944+0000 7f2656200700 0 [prometheus INFO > root] Cache enabled > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > root] Loaded [<progress.module.GhostEvent object at 0x7f27363ac400>, > <progress.module.GhostEvent object at 0x7f27363ac438>, > <progress.module.GhostEvent object at 0x7f27363ac3c8>, > <progress.module.GhostEvent object at 0x7f27363ac5c0>, > <progress.module.GhostEvent object at 0x7f27363ac4e0>, > <progress.module.GhostEvent object at 0x7f27363ac4a8>, > <progress.module.GhostEvent object at 0x7f278d317e10>, > <progress.module.GhostEvent object at 0x7f278d4397b8>, > <progress.module.GhostEvent object at 0x7f278d439908>, > <progress.module.GhostEvent object at 0x7f27363ed2e8>, > <progress.module.GhostEvent object at 0x7f27363ed4e0>, > <progress.module.GhostEvent object at 0x7f27363ed358>, > <progress.module.GhostEvent object at 0x7f27363ed278>, > <progress.module.GhostEvent object at 0x7f27363edeb8>, > <progress.module.GhostEvent object at 0x7f27363ed438>, > <progress.module.GhostEvent object at 0x7f27363ed400>, > <progress.module.GhostEvent object at 0x7f27363edb00>, > <progress.module.GhostEvent object at 0x7f27363ed780>, > <progress.module.GhostEvent object at 0x7f27363ed240>, > <progress.module.GhostEvent object at 0x7f27363edba8>, > <progress.module.GhostEvent object at 0x7f27363ed2b0>, > <progress.module.GhostEvent object at 0x7f27363ed0b8>, > <progress.module.GhostEvent object at 0x7f27363edcf8>, > <progress.module.GhostEvent object at 0x7f27363ed128>, > <progress.module.GhostEvent object at 0x7f27363ed080>, > <progress.module.GhostEvent object at 0x7f27363edc18>, > <progress.module.GhostEvent object at 0x7f27363ed860>, > <progress.module.GhostEvent object at 0x7f27363eda58>, > <progress.module.GhostEvent object at 0x7f27363eda90>, > <progress.module.GhostEvent object at 0x7f27363ed160>, > <progress.module.GhostEvent object at 0x7f27363ed908>, > <progress.module.GhostEvent object at 0x7f27363ed940>, > <progress.module.GhostEvent object at 0x7f27363ed8d0>, > <progress.module.GhostEvent object at 0x7f27363ed978>, > <progress.module.GhostEvent object at 0x7f27363ed748>, > <progress.module.GhostEvent object at 0x7f27363ed898>, > <progress.module.GhostEvent object at 0x7f27363eda20>, > <progress.module.GhostEvent object at 0x7f27363ed9b0>, > <progress.module.GhostEvent object at 0x7f27363ed9e8>, > <progress.module.GhostEvent object at 0x7f27363ed828>, > <progress.module.GhostEvent object at 0x7f27363ed7f0>, > <progress.module.GhostEvent object at 0x7f27363ed7b8>, > <progress.module.GhostEvent object at 0x7f27363ed668>, > <progress.module.GhostEvent object at 0x7f27363ed320>, > <progress.module.GhostEvent object at 0x7f27363ede48>, > <progress.module.GhostEvent object at 0x7f27363edc50>, > <progress.module.GhostEvent object at 0x7f27363ede80>, > <progress.module.GhostEvent object at 0x7f27363edf60>, > <progress.module.GhostEvent object at 0x7f27363edd68>, > <progress.module.GhostEvent object at 0x7f27363edda0>] historic events > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.945+0000 7f2656c00700 0 [progress INFO > root] Loaded OSDMap, ready. > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.946+0000 7f2663400700 0 [rbd_support DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.946+0000 7f2655800700 0 [prometheus INFO > root] starting metric collection thread > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.947+0000 7f2656200700 0 [prometheus INFO > root] Starting engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:48] ENGINE Bus STARTING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.948+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:48] ENGINE Bus STARTING > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.948+0000 7f2660c00700 0 [dashboard INFO > root] Configured CherryPy, starting engine... > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.953+0000 7f2663400700 0 [rbd_support INFO > root] MirrorSnapshotScheduleHandler: load_schedules > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.957+0000 7f2663400700 0 [rbd_support INFO > root] load_schedules: csxyz, start_after= > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.989+0000 7f2649a00700 0 [rbd_support INFO > root] MirrorSnapshotScheduleHandler: starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.990+0000 7f2649000700 0 [rbd_support INFO > root] PerfHandler: starting > Apr 05 20:33:48 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:48.991+0000 7f2663400700 0 [rbd_support INFO > root] load_task_task: csxyz, start_after= > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.019+0000 7f2648600700 0 [rbd_support INFO > root] TaskHandler: starting > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.020+0000 7f2663400700 0 [rbd_support INFO > root] TrashPurgeScheduleHandler: load_schedules > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.022+0000 7f2663400700 0 [rbd_support INFO > root] load_schedules: csxyz, start_after= > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.024+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CephFS -> /api/cephfs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.026+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CephFsUi -> /ui-api/cephfs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > IscsiUi -> /ui-api/iscsi > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Iscsi -> /api/iscsi > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.028+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > IscsiTarget -> /api/iscsi/target > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGanesha -> /api/nfs-ganesha > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaCluster -> /api/nfs-ganesha/cluster > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.030+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaExports -> /api/nfs-ganesha/export > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > NFSGaneshaUi -> /ui-api/nfs-ganesha > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.031+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Rbd > -> /api/block/image > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.034+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdSnapshot -> /api/block/image/{image_spec}/snap > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdTrash -> /api/block/image/trash > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.035+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdNamespace -> /api/block/pool/{pool_name}/namespace > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroring -> /api/block/mirroring > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.036+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringSummary -> /api/block/mirroring/summary > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolMode -> /api/block/mirroring/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolBootstrap -> > /api/block/mirroring/pool/{pool_name}/bootstrap > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.037+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirroringPoolPeer -> /api/block/mirroring/pool/{pool_name}/peer > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.038+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Rgw > -> /api/rgw > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwDaemon -> /api/rgw/daemon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.039+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwSite -> /api/rgw/site > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.040+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwBucket -> /api/rgw/bucket > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.041+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwUser -> /api/rgw/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Auth -> /api/auth > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.044+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Cluster -> /api/cluster > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.045+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ClusterConfiguration -> /api/cluster_conf > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.046+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CrushRule -> /api/crush_rule > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.047+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > CrushRuleUi -> /ui-api/crush_rule > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Orchestrator -> /api/orchestrator > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.048+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Daemon -> /api/daemon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Docs -> /docs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.049+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ErasureCodeProfile -> /api/erasure_code_profile > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > ErasureCodeProfileUi -> /ui-api/erasure_code_profile > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.050+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackController -> /api/feedback > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.051+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackApiController -> /api/feedback/api_key > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeedbackUiController -> /ui-api/feedback/api_key > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FrontendLogging -> /ui-api/logging > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Grafana -> /api/grafana > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.052+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Host -> /api/host > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > HostUi -> /ui-api/host > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Health -> /api/health > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.054+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > HomeController -> / > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:49] ENGINE Serving on http://:::9283 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Serving on > http://:::9283 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > [05/Apr/2025:20:33:49] ENGINE Bus STARTED > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > cherrypy.error] [05/Apr/2025:20:33:49] ENGINE Bus STARTED > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.055+0000 7f2656200700 0 [prometheus INFO > root] Engine started. > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > LangsController -> /ui-api/langs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Logs -> /api/logs > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.056+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MgrModules -> /api/mgr/module > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Monitor -> /api/monitor > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.058+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: Osd > -> /api/osd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.060+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > OsdFlagsController -> /api/osd/flags > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MdsPerfCounter -> /api/perf_counters/mds > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.061+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MonPerfCounter -> /api/perf_counters/mon > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > OsdPerfCounter -> /api/perf_counters/osd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.062+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RgwPerfCounter -> /api/perf_counters/rgw > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > RbdMirrorPerfCounter -> /api/perf_counters/rbd-mirror > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.063+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MgrPerfCounter -> /api/perf_counters/mgr > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > TcmuRunnerPerfCounter -> /api/perf_counters/tcmu-runner > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.064+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PerfCounters -> /api/perf_counters > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.065+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Pool -> /api/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.066+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PoolUi -> /ui-api/pool > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PrometheusReceiver -> /api/prometheus_receiver > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.067+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Prometheus -> /api/prometheus > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.068+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > PrometheusNotifications -> /api/prometheus/notifications > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.069+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Role -> /api/role > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Scope -> /ui-api/scope > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Saml2 -> /auth/saml2 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.070+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Service -> /api/service > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.071+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Settings -> /api/settings > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > StandardSettings -> /ui-api/standard_settings > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Summary -> /api/summary > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.073+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Task -> /api/task > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > Telemetry -> /api/telemetry > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.074+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > User -> /api/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > UserPasswordPolicy -> /api/user > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > UserChangePassword -> /api/user/{username} > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > FeatureTogglesEndpoint -> /api/feature_toggles > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.076+0000 7f2660c00700 0 [dashboard INFO > dashboard.controllers._base_controller] Initializing controller: > MessageOfTheDay -> /ui-api/motd > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.090+0000 7f2647200700 0 [rbd_support INFO > root] TrashPurgeScheduleHandler: starting > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.096+0000 7f2663400700 1 mgr load > Constructed class from module: rbd_support > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.102+0000 7f2663400700 0 [restful DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.102+0000 7f2663400700 1 mgr load > Constructed class from module: restful > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.108+0000 7f2645e00700 0 [restful INFO > root] server_addr: :: server_port: 8003 > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.109+0000 7f2663400700 0 [snap_schedule > DEBUG root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.116+0000 7f2645e00700 0 [restful WARNING > root] server not running: no certificate configured > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.257+0000 7f2660c00700 0 [dashboard INFO > dashboard.module] Engine started. > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed > to construct class in 'snap_schedule' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load > Traceback (most recent call last): > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, in > __init__ > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > self.client = SnapSchedClient(self) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > 158, in __init__ > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > with self.get_schedule_db(fs_name) as conn_mgr: > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > 192, in get_schedule_db > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > db.executescript(dump) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > sqlite3.OperationalError: table schedules already exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() > Failed to run module in active mode ('snap_schedule') > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 0 [status DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.274+0000 7f2663400700 1 mgr load > Constructed class from module: status > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.275+0000 7f2663400700 0 [telemetry DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.276+0000 7f2663400700 1 mgr load > Constructed class from module: telemetry > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.276+0000 7f2663400700 0 [volumes DEBUG > root] setting log level based on debug_mgr: INFO (2/5) > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.284+0000 7f2660200700 0 [devicehealth INFO > root] Check health > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'cssnfsxyz' > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.290+0000 7f2663400700 1 mgr load > Constructed class from module: volumes > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.corp > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > debug 2025-04-05T20:33:49.295+0000 7f2627e00700 -1 client.0 error > registering admin socket command: (17) File exists > Apr 05 20:33:49 cn03.ceph.xyz.co [Text gekürzt]