- ceph-users - lists.ceph.io

Most OSDs down and all PGs unknown after P2V migration
by leo＠mylonas.co 13 Jan '25

13 Jan '25

I run a small single-node Ceph deployment (had plans to scale up once I got more equipment) for home file storage deployed using cephadm. It was running bare-metal, and I attempted a physical-to-virtual migration to a Proxmox VM. After doing so, all of my PGs seemed to be "unknown". Initial after a boot, the OSDs appear to be up, but after a while, most of them go down. I assume some sort of timeout in the OSD start process. The systemd processes (and podman containers) are still running and appear to be happy. I don't see anything crazy in their logs. I'm relatively new to Ceph, so I don't really know where to go from here. Can anyone provide any guidance? The logs for the monitor and one of the OSDs can be found here - https://gitlab.com/-/snippets/4793143 And here is the output from a few commands that might be useful. ceph -s ``` cluster: id: 768819b0-a83f-11ee-81d6-74563c5bfc7b health: HEALTH_WARN Reduced data availability: 545 pgs inactive 139 pgs not deep-scrubbed in time 17 slow ops, oldest one blocked for 1668 sec, mon.fileserver has slow ops services: mon: 1 daemons, quorum fileserver (age 28m) mgr: fileserver.rgtdvr(active, since 28m), standbys: fileserver.gikddq osd: 17 osds: 5 up (since 116m), 5 in (since 10m) data: pools: 3 pools, 545 pgs objects: 1.97M objects, 7.5 TiB usage: 7.7 TiB used, 1.4 TiB / 9.1 TiB avail pgs: 100.000% pgs unknown 545 unknown ``` ceph osd df ``` ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 1 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 3 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 112 down 4 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 117 down 5 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 6 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 7 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 8 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 106 down 20 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 115 down 21 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 94 down 22 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 98 down 23 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 109 down 24 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 4 KiB 3.0 GiB 186 GiB 90.00 1.06 117 up 25 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 10 KiB 2.8 GiB 220 GiB 88.18 1.04 114 up 26 hdd 1.81940 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 9 KiB 2.8 GiB 297 GiB 84.07 0.99 109 up 27 hdd 1.81940 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 7 KiB 2.5 GiB 474 GiB 74.58 0.88 98 up 28 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 10 KiB 3.0 GiB 206 GiB 88.93 1.04 115 up TOTAL 9.1 TiB 7.7 TiB 7.7 TiB 42 KiB 14 GiB 1.4 TiB 85.15 MIN/MAX VAR: 0.88/1.06 STDDEV: 5.65 ``` ceph pg stat ``` 545 pgs: 545 unknown; 7.5 TiB data, 7.7 TiB used, 1.4 TiB / 9.1 TiB avail ``` systemctl | grep ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b ``` ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)alertmanager.fileserver.serv ice loaded active running Ceph alertmanager.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)ceph-exporter.fileserver.ser vice loaded active running Ceph ceph-exporter.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)crash.fileserver.service loaded active running Ceph crash.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)grafana.fileserver.service loaded active running Ceph grafana.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mgr.fileserver.gikddq.servic e loaded active running Ceph mgr.fileserver.gikddq for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mgr.fileserver.rgtdvr.servic e loaded active running Ceph mgr.fileserver.rgtdvr for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mon.fileserver.service loaded active running Ceph mon.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.0.service loaded active running Ceph osd.0 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.1.service loaded active running Ceph osd.1 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.20.service loaded active running Ceph osd.20 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.21.service loaded active running Ceph osd.21 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.22.service loaded active running Ceph osd.22 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.23.service loaded active running Ceph osd.23 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.24.service loaded active running Ceph osd.24 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.25.service loaded active running Ceph osd.25 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.26.service loaded active running Ceph osd.26 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.27.service loaded active running Ceph osd.27 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.28.service loaded active running Ceph osd.28 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.3.service loaded active running Ceph osd.3 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.4.service loaded active running Ceph osd.4 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.5.service loaded active running Ceph osd.5 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.6.service loaded active running Ceph osd.6 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.7.service loaded active running Ceph osd.7 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.8.service loaded active running Ceph osd.8 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)prometheus.fileserver.servic e loaded active running Ceph prometheus.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b system-ceph\x2d768819b0\x2da83f\x2d11ee\x2d81d6\x2d74563c5bfc7b.slice loaded active active Slice /system/ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b.target loaded active active Ceph cluster 768819b0-a83f-11ee-81d6-74563c5bfc7b ```

1 0

Most OSDs down and all PGs unknown after P2V migration
by leo＠mylonas.co 13 Jan '25

13 Jan '25

I run a small single-node ceph cluster (the plan was to scale out when I could get more equipment) for home file storage deployed by cephadm. It was running bare-metal, and I attempted a physical-to-virtual migration to a Proxmox VM After doing so, all of my PGs seemed to be "unknown". Initial after a boot, the OSDs appear to be up, but after a while, they go down. I assume some sort of timeout in the OSD start process. The systemd processes (and podman containers) are still running and appear to be happy. I don't see anything that jumps out at me in their logs. I'm relatively new to Ceph, so I don't really know where to go from here. Can anyone provide any guidance? **ceph -s** ``` cluster: id: 768819b0-a83f-11ee-81d6-74563c5bfc7b health: HEALTH_WARN Reduced data availability: 545 pgs inactive 139 pgs not deep-scrubbed in time 17 slow ops, oldest one blocked for 1668 sec, mon.fileserver has slow ops services: mon: 1 daemons, quorum fileserver (age 28m) mgr: fileserver.rgtdvr(active, since 28m), standbys: fileserver.gikddq osd: 17 osds: 5 up (since 116m), 5 in (since 10m) data: pools: 3 pools, 545 pgs objects: 1.97M objects, 7.5 TiB usage: 7.7 TiB used, 1.4 TiB / 9.1 TiB avail pgs: 100.000% pgs unknown 545 unknown ``` **ceph osd df** ``` ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 1 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 3 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 112 down 4 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 117 down 5 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 6 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 7 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 8 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 106 down 20 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 115 down 21 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 94 down 22 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 98 down 23 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 109 down 24 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 4 KiB 3.0 GiB 186 GiB 90.00 1.06 117 up 25 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 10 KiB 2.8 GiB 220 GiB 88.18 1.04 114 up 26 hdd 1.81940 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 9 KiB 2.8 GiB 297 GiB 84.07 0.99 109 up 27 hdd 1.81940 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 7 KiB 2.5 GiB 474 GiB 74.58 0.88 98 up 28 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 10 KiB 3.0 GiB 206 GiB 88.93 1.04 115 up TOTAL 9.1 TiB 7.7 TiB 7.7 TiB 42 KiB 14 GiB 1.4 TiB 85.15 MIN/MAX VAR: 0.88/1.06 STDDEV: 5.65 ``` **ceph pg stat** ``` 545 pgs: 545 unknown; 7.5 TiB data, 7.7 TiB used, 1.4 TiB / 9.1 TiB avail ``` **systemctl | grep ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b** ``` ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)alertmanager.fileserver.service loaded active running Ceph alertmanager.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)ceph-exporter.fileserver.service loaded active running Ceph ceph-exporter.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)crash.fileserver.service loaded active running Ceph crash.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)grafana.fileserver.service loaded active running Ceph grafana.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mgr.fileserver.gikddq.service loaded active running Ceph mgr.fileserver.gikddq for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mgr.fileserver.rgtdvr.service loaded active running Ceph mgr.fileserver.rgtdvr for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mon.fileserver.service loaded active running Ceph mon.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.0.service loaded active running Ceph osd.0 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.1.service loaded active running Ceph osd.1 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.20.service loaded active running Ceph osd.20 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.21.service loaded active running Ceph osd.21 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.22.service loaded active running Ceph osd.22 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.23.service loaded active running Ceph osd.23 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.24.service loaded active running Ceph osd.24 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.25.service loaded active running Ceph osd.25 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.26.service loaded active running Ceph osd.26 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.27.service loaded active running Ceph osd.27 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.28.service loaded active running Ceph osd.28 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.3.service loaded active running Ceph osd.3 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.4.service loaded active running Ceph osd.4 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.5.service loaded active running Ceph osd.5 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.6.service loaded active running Ceph osd.6 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.7.service loaded active running Ceph osd.7 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.8.service loaded active running Ceph osd.8 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)prometheus.fileserver.service loaded active running Ceph prometheus.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b system-ceph\x2d768819b0\x2da83f\x2d11ee\x2d81d6\x2d74563c5bfc7b.slice loaded active active Slice /system/ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b.target loaded active active Ceph cluster 768819b0-a83f-11ee-81d6-74563c5bfc7b ``` The logs for `mon` and `osd.3` can be found here - https://gitlab.com/-/snippets/4793143

1 0

MDSs report oversized cache during forward scrub
by Frank Schilder 11 Jan '25

11 Jan '25

Hi all, we started a forward scrub on our 5.x PB ceph file system and observe a massive ballooning of MDS caches. Our status is: # ceph status cluster: id: xxx health: HEALTH_WARN 1 MDSs report oversized cache (muted: MDS_CLIENT_LATE_RELEASE(12d) MDS_CLIENT_RECALL(12d) PG_NOT_DEEP_SCRUBBED(5d) PG_NOT_SCRUBBED(5d) POOL_NEAR_FULL(4w)) services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 4w) mgr: ceph-03(active, since 3w), standbys: ceph-01, ceph-25, ceph-26, ceph-02 mds: 8/8 daemons up, 4 standby osd: 1317 osds: 1316 up (since 5d), 1316 in (since 5w) task status: scrub status: mds.ceph-08: active paths [/] mds.ceph-11: active paths [/] mds.ceph-12: active paths [/] mds.ceph-14: active paths [/] mds.ceph-15: active paths [/] mds.ceph-17: active paths [/] mds.ceph-24: active paths [/] data: volumes: 1/1 healthy pools: 14 pools, 29161 pgs objects: 4.38G objects, 5.5 PiB usage: 7.2 PiB used, 5.9 PiB / 13 PiB avail pgs: 29129 active+clean 28 active+clean+scrubbing+deep 2 active+clean+snaptrim 2 active+clean+scrubbing io: client: 484 MiB/s rd, 64 MiB/s wr, 4.64k op/s rd, 1.50k op/s wr # ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-12 Reqs: 3 /s 13.8M 13.8M 478k 29.2k 1 active ceph-15 Reqs: 31 /s 10.5M 10.5M 140k 441k 2 active ceph-14 Reqs: 0 /s 11.5M 11.5M 504k 32.8k 3 active ceph-17 Reqs: 5 /s 12.4M 12.4M 487k 30.9k 4 active ceph-08 Reqs: 0 /s 15.3M 15.3M 247k 47.2k 5 active ceph-11 Reqs: 7 /s 4414k 4413k 262k 72.5k 6 active ceph-16 Reqs: 409 /s 1079k 1057k 7766 17.3k 7 active ceph-24 Reqs: 41 /s 4074k 4074k 448k 109k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 6657G con-fs2-meta2 data 0 6657G con-fs2-data data 1225T 2272T con-fs2-data-ec-ssd data 794G 20.8T con-fs2-data2 data 5745T 2066T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-13 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) Ranks 0-4 are massively oversized. Ranks 5 and 7 show usual values. Rank 6 is low, because we restarted it already due to this warning. It seems as if MDSes don't trim cache during scrub, the sizes only increase. Is this ballooning normal or a bug? Is there a workaround apart from restarting MDSes all the time? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

MDSs report oversized cache during forward scrub
by Frank Schilder 10 Jan '25

10 Jan '25

Hi all, we started a forward scrub on our 5.x PB ceph file system and observe a massive ballooning of MDS caches. Our status is: # ceph status cluster: id: xxx health: HEALTH_WARN 1 MDSs report oversized cache (muted: MDS_CLIENT_LATE_RELEASE(12d) MDS_CLIENT_RECALL(12d) PG_NOT_DEEP_SCRUBBED(5d) PG_NOT_SCRUBBED(5d) POOL_NEAR_FULL(4w)) services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 4w) mgr: ceph-03(active, since 3w), standbys: ceph-01, ceph-25, ceph-26, ceph-02 mds: 8/8 daemons up, 4 standby osd: 1317 osds: 1316 up (since 5d), 1316 in (since 5w) task status: scrub status: mds.ceph-08: active paths [/] mds.ceph-11: active paths [/] mds.ceph-12: active paths [/] mds.ceph-14: active paths [/] mds.ceph-15: active paths [/] mds.ceph-17: active paths [/] mds.ceph-24: active paths [/] data: volumes: 1/1 healthy pools: 14 pools, 29161 pgs objects: 4.38G objects, 5.5 PiB usage: 7.2 PiB used, 5.9 PiB / 13 PiB avail pgs: 29129 active+clean 28 active+clean+scrubbing+deep 2 active+clean+snaptrim 2 active+clean+scrubbing io: client: 484 MiB/s rd, 64 MiB/s wr, 4.64k op/s rd, 1.50k op/s wr # ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-12 Reqs: 3 /s 13.8M 13.8M 478k 29.2k 1 active ceph-15 Reqs: 31 /s 10.5M 10.5M 140k 441k 2 active ceph-14 Reqs: 0 /s 11.5M 11.5M 504k 32.8k 3 active ceph-17 Reqs: 5 /s 12.4M 12.4M 487k 30.9k 4 active ceph-08 Reqs: 0 /s 15.3M 15.3M 247k 47.2k 5 active ceph-11 Reqs: 7 /s 4414k 4413k 262k 72.5k 6 active ceph-16 Reqs: 409 /s 1079k 1057k 7766 17.3k 7 active ceph-24 Reqs: 41 /s 4074k 4074k 448k 109k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 6657G con-fs2-meta2 data 0 6657G con-fs2-data data 1225T 2272T con-fs2-data-ec-ssd data 794G 20.8T con-fs2-data2 data 5745T 2066T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-13 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) Ranks 0-4 are massively oversized. Ranks 5 and 7 show usual values. Rank 6 is low, because we restarted it already due to this warning. It seems as if MDSes don't trim cache during scrub, the sizes only increase. Is this ballooning normal or a bug? Is there a workaround apart from restarting MDSes all the time? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Per-Client Quality of Service settings
by Olaf Seibert 10 Jan '25

10 Jan '25

Hi! I am trying to find if Ceph has any QoS settings that apply per-client. I would like to be able, say, to have different QoS settings for RBD clients named "nova" and "cinder" and different again from an RGW client named "enduser". While researching this, I found amongst other things, the following references, which do not appear to be what I want: - https://docs.ceph.com/en/reef/dev/osd_internals/mclock_wpq_cmp_study/ and https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/#dmclock-q… . This appears to be instead about the balancing of user-iops against ceph-internal iops, not balancing clients against each other. (aside: somebody should rewrite this paragraph, I find it totally incomprehensible: https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/#subtletie… ) - https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#qos-settings and https://tracker.ceph.com/issues/36191 This seems to be about setting limits on the RBD pools as a whole, or on separate images inside them. However separate clients each still get to use up to the limits. Is there anything else that I have not found so far but which is about balancing individual clients across all services? -- Olaf Seibert Site Reliability Engineer SysEleven GmbH Boxhagener Straße 80 10245 Berlin T +49 30 233 2012 0 F +49 30 616 7555 0 https://www.syseleven.de https://www.linkedin.com/company/syseleven-gmbh/ Current system status always at: https://www.syseleven-status.net/ Company headquarters: Berlin Registered court: AG Berlin Charlottenburg, HRB 108571 Berlin Managing directors: Andreas Hermann, Jens Ihlenfeld, Norbert Müller, Jens Plogsties

2 2

Ceph Orchestrator ignores attribute filters for SSDs
by Janek Bevendorff 10 Jan '25

10 Jan '25

Hi, I'm having a strange problem with the orchestrator. My cluster has the following OSD services configured based on certain attributes of the disks: NAME PORTS RUNNING REFRESHED AGE PLACEMENT ... osd.osd-default-hdd 1351 2m ago 22m label:osd;HOSTPREFIX* osd.osd-default-ssd 0 - 22m label:osd;HOSTPREFIX* osd.osd-small-hdd 41 2m ago 22m label:osd;HOSTPREFIX* These apply to three device types: large HDDs (8TB+), small HDDs (250G-7TB), and SSDs (1TB+). I did that with the following YAML definition: service_type: osd service_id: osd-default-hdd service_name: osd.osd-default-hdd placement: host_pattern: HOSTPREFIX* label: osd spec: crush_device_class: hdd data_devices: rotational: 1 size: '8T:' filter_logic: AND objectstore: bluestore osds_per_device: 1 --- service_type: osd service_id: osd-default-ssd service_name: osd.osd-default-ssd placement: host_pattern: HOSTPREFIX* label: osd spec: crush_device_class: ssd data_devices: rotational: 0 size: '1T:' filter_logic: AND objectstore: bluestore osds_per_device: 1 --- service_type: osd service_id: osd-small-hdd service_name: osd.osd-small-hdd placement: host_pattern: HOSTPREFIX* label: osd spec: crush_device_class: hdd-small data_devices: rotational: 1 size: 250G:7T filter_logic: AND objectstore: bluestore osds_per_device: 1 Previously, this worked perfectly, but as you can see in the summary above, now the orchestrator suddenly started to ignore the device class and data_devices filters for SSDs and incorrectly added all SSDs to the osd.osd-default-hdd service (weirdly enough, hdd-small still works). The affected devices still have the correct device class in the CRUSH tree and it also looks like the data placement is fine. The orchestrator service listing, however, is incorrect. I tried cleaning out and freshly redeploying one of the SSD OSDs, but the redeployed service still has the following in the unit.meta file: { "service_name": "osd.osd-default-hdd", "ports": [], "ip": null, "deployed_by": [ "quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77", "quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906" ], "rank": null, "rank_generation": null, "extra_container_args": null, "extra_entrypoint_args": null, "memory_request": null, "memory_limit": null } Any idea what might be causing this? I'm on Ceph 18.2.4 (upgrade planned, but I need to wait out some remapped PGs first). Janek

2 1

ceph tell throws WARN: the service id you provided does not exist.
by Frank Schilder 10 Jan '25

10 Jan '25

Hi all, following the documentation for initiating a file system scrub (pacific: https://docs.ceph.com/en/pacific/cephfs/scrub/) I can't get the syntax with "mds.<fsname>" to work: # ceph tell mds.con-fs2:0 help WARN: the service id you provided does not exist. service id should be one of ceph-09/ceph-10/ceph-23/ceph-16/ceph-24/ceph-08/ceph-17/ceph-12/ceph-13/ceph-15/ceph-11/ceph-14. target mds.con-fs2:0 doesn't exists, please pass correct target to tell command, such as mon.a/osd.1/mds.a/mgr The file system name is "con-fs2". I can reach the current rank-0 mds though. We are on the latest pacific version. Is this expected or do we need to configure something? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

Find out num of PGs that would go offline on OSD shutdown
by Andre Tann 10 Jan '25

10 Jan '25

Hi all, Proxmox VE warns me if I want to shut down an OSD, and some PGs would go offline if I continue. In the warning it tells me the exact number of affected PGs. I have now searched a bit, but wasn't able to find out how Proxmox knows the exact number. How can that be done on the CLI? Thanks for some hints. -- Andre Tann

3 3

Can I delete cluster_network?
by 苏察哈尔灿 10 Jan '25

10 Jan '25

   Hello, I have installed a ceph cluster, version v17.2.8, the cluster has public_network and cluster_network. For some other reason, I want to temporarily remove cluster_network and let the service data, heartbeat detection, and data recovery be done on public_network. Is this achievable? After removing it, if you want to configure it back, can you also achieve it? Thank you!

2 1