- ceph-users - lists.ceph.io

MANY_OBJECTS_PER_PG & TOO_MANY_PGS
by Torkil Svensgaard 12 May '25

12 May '25

Hi 19.2.2 Got a couple new warnings when I doubled pg_num for the cephfs.hdd.data pool: [root@lazy ~]# ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average; too many PGs per OSD (261 > max 250) [WRN] MANY_OBJECTS_PER_PG: 1 pools have many more objects per pg than average pool cephfs.cephfs.data objects per pg (4084905) is more than 161.862 times cluster average (25237) [WRN] TOO_MANY_PGS: too many PGs per OSD (261 > max 250) [root@lazy ~]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 4.1 PiB 1.6 PiB 2.5 PiB 2.5 PiB 61.12 nvme 210 TiB 58 TiB 152 TiB 152 TiB 72.49 nvmebulk 196 TiB 149 TiB 47 TiB 47 TiB 23.83 ssd 49 TiB 32 TiB 16 TiB 16 TiB 33.66 TOTAL 4.5 PiB 1.8 PiB 2.7 PiB 2.7 PiB 59.77 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL rbd 4 2048 111 TiB 29.39M 288 TiB 25.04 288 TiB libvirt 5 256 3.3 TiB 872.28k 6.8 TiB 37.73 3.7 TiB rbd_internal 6 2048 103 TiB 32.71M 243 TiB 21.96 288 TiB .mgr 8 1 4.9 GiB 1.26k 2.1 GiB 0.02 5.6 TiB rbd_ec 10 32 9.0 MiB 27 4.6 MiB 0 3.7 TiB rbd_ec_data 11 16384 1.1 PiB 284.37M 1.4 PiB 63.10 576 TiB rbd.nvme 23 2048 95 TiB 25.16M 147 TiB 77.45 21 TiB .nfs 25 32 20 KiB 73 286 KiB 0 3.7 TiB cephfs.cephfs.meta 31 128 15 GiB 3.05M 46 GiB 0.40 3.7 TiB cephfs.cephfs.data 32 512 449 B 130.72M 48 KiB 0 3.7 TiB cephfs.nvme.data 34 32 977 GiB 250k 122 GiB 0.28 21 TiB cephfs.ssd.data 35 32 853 GiB 1.10M 1.9 TiB 14.65 3.7 TiB cephfs.hdd.data 37 3098 208 TiB 175.22M 429 TiB 33.18 384 TiB rbd.ssd 39 64 1.6 TiB 433.36k 4.3 TiB 27.65 3.7 TiB rbd.ssd.ec 43 32 2.5 KiB 5 20 KiB 0 3.7 TiB rbd.ssd.ec.data 44 32 1.0 TiB 269.92k 2.0 TiB 14.89 5.0 TiB rbd.nvmebulk.ec 47 32 3.0 MiB 6 6.1 MiB 0 8.3 TiB rbd.nvmebulk.data 48 512 23 TiB 6.00M 46 TiB 64.98 11 TiB MANY_OBJECTS_PER_PG: The ceph.cephfs.data pool stores 449B in 130.72M objects? That is weird? We don't actually use that pool as as far as I know, as we use media specific data pools for cephfs, but is evidently being used for something regardless? It was created as part of the original cephfs file system where the command created both .meta and .data. TOO_MANY_PGS: [root@lazy ~]# ceph config dump | grep mon| grep max mon advanced mon_max_pg_per_osd 1000 osd advanced mon_max_pg_per_osd 1000 Well above 250. There used to be a mon_pg_warn_max_per_osd key but that was removed back in Luminous so what controls this warning? [root@lazy ~]# ceph config get mon mon_pg_warn_max_per_osd Error ENOENT: unrecognized key 'mon_pg_warn_max_per_osd' Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 1

Staggered upgrade or not
by Kasper Rasmussen 12 May '25

12 May '25

Hi Upgrading from Pacific 16.2.15 to Quincy 17.2.5 Environment: Production Size: 9PB RAW capacity 5 mon/mgr/mds host. 56 storage nodes I’m looking for advice to help me decide between - Staggered upgrade or not. Thanks

5 5

We lost the stability of the cluster, 18.2.2 -> 18.2.6 -> 19.2.1 Chain of upgrade failure
by Senol COLAK 12 May '25

12 May '25

Hello, After upgrading from ceph reef 18.2.6 to ceph squid 19.2.1 I restarted the osds and they remained down. The events contain the following records: root@cmt6770:~# ceph health detail HEALTH_WARN 1 filesystem is degraded; 1 MDSs report slow metadata IOs; mon cmt6770 is low on available space; 9 osds down; 1 host (3 osds) down; 5 nearfull osd(s); Reduced data availability: 866 pgs inactive, 489 pgs down, 5 pgs incomplete, 60 pgs stale; Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull; Degraded data redundancy: 432856/3408880 objects degraded (12.698%), 191 pgs degraded, 181 pgs undersized; 12 pool(s) nearfull; 255 slow ops, oldest one blocked for 2417 sec, daemons [osd.10,osd.12,osd.21,osd.22,osd.23] have slow ops. [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.cmt5923(mds.0): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 38201 secs [WRN] MON_DISK_LOW: mon cmt6770 is low on available space mon.cmt6770 has 28% avail [WRN] OSD_DOWN: 9 osds down osd.0 (root=default,host=cmt6770) is down osd.1 (root=default,host=cmt6770) is down osd.2 (root=default,host=cmt6461) is down osd.7 (root=default,host=cmt7773) is down osd.8 (root=default,host=cmt5923) is down osd.9 (root=default,host=cmt5923) is down osd.14 (root=default,host=cmt6770) is down osd.17 (root=default,host=cmt6461) is down osd.24 (root=default,host=cmt6461) is down [WRN] OSD_HOST_DOWN: 1 host (3 osds) down host cmt6461 (root=default) (3 osds) is down [WRN] OSD_NEARFULL: 5 nearfull osd(s) osd.3 is near full osd.12 is near full osd.16 is near full osd.21 is near full osd.23 is near full [WRN] PG_AVAILABILITY: Reduced data availability: 866 pgs inactive, 489 pgs down, 5 pgs incomplete, 60 pgs stale pg 7.1c5 is down, acting [3] pg 7.1c7 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1c8 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1cb is stuck inactive for 3h, current state unknown, last acting [] pg 7.1cd is down, acting [15,3] pg 7.1ce is stuck inactive for 3h, current state unknown, last acting [] pg 7.1cf is stuck inactive for 3h, current state unknown, last acting [] pg 7.1d0 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1d1 is down, acting [29,13] pg 7.1d2 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1d3 is down, acting [23] pg 7.1d4 is down, acting [16] pg 7.1d5 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1d6 is down, acting [3] pg 7.1d9 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1da is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e0 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e1 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e2 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e4 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e5 is down, acting [12,29] pg 7.1e7 is stuck stale for 7m, current state stale+down, last acting [9] pg 7.1e8 is down, acting [12] pg 7.1e9 is stuck stale for 31m, current state stale, last acting [2] pg 7.1eb is stuck inactive for 3h, current state unknown, last acting [] pg 7.1ed is down, acting [3] pg 7.1ee is stuck inactive for 3h, current state unknown, last acting [] pg 7.1ef is down, acting [12] pg 7.1f0 is down, acting [10] pg 7.1f1 is down, acting [12,29] pg 7.1f2 is down, acting [16] pg 7.1f3 is stuck stale for 7m, current state stale, last acting [9] pg 7.1f4 is down, acting [22] pg 7.1f5 is down, acting [22] pg 7.1f8 is down, acting [29] pg 7.1f9 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1fb is stuck inactive for 3h, current state unknown, last acting [] pg 7.1fc is stuck stale for 15m, current state stale+down, last acting [29] pg 7.1fd is down, acting [3] pg 7.1fe is down, acting [3,15] pg 7.1ff is down, acting [3] pg 7.201 is down, acting [12] pg 7.204 is down, acting [10] pg 7.205 is down, acting [13] pg 7.207 is down, acting [11] pg 7.20a is down, acting [3] pg 7.20b is down, acting [22] pg 7.20d is stuck inactive for 3h, current state unknown, last acting [] pg 7.210 is stuck inactive for 3h, current state unknown, last acting [] pg 7.211 is stuck inactive for 3h, current state unknown, last acting [] pg 7.21b is down, acting [16] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull pg 7.20 is active+undersized+degraded+remapped+backfill_toofull, acting [3] pg 19.2b is active+remapped+backfill_toofull, acting [12,3] pg 19.6b is active+remapped+backfill_toofull, acting [12,3] pg 20.55 is active+remapped+backfill_toofull, acting [29,3] pg 24.6 is active+undersized+degraded+remapped+backfill_toofull, acting [22] pg 24.b is active+undersized+degraded+remapped+backfill_toofull, acting [21] pg 24.13 is active+undersized+degraded+remapped+backfill_toofull, acting [23] pg 24.16 is active+undersized+degraded+remapped+backfill_toofull, acting [21] pg 24.1d is active+undersized+degraded+remapped+backfill_toofull, acting [21] [WRN] PG_DEGRADED: Degraded data redundancy: 432856/3408880 objects degraded (12.698%), 191 pgs degraded, 181 pgs undersized pg 7.3f is stuck undersized for 9m, current state active+undersized+degraded, last acting [16] pg 7.56 is stuck undersized for 21m, current state active+undersized+degraded, last acting [13] pg 7.61 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.66 is stuck undersized for 7m, current state active+undersized+degraded, last acting [29] pg 7.6b is stuck undersized for 25m, current state active+undersized+degraded, last acting [21] pg 7.102 is active+undersized+degraded, acting [23] pg 7.118 is stuck undersized for 14m, current state active+undersized+degraded, last acting [23] pg 7.11c is stuck undersized for 9m, current state active+undersized+degraded, last acting [3] pg 7.133 is stuck undersized for 7m, current state active+undersized+degraded, last acting [12] pg 7.139 is stuck undersized for 7m, current state active+undersized+degraded, last acting [13] pg 7.143 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.155 is stuck undersized for 25m, current state active+undersized+degraded, last acting [3] pg 7.156 is active+undersized+degraded, acting [12] pg 7.15e is stuck undersized for 31m, current state active+undersized+degraded, last acting [3] pg 7.15f is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.168 is stuck undersized for 67m, current state active+undersized+degraded, last acting [22] pg 7.17f is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.180 is stuck undersized for 14m, current state active+undersized+degraded, last acting [21] pg 7.18e is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.193 is active+undersized+degraded, acting [16] pg 7.197 is stuck undersized for 14m, current state active+undersized+degraded, last acting [21] pg 7.1a6 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.1b7 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.1c6 is stuck undersized for 14m, current state active+undersized+degraded, last acting [22] pg 7.1ca is stuck undersized for 14m, current state active+undersized+degraded, last acting [22] pg 7.1d7 is stuck undersized for 9m, current state active+undersized+degraded, last acting [22] pg 7.1df is active+undersized+degraded, acting [21] pg 7.1e6 is stuck undersized for 10h, current state active+undersized+degraded, last acting [23] pg 7.200 is active+undersized+degraded, acting [29] pg 7.202 is stuck undersized for 7m, current state active+undersized+degraded, last acting [13] pg 7.20c is stuck undersized for 10h, current state active+undersized+degraded, last acting [16] pg 7.20e is stuck undersized for 47m, current state active+undersized+degraded, last acting [23] pg 7.20f is stuck undersized for 47m, current state active+undersized+degraded, last acting [23] pg 7.217 is stuck undersized for 7m, current state active+undersized+degraded, last acting [21] pg 15.35 is active+undersized+degraded, acting [22] pg 16.2a is stuck undersized for 10h, current state active+undersized+degraded, last acting [21] pg 19.43 is stuck undersized for 31m, current state active+undersized+degraded, last acting [23] pg 19.44 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 19.4e is stuck undersized for 14m, current state active+undersized+degraded, last acting [16] pg 19.52 is active+undersized+degraded+wait, acting [3] pg 19.55 is stuck undersized for 25m, current state active+undersized+degraded, last acting [23] pg 19.61 is stuck undersized for 25m, current state active+undersized+degraded, last acting [21] pg 19.72 is stuck undersized for 31m, current state active+undersized+degraded, last acting [3] pg 20.42 is stuck undersized for 7m, current state active+undersized+degraded, last acting [23] pg 20.48 is stuck undersized for 67m, current state active+undersized+degraded, last acting [16] pg 20.5b is stuck undersized for 21m, current state active+undersized+degraded, last acting [12] pg 20.5f is stuck undersized for 10h, current state active+undersized+degraded, last acting [12] pg 20.65 is stuck undersized for 10m, current state active+undersized+degraded, last acting [23] pg 20.6a is active+undersized+degraded, acting [13] pg 20.71 is stuck undersized for 31m, current state active+undersized+degraded, last acting [13] pg 20.7d is stuck undersized for 7m, current state active+undersized+degraded, last acting [29] [WRN] POOL_NEARFULL: 12 pool(s) nearfull pool '.mgr' is nearfull pool 'DataStore' is nearfull pool 'cephfs_data' is nearfull pool 'cephfs_metadata' is nearfull pool 'OS' is nearfull pool 'cloud' is nearfull pool 'DataStore_2' is nearfull pool 'DataStore_3' is nearfull pool 'MGMT' is nearfull pool 'DataStore_4' is nearfull pool 'DataStore_5' is nearfull pool 'fast' is nearfull [WRN] SLOW_OPS: 255 slow ops, oldest one blocked for 2417 sec, daemons [osd.10,osd.12,osd.21,osd.22,osd.23] have slow ops. root@cmt6770:~# ceph -s cluster: id: 9319dafb-3408-46cb-9b09-b3d381114545 health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs mon cmt6770 is low on available space 9 osds down 1 host (3 osds) down 5 nearfull osd(s) Reduced data availability: 866 pgs inactive, 489 pgs down, 5 pgs incomplete, 60 pgs stale Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull Degraded data redundancy: 432856/3408880 objects degraded (12.698%), 191 pgs degraded, 181 pgs undersized 12 pool(s) nearfull 255 slow ops, oldest one blocked for 2422 sec, daemons [osd.10,osd.12,osd.21,osd.22,osd.23] have slow ops. services: mon: 2 daemons, quorum cmt6770,cmt5923 (age 70m) mgr: cmt6770(active, since 3h) mds: 1/1 daemons up, 1 standby osd: 25 osds: 11 up (since 14s), 20 in (since 9m); 182 remapped pgs data: volumes: 0/1 healthy, 1 recovering pools: 12 pools, 1589 pgs objects: 1.70M objects, 6.2 TiB usage: 9.1 TiB used, 6.2 TiB / 15 TiB avail pgs: 28.760% pgs unknown 34.991% pgs not active 432856/3408880 objects degraded (12.698%) 388136/3408880 objects misplaced (11.386%) 466 down 457 unknown 209 active+clean 185 active+undersized+degraded 157 active+clean+remapped 62 stale 20 stale+down 9 active+undersized+remapped 6 active+undersized+degraded+remapped+backfill_toofull 5 incomplete 3 active+remapped+backfill_toofull 2 active+clean+scrubbing+deep 2 active+clean+remapped+scrubbing+deep 2 down+remapped 1 stale+creating+down 1 active+remapped+backfilling 1 active+remapped+backfill_wait 1 active+undersized+remapped+wait io: recovery: 12 MiB/s, 3 objects/s root@cmt6770:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 34.05125 root default -3 5.38539 host cmt5923 3 ssd 1.74660 osd.3 up 0.79999 1.00000 8 ssd 1.81940 osd.8 down 1.00000 1.00000 9 ssd 1.81940 osd.9 down 1.00000 1.00000 -15 4.40289 host cmt6461 24 nvme 0.90970 osd.24 down 0.79999 1.00000 2 ssd 1.74660 osd.2 down 1.00000 1.00000 17 ssd 1.74660 osd.17 down 1.00000 1.00000 -5 5.35616 host cmt6770 0 ssd 0.87329 osd.0 down 1.00000 1.00000 1 ssd 0.87329 osd.1 down 1.00000 1.00000 4 ssd 1.86299 osd.4 down 0 1.00000 14 ssd 0.87329 osd.14 down 1.00000 1.00000 15 ssd 0.87329 osd.15 up 1.00000 1.00000 -9 7.24838 host cmt7773 5 nvme 1.81940 osd.5 down 0 1.00000 19 nvme 1.81940 osd.19 down 0 1.00000 7 ssd 1.74660 osd.7 down 1.00000 1.00000 29 ssd 1.86299 osd.29 up 1.00000 1.00000 -13 7.93245 host dc2943 22 nvme 0.90970 osd.22 up 1.00000 1.00000 23 nvme 0.90970 osd.23 up 1.00000 1.00000 6 ssd 1.74660 osd.6 down 0 1.00000 10 ssd 0.87329 osd.10 up 1.00000 1.00000 11 ssd 0.87329 osd.11 up 1.00000 1.00000 12 ssd 0.87329 osd.12 up 1.00000 1.00000 13 ssd 0.87329 osd.13 up 1.00000 1.00000 16 ssd 0.87329 osd.16 up 0.79999 1.00000 -11 3.72598 host dc3658 20 nvme 1.86299 osd.20 down 0 1.00000 21 nvme 1.86299 osd.21 up 0.90002 1.00000 root@cmt6770:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 3 ssd 1.74660 0.79999 1.7 TiB 1.5 TiB 1.5 TiB 228 KiB 2.9 GiB 251 GiB 85.96 1.44 214 up 8 ssd 1.81940 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 9 ssd 1.81940 1.00000 1.8 TiB 776 MiB 745 MiB 8 KiB 31 MiB 1.8 TiB 0.04 0 46 down 24 nvme 0.90970 0.79999 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 2 ssd 1.74660 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 4 down 17 ssd 1.74660 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 0 ssd 0.87329 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 3 down 1 ssd 0.87329 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 4 ssd 1.86299 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 14 ssd 0.87329 1.00000 894 GiB 792 MiB 752 MiB 14 KiB 40 MiB 893 GiB 0.09 0.00 25 down 15 ssd 0.87329 1.00000 894 GiB 232 GiB 231 GiB 14 KiB 1.4 GiB 662 GiB 25.98 0.44 83 up 5 nvme 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 19 nvme 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 7 ssd 1.74660 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 29 ssd 1.86299 1.00000 1.9 TiB 1.5 TiB 1.5 TiB 323 KiB 2.8 GiB 354 GiB 81.44 1.37 222 up 22 nvme 0.90970 1.00000 932 GiB 689 GiB 687 GiB 181 KiB 1.6 GiB 243 GiB 73.96 1.24 139 up 23 nvme 0.90970 1.00000 932 GiB 820 GiB 818 GiB 138 KiB 2.0 GiB 112 GiB 87.98 1.48 144 up 6 ssd 1.74660 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 10 ssd 0.87329 1.00000 894 GiB 237 GiB 235 GiB 1 KiB 1.2 GiB 658 GiB 26.46 0.44 82 up 11 ssd 0.87329 1.00000 894 GiB 264 GiB 263 GiB 1 KiB 1.4 GiB 630 GiB 29.54 0.50 67 up 12 ssd 0.87329 1.00000 894 GiB 780 GiB 778 GiB 123 KiB 1.8 GiB 114 GiB 87.26 1.46 113 up 13 ssd 0.87329 1.00000 894 GiB 684 GiB 682 GiB 170 KiB 1.9 GiB 210 GiB 76.48 1.28 98 up 16 ssd 0.87329 0.79999 894 GiB 779 GiB 777 GiB 149 KiB 1.8 GiB 116 GiB 87.06 1.46 86 up 20 nvme 1.86299 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 21 nvme 1.86299 0.90002 1.9 TiB 1.7 TiB 1.7 TiB 430 KiB 3.5 GiB 194 GiB 89.84 1.51 314 up TOTAL 15 TiB 9.1 TiB 9.1 TiB 1.7 MiB 22 GiB 6.2 TiB 59.60 MIN/MAX VAR: 0/1.51 STDDEV: 44.89 also osd start logs: May 10 15:38:32 cmt5923 systemd[1]: ceph-osd(a)9.service: Failed with result 'signal'. May 10 15:38:37 cmt5923 ceph-osd[2383902]: 2025-05-10T15:38:37.579+0300 764caf13f880 -1 osd.8 100504 log_to_monitors true May 10 15:38:37 cmt5923 ceph-osd[2383902]: 2025-05-10T15:38:37.791+0300 764c8c64b6c0 -1 log_channel(cluster) log [ERR] : 7.26a past_intervals [96946,100253) start interval does not contain the required bound [93903,100253) start May 10 15:38:37 cmt5923 ceph-osd[2383902]: 2025-05-10T15:38:37.791+0300 764c8c64b6c0 -1 osd.8 pg_epoch: 100377 pg[7.26a( empty local-lis/les=0/0 n=0 ec=96946/96946 lis/c=96236/93898 les/c/f=96237/93903/91308 sis=100253) [3,1] r=-1 lpr=100376 pi=[96946,100253)/3 crt=0'0 mlcod 0'0 unknown mbc={}] PeeringState::check_past_interval_bounds 7.26a past_intervals [96946,100253) start interval does not contain the required bound [93903,100253) start We appreciate any support and guidance, Thanks in advance

5 6

Ceph Day Seattle is almost here!
by Anthony D'Atri 11 May '25

11 May '25

There’s still time to register to attend this community event sponsored by OSNexus and Clyso. Seattle, Washington USA May 15 2025 The schedule and registration link can be found here: https://ceph.io/en/community/events/2025/ceph-day-seattle/ Ceph Days Seattle 2025 ceph.io https://ceph.io/en/community/events/2025/ceph-day-seattle/

1 0

failing to enable disk failure prediction
by Jens Galsgaard 10 May '25

10 May '25

Hello, I’ve a cluster built with cephadm running on Debian 12/Bookworm. Ceph 18.2.5. I want to enable disk failure prediction and run this command: ceph mgr module enable diskprediction_local Then the cluster goes into ERROR state and the logs shows: 2025-04-14T08:55:37.073118+0200 mgr.host01.eqvsde [ERR] Unhandled exception from module 'diskprediction_local' while running on mgr.host01.eqvsde: No module named 'sklearn.svm.classes' 2025-04-14T08:55:38.388289+0200 mon.host03 [ERR] Health check failed: Module 'diskprediction_local' has failed: No module named 'sklearn.svm.classes' (MGR_MODULE_ERROR) How to add sklearn to the container as it is obviously missing? Venlig hilsen - Mit freundlichen Grüßen - Kind Regards, Jens Galsgaard Gitservice.dk v/ Jens Galsgaard Sparresmindevej 7 4780 Stege Danmark Tlf: +45 46551100 Mob: +45 28864340 CVR/VAT: DK30128540

2 5

Full OSD with almost empty pools
by Adam Prycki 10 May '25

10 May '25

Hello, we have just discovered new issue with our cephfs cluster. All of our SSD osd have been flagged as nearful. Ceph df on our cluster looks very wrong. Raw storage usage shows that SSDs are almost full, but SSD pools are hardly using any space. ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 27 PiB 16 PiB 11 PiB 11 PiB 40.57 ssd 71 TiB 10 TiB 60 TiB 60 TiB 85.49 TOTAL 27 PiB 16 PiB 11 PiB 11 PiB 40.68 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 16 47 GiB 12.12k 142 GiB 2.48 1.8 TiB cephfs.arm-vol.meta 2 4096 19 GiB 2.02M 57 GiB 1.01 1.8 TiB cephfs.arm-vol.data 4 16384 7.7 PiB 2.14G 10 PiB 58.29 5.5 PiB .rgw.root 24 32 16 KiB 24 264 KiB 0 2.4 PiB bst-ro-backup.rgw.log 25 32 343 MiB 2.90k 1.0 GiB 0 2.4 PiB bst-ro-backup.rgw.control 26 32 0 B 8 0 B 0 2.4 PiB bst-ro-backup.rgw.meta 27 32 198 KiB 440 5.0 MiB 0 2.4 PiB .nfs 32 32 8.2 KiB 10 99 KiB 0 2.4 PiB cephfs.arm-vol.meta pool seems to store 19GiB of data. rados -p cephfs.arm-vol.meta | wc -l confirms that there are about 2.02M objects in that pool. Our SSD OSDs are almost full ceph osd df ssd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 807 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 395 MiB 1.3 GiB 64 GiB 85.78 1.00 85 up 812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.0 GiB 66 GiB 85.23 1.00 74 up 569 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.0 GiB 66 GiB 85.24 1.00 72 up 577 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 334 MiB 1.5 GiB 67 GiB 85.12 1.00 72 up 571 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 369 MiB 2.3 GiB 66 GiB 85.30 1.00 86 up 578 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 329 MiB 1.8 GiB 66 GiB 85.19 1.00 68 up 1775 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 379 MiB 1.2 GiB 64 GiB 85.70 1.00 76 up 1838 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 2.3 GiB 66 GiB 85.29 1.00 74 up 1783 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 443 MiB 2.5 GiB 62 GiB 86.05 1.01 98 up 1846 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.7 GiB 66 GiB 85.18 1.00 85 up 1854 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 384 MiB 2.4 GiB 66 GiB 85.33 1.00 80 up 1855 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.6 GiB 65 GiB 85.37 1.00 78 up 1761 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 368 MiB 1.7 GiB 63 GiB 85.86 1.00 79 up 1826 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 331 MiB 2.2 GiB 66 GiB 85.29 1.00 73 up 1773 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 324 MiB 2.0 GiB 66 GiB 85.24 1.00 73 up 1836 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 390 MiB 2.3 GiB 66 GiB 85.30 1.00 82 up 1771 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB 66 GiB 85.31 1.00 75 up 1834 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 376 MiB 2.1 GiB 66 GiB 85.27 1.00 79 up 2111 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 396 MiB 2.0 GiB 63 GiB 85.90 1.01 84 up 2112 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 338 MiB 2.2 GiB 66 GiB 85.27 1.00 77 up 2272 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 374 MiB 2.0 GiB 66 GiB 85.24 1.00 76 up 2279 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 332 MiB 2.5 GiB 66 GiB 85.34 1.00 77 up 2276 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 360 MiB 2.4 GiB 66 GiB 85.34 1.00 82 up 2285 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 371 MiB 2.2 GiB 66 GiB 85.27 1.00 79 up 2275 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 368 MiB 2.3 GiB 66 GiB 85.31 1.00 79 up 2282 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.0 GiB 66 GiB 85.23 1.00 75 up 2271 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 1.6 GiB 66 GiB 85.15 1.00 81 up 2278 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB 63 GiB 85.85 1.00 77 up 2277 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 334 MiB 1.3 GiB 64 GiB 85.70 1.00 72 up 2286 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 351 MiB 2.3 GiB 66 GiB 85.30 1.00 74 up 2347 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 362 MiB 2.6 GiB 65 GiB 85.38 1.00 81 up 2348 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.0 GiB 66 GiB 85.23 1.00 76 up 573 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 404 MiB 2.7 GiB 65 GiB 85.40 1.00 85 up 580 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 320 MiB 2.3 GiB 60 GiB 86.65 1.01 70 up 169 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.0 GiB 66 GiB 85.24 1.00 88 up 171 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 323 MiB 2.5 GiB 66 GiB 85.34 1.00 71 up 1216 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.5 GiB 66 GiB 85.35 1.00 76 up 1218 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 334 MiB 1.9 GiB 63 GiB 85.84 1.00 70 up 811 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 377 MiB 2.0 GiB 66 GiB 85.24 1.00 80 up 815 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 419 MiB 2.6 GiB 65 GiB 85.38 1.00 91 up 810 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.2 GiB 66 GiB 85.27 1.00 67 up 813 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 340 MiB 2.4 GiB 63 GiB 85.93 1.01 72 up 1760 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 388 MiB 974 MiB 67 GiB 85.01 0.99 77 up 1823 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 366 MiB 2.5 GiB 66 GiB 85.35 1.00 73 up 1762 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 361 MiB 1.4 GiB 67 GiB 85.10 1.00 75 up 1824 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 278 MiB 2.6 GiB 62 GiB 86.06 1.01 65 up 1766 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 377 MiB 1.9 GiB 60 GiB 86.58 1.01 82 up 1830 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 339 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1755 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 388 MiB 2.5 GiB 65 GiB 85.36 1.00 77 up 1818 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB 66 GiB 85.31 1.00 75 up 1774 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 2.2 GiB 66 GiB 85.26 1.00 67 up 1837 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 311 MiB 2.5 GiB 66 GiB 85.33 1.00 68 up 1748 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.4 GiB 66 GiB 85.31 1.00 76 up 1812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.2 GiB 66 GiB 85.28 1.00 77 up 1337 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.3 GiB 66 GiB 85.30 1.00 64 up 1338 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 331 MiB 1.9 GiB 63 GiB 85.89 1.00 76 up 1091 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 329 MiB 2.5 GiB 66 GiB 85.34 1.00 69 up 1096 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 373 MiB 1.6 GiB 63 GiB 85.85 1.00 82 up 1093 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 379 MiB 2.6 GiB 65 GiB 85.38 1.00 84 up 1098 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.3 GiB 66 GiB 85.30 1.00 79 up 1752 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 350 MiB 1.5 GiB 67 GiB 85.11 1.00 80 up 1815 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB 66 GiB 85.30 1.00 70 up 1754 ssd 0.43660 1.00000 447 GiB 387 GiB 384 GiB 362 MiB 2.0 GiB 60 GiB 86.48 1.01 75 up 1816 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 346 MiB 2.5 GiB 66 GiB 85.35 1.00 72 up 1753 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.6 GiB 64 GiB 85.80 1.00 75 up 1817 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB 66 GiB 85.33 1.00 75 up 1739 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.3 GiB 67 GiB 85.09 1.00 75 up 1804 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 365 MiB 2.6 GiB 62 GiB 86.03 1.01 78 up 1763 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 369 MiB 2.6 GiB 62 GiB 86.04 1.01 76 up 1825 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 379 MiB 2.2 GiB 63 GiB 85.91 1.01 83 up 1738 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 354 MiB 1.9 GiB 63 GiB 85.84 1.00 75 up 1802 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 339 MiB 1.6 GiB 63 GiB 85.83 1.00 77 up 1661 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 320 MiB 2.4 GiB 66 GiB 85.33 1.00 65 up 1666 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.3 GiB 66 GiB 85.31 1.00 80 up 1663 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 444 MiB 1.9 GiB 63 GiB 85.91 1.01 98 up 1669 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 375 MiB 2.2 GiB 66 GiB 85.29 1.00 76 up 1665 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 341 MiB 2.1 GiB 66 GiB 85.26 1.00 75 up 1671 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 363 MiB 2.4 GiB 66 GiB 85.33 1.00 76 up 1726 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB 64 GiB 85.78 1.00 77 up 1790 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB 66 GiB 85.29 1.00 80 up 1765 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 437 MiB 2.5 GiB 62 GiB 86.07 1.01 91 up 1828 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 356 MiB 2.3 GiB 66 GiB 85.31 1.00 78 up 1744 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 355 MiB 2.5 GiB 62 GiB 86.03 1.01 77 up 1807 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 327 MiB 1.9 GiB 60 GiB 86.51 1.01 74 up 1768 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.2 GiB 66 GiB 85.29 1.00 88 up 1831 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.0 GiB 66 GiB 85.23 1.00 71 up 1756 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 339 MiB 1.3 GiB 67 GiB 85.07 1.00 65 up 1819 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 323 MiB 2.3 GiB 66 GiB 85.28 1.00 67 up 1772 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 407 MiB 1.8 GiB 66 GiB 85.21 1.00 87 up 1835 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 1.5 GiB 67 GiB 85.11 1.00 74 up 1879 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 342 MiB 2.5 GiB 66 GiB 85.34 1.00 70 up 1882 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 310 MiB 2.2 GiB 66 GiB 85.28 1.00 66 up 1880 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 344 MiB 2.6 GiB 65 GiB 85.36 1.00 79 up 1883 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 319 MiB 2.6 GiB 65 GiB 85.36 1.00 65 up 1881 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 313 MiB 2.1 GiB 66 GiB 85.25 1.00 64 up 1884 ssd 0.43660 1.00000 447 GiB 390 GiB 388 GiB 387 MiB 2.1 GiB 57 GiB 87.27 1.02 87 up 1747 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 426 MiB 2.6 GiB 65 GiB 85.38 1.00 87 up 1809 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 359 MiB 1.6 GiB 66 GiB 85.14 1.00 77 up 1751 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 340 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1813 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 367 MiB 2.2 GiB 63 GiB 85.93 1.01 81 up 1785 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 403 MiB 2.5 GiB 66 GiB 85.35 1.00 81 up 1848 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.5 GiB 66 GiB 85.34 1.00 74 up 1749 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.5 GiB 67 GiB 85.12 1.00 74 up 1811 ssd 0.43660 1.00000 447 GiB 388 GiB 385 GiB 335 MiB 2.5 GiB 59 GiB 86.70 1.01 74 up 1725 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 361 MiB 2.5 GiB 65 GiB 85.35 1.00 73 up 1788 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1779 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.1 GiB 67 GiB 85.03 0.99 77 up 1843 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 387 MiB 2.4 GiB 66 GiB 85.34 1.00 78 up 1967 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.2 GiB 66 GiB 85.28 1.00 66 up 1970 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.9 GiB 66 GiB 85.21 1.00 78 up 1968 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.9 GiB 66 GiB 85.22 1.00 71 up 1971 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 354 MiB 2.1 GiB 66 GiB 85.26 1.00 79 up 1966 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 352 MiB 1.8 GiB 64 GiB 85.79 1.00 82 up 1969 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 315 MiB 2.3 GiB 63 GiB 85.99 1.01 73 up 1781 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 381 MiB 1.8 GiB 66 GiB 85.20 1.00 81 up 1844 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 330 MiB 2.1 GiB 66 GiB 85.25 1.00 71 up 1743 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 360 MiB 2.1 GiB 63 GiB 85.95 1.01 79 up 1806 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 385 MiB 2.4 GiB 66 GiB 85.34 1.00 78 up 1778 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 354 MiB 2.5 GiB 66 GiB 85.35 1.00 74 up 1839 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 348 MiB 1.6 GiB 66 GiB 85.15 1.00 79 up 1757 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 409 MiB 2.5 GiB 65 GiB 85.36 1.00 82 up 1820 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.1 GiB 66 GiB 85.26 1.00 73 up 1758 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 403 MiB 2.5 GiB 63 GiB 86.02 1.01 81 up 1821 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.1 GiB 66 GiB 85.27 1.00 80 up 1787 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 281 MiB 2.1 GiB 66 GiB 85.24 1.00 64 up 1850 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 260 MiB 2.6 GiB 65 GiB 85.36 1.00 59 up 2054 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 318 MiB 1.6 GiB 66 GiB 85.14 1.00 68 up 2057 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.3 GiB 66 GiB 85.31 1.00 78 up 2055 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 2.1 GiB 66 GiB 85.26 1.00 79 up 2058 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 289 MiB 2.5 GiB 63 GiB 85.99 1.01 65 up 2053 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.0 GiB 66 GiB 85.23 1.00 74 up 2056 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 362 MiB 2.3 GiB 63 GiB 85.93 1.01 77 up 1735 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 363 MiB 1.8 GiB 66 GiB 85.20 1.00 76 up 1799 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 360 MiB 2.1 GiB 66 GiB 85.27 1.00 74 up 1784 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 361 MiB 1.8 GiB 66 GiB 85.19 1.00 74 up 1847 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.4 GiB 66 GiB 85.33 1.00 74 up 1740 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 368 MiB 2.4 GiB 66 GiB 85.33 1.00 72 up 1801 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 362 MiB 2.1 GiB 66 GiB 85.26 1.00 74 up 1776 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.1 GiB 66 GiB 85.25 1.00 80 up 1841 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 393 MiB 2.1 GiB 66 GiB 85.26 1.00 86 up 1742 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 1.6 GiB 66 GiB 85.14 1.00 74 up 1805 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 330 MiB 2.6 GiB 62 GiB 86.05 1.01 75 up 1764 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.4 GiB 66 GiB 85.32 1.00 74 up 1827 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 434 MiB 2.3 GiB 63 GiB 86.00 1.01 88 up 2301 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 355 MiB 2.3 GiB 66 GiB 85.30 1.00 79 up 2309 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 1.7 GiB 66 GiB 85.17 1.00 69 up 2302 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 314 MiB 2.5 GiB 66 GiB 85.33 1.00 68 up 2315 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.3 GiB 66 GiB 85.30 1.00 70 up 2308 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB 66 GiB 85.33 1.00 79 up 2316 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 366 MiB 2.3 GiB 66 GiB 85.29 1.00 76 up 1733 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.9 GiB 63 GiB 85.86 1.00 74 up 1796 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB 65 GiB 85.38 1.00 74 up 1782 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 313 MiB 1.4 GiB 64 GiB 85.77 1.00 70 up 1845 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 379 MiB 2.2 GiB 66 GiB 85.29 1.00 81 up 1730 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.2 GiB 66 GiB 85.29 1.00 74 up 1794 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 377 MiB 2.1 GiB 63 GiB 85.94 1.01 80 up 1786 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 381 MiB 2.6 GiB 65 GiB 85.37 1.00 76 up 1849 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 337 MiB 963 MiB 64 GiB 85.63 1.00 76 up 1741 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 358 MiB 1.4 GiB 67 GiB 85.09 1.00 75 up 1803 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 383 MiB 2.7 GiB 65 GiB 85.40 1.00 78 up 1750 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 353 MiB 2.1 GiB 66 GiB 85.26 1.00 70 up 1814 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 351 MiB 772 MiB 67 GiB 84.95 0.99 81 up Both ssd pools run with basic 3x replication and rack failure domain. cephfs.arm-vol.data pool is EC pool on HDD OSDs. I've checked few OSDs and pools are not overlapping over. Cluster runs on 19.2.0 and OSD nodes are ARM64. In emergency I'm planning to just move these pool to HDDs OSD. Do you have any idea what could have filled these SSDs and how to clean it up? Today I've Best regards Adam Prycki

7 20

Network traffic with failure domain datacenter
by Torkil Svensgaard 10 May '25

10 May '25

Hi We are looking at a cluster split between two DCs with the DCs as failure domains. Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

7 13

Re: *** Spam *** Re: v18.2.7 Reef released
by Erwin Bogaard 09 May '25

09 May '25

Yeah, In think this is a difference between Centos 9 Stream and Almalinux 9 (which we use): openssl-libs-3.5.0 is only available for Centos 9 Stream, not for Almalinux. So we probably need to wait a little bit longer until this dependency is available at Almalinux we can install 18.2.7.

3 2

v18.2.7 Reef released
by Yuri Weinstein 09 May '25

09 May '25

We're happy to announce the seventh backport (hotfix) release in the Reef series. https://ceph.io/en/news/blog/2025/v18-2-7-reef-released/ Notable Changes This release fixes a critical bluestore regression in versions 18.2.5 and 18.2.6 (https://github.com/ceph/ceph/pull/61653) Users running either of those releases are recommended to upgrade at the earliest convenience. This release also includes several other important BlueStore fixes: * https://github.com/ceph/ceph/pull/62840 * https://github.com/ceph/ceph/pull/62054 * https://github.com/ceph/ceph/pull/62152 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad

3 2

check bucket index on large bucket leads to laggy index PGs
by Boris 09 May '25

09 May '25

Hi, I have a bucket that got >20m index entries but only got 11 shards. When I try to run a radosgw-admin bucket check the PGs that hold the index start to become laggy after a couple of seconds. I need to stop it because it kills the whole object storage. This is a latest reef cluster and the master of a multisite which only replicates the metadata (1 realm, multiple zonegroups, one zone per zonegroup). Any ideas what I can do? I fear to reshard the bucket, because I am not sure if I can stop the resharding if the PGs become laggy. Cheers Boris

1 0