Cluster Not recovering after multiple disk failure
Hello all Need urgent help on below… I tried reducing min_size but still showing same… cluster: id: 15688cb4-044a-11ec-942e-516035adea04 health: HEALTH_ERR 3 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 20/16670718 objects unfound (0.000%) Reduced data availability: 283 pgs inactive, 464 pgs incomplete Possible data damage: 2 pgs recovery_unfound Degraded data redundancy: 42998/61175329 objects degraded (0.070%), 2 pgs degraded, 1 pg undersized 304 pgs not deep-scrubbed in time 1055 slow ops, oldest one blocked for 94042 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops. services: mon: 5 daemons, quorum van2-converged04n,van2-converged05n,van2-converged01n,van2-converged03n,van2-converged02n (age 17h) mgr: van2-converged05n.lybhho(active, since 22h), standbys: van2-converged04n.azmqik mds: 3/3 daemons up, 11 standby osd: 31 osds: 31 up (since 5h), 31 in (since 5h); 2 remapped pgs rbd-mirror: 2 daemons active (2 hosts) data: volumes: 1/2 healthy, 1 recovering pools: 32 pools, 1505 pgs objects: 16.67M objects, 56 TiB usage: 152 TiB used, 234 TiB / 387 TiB avail pgs: 0.332% pgs unknown 30.897% pgs not active 42998/61175329 objects degraded (0.070%) 21469/61175329 objects misplaced (0.035%) 20/16670718 objects unfound (0.000%) 1034 active+clean 463 incomplete 5 unknown 1 remapped+incomplete 1 active+recovery_unfound+degraded 1 recovery_unfound+undersized+degraded+remapped+peered progress: Global Recovery Event (22h) [===================.........] (remaining: 10h) [WRN] SLOW_OPS: 1055 slow ops, oldest one blocked for 93877 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops. [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.van2.van2-converged05n.mbbzfj(mds.1): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 62279 secs [WRN] OBJECT_UNFOUND: 20/16670718 objects unfound (0.000%) pg 28.1f has 9 unfound objects pg 29.2 has 11 unfound objects [WRN] PG_AVAILABILITY: Reduced data availability: 470 pgs inactive, 464 pgs incomplete pg 16.58 is incomplete, acting [7,14,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5c is incomplete, acting [26,3,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5f is incomplete, acting [13,6,21] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.63 is incomplete, acting [9,7,15] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.64 is incomplete, acting [18,19,5] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.68 is incomplete, acting [2,5,27] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.70 is incomplete, acting [19,1,24] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.74 is incomplete, acting [8,25,14] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.78 is incomplete, acting [23,14,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7c is incomplete, acting [24,10,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7d is incomplete, acting [12,22,13] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.64 is incomplete, acting [20,11,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.66 is incomplete, acting [21,13,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6a is incomplete, acting [4,26,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6c is incomplete, acting [13,12,31] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6e is incomplete, acting [8,31,24] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.7a is incomplete, acting [8,28,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 19.54 is stuck inactive since forever, current state incomplete, last acting [10,4,20,15,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.58 is incomplete, acting [15,18,12,28,20] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.59 is incomplete, acting [6,9,18,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5a is incomplete, acting [11,31,20,17,24] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for ‘incomplceph -s Ceph Any way to make these scrubs complete faster? pg 19.63 not deep-scrubbed since 2025-05-24T08:18:29.738427+0000 pg 34.6d not deep-scrubbed since 2025-06-04T08:33:54.534882+0000 pg 16.5f not deep-scrubbed since 2025-05-21T10:44:09.996254+0000 pg 19.5d not deep-scrubbed since 2025-05-26T09:36:27.064154+0000 pg 19.5e not deep-scrubbed since 2025-06-05T00:52:03.859984+0000 pg 16.5c not deep-scrubbed since 2025-06-06T00:36:22.021390+0000 pg 19.5f not deep-scrubbed since 2025-06-03T16:27:42.356213+0000 pg 22.5a not deep-scrubbed since 2025-06-05T23:00:28.066065+0000 pg 34.69 not deep-scrubbed since 2025-06-03T05:07:58.209808+0000 pg 19.58 not deep-scrubbed since 2025-05-27T23:32:29.963976+0000 pg 19.59 not deep-scrubbed since 2025-05-25T11:50:44.735318+0000 pg 19.5a not deep-scrubbed since 2025-06-06T02:34:05.486126+0000 pg 16.58 not deep-scrubbed since 2025-05-13T14:10:44.570493+0000 Regards Dev
On Jun 18, 2025, at 5:03 PM, Devender Singh <devender@netskrt.io> wrote:
Hello all
Need urgent help on below… I tried reducing min_size but still showing same…
cluster: id: 15688cb4-044a-11ec-942e-516035adea04 health: HEALTH_ERR 3 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 20/16670718 objects unfound (0.000%)
Did you suffer a power outage or something? What are your OSDs? HDD? SSD? If SSD, are they *enterprise* not client?
Reduced data availability: 283 pgs inactive, 464 pgs incomplete Possible data damage: 2 pgs recovery_unfound Degraded data redundancy: 42998/61175329 objects degraded (0.070%), 2 pgs degraded, 1 pg undersized 304 pgs not deep-scrubbed in time 1055 slow ops, oldest one blocked for 94042 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops.
Look for a pattern. Are these all on the same host? Send `ceph osd tree` In the meantime run `ceph osd down 1`, wait for a couple of minutes for recovery, and see if that improves the numbers. If it does, repeat with the other OSDs above, waiting for the PGs to peer and all 31 OSDs to show up before proceeding to the next.
services: mon: 5 daemons, quorum van2-converged04n,van2-converged05n,van2-converged01n,van2-converged03n,van2-converged02n (age 17h) mgr: van2-converged05n.lybhho(active, since 22h), standbys: van2-converged04n.azmqik mds: 3/3 daemons up, 11 standby osd: 31 osds: 31 up (since 5h), 31 in (since 5h); 2 remapped pgs rbd-mirror: 2 daemons active (2 hosts)
data: volumes: 1/2 healthy, 1 recovering pools: 32 pools, 1505 pgs objects: 16.67M objects, 56 TiB usage: 152 TiB used, 234 TiB / 387 TiB avail pgs: 0.332% pgs unknown 30.897% pgs not active 42998/61175329 objects degraded (0.070%) 21469/61175329 objects misplaced (0.035%) 20/16670718 objects unfound (0.000%) 1034 active+clean 463 incomplete 5 unknown 1 remapped+incomplete 1 active+recovery_unfound+degraded 1 recovery_unfound+undersized+degraded+remapped+peered
progress: Global Recovery Event (22h) [===================.........] (remaining: 10h)
[WRN] SLOW_OPS: 1055 slow ops, oldest one blocked for 93877 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops. [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.van2.van2-converged05n.mbbzfj(mds.1): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 62279 secs [WRN] OBJECT_UNFOUND: 20/16670718 objects unfound (0.000%) pg 28.1f has 9 unfound objects pg 29.2 has 11 unfound objects [WRN] PG_AVAILABILITY: Reduced data availability: 470 pgs inactive, 464 pgs incomplete pg 16.58 is incomplete, acting [7,14,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5c is incomplete, acting [26,3,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5f is incomplete, acting [13,6,21] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.63 is incomplete, acting [9,7,15] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.64 is incomplete, acting [18,19,5] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.68 is incomplete, acting [2,5,27] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.70 is incomplete, acting [19,1,24] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.74 is incomplete, acting [8,25,14] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.78 is incomplete, acting [23,14,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7c is incomplete, acting [24,10,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7d is incomplete, acting [12,22,13] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.64 is incomplete, acting [20,11,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.66 is incomplete, acting [21,13,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6a is incomplete, acting [4,26,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6c is incomplete, acting [13,12,31] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6e is incomplete, acting [8,31,24] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.7a is incomplete, acting [8,28,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 19.54 is stuck inactive since forever, current state incomplete, last acting [10,4,20,15,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.58 is incomplete, acting [15,18,12,28,20] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.59 is incomplete, acting [6,9,18,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5a is incomplete, acting [11,31,20,17,24] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for ‘incomplceph -s Ceph
Any way to make these scrubs complete faster?
pg 19.63 not deep-scrubbed since 2025-05-24T08:18:29.738427+0000 pg 34.6d not deep-scrubbed since 2025-06-04T08:33:54.534882+0000 pg 16.5f not deep-scrubbed since 2025-05-21T10:44:09.996254+0000 pg 19.5d not deep-scrubbed since 2025-05-26T09:36:27.064154+0000 pg 19.5e not deep-scrubbed since 2025-06-05T00:52:03.859984+0000 pg 16.5c not deep-scrubbed since 2025-06-06T00:36:22.021390+0000 pg 19.5f not deep-scrubbed since 2025-06-03T16:27:42.356213+0000 pg 22.5a not deep-scrubbed since 2025-06-05T23:00:28.066065+0000 pg 34.69 not deep-scrubbed since 2025-06-03T05:07:58.209808+0000 pg 19.58 not deep-scrubbed since 2025-05-27T23:32:29.963976+0000 pg 19.59 not deep-scrubbed since 2025-05-25T11:50:44.735318+0000 pg 19.5a not deep-scrubbed since 2025-06-06T02:34:05.486126+0000 pg 16.58 not deep-scrubbed since 2025-05-13T14:10:44.570493+0000
Regards Dev _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dev, Please also share a 'ceph config dump' (remove any passwords / confidential information from the output) so we can look for any configuration issues. And share the manufacturer / model of the drives. Did you check /var/log/messages for any hardware or network issues? Regards, Frédéric. ----- Le 18 Juin 25, à 23:33, Anthony D'Atri aad@dreamsnake.net a écrit :
On Jun 18, 2025, at 5:03 PM, Devender Singh <devender@netskrt.io> wrote:
Hello all
Need urgent help on below… I tried reducing min_size but still showing same…
cluster: id: 15688cb4-044a-11ec-942e-516035adea04 health: HEALTH_ERR 3 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 20/16670718 objects unfound (0.000%)
Did you suffer a power outage or something? What are your OSDs? HDD? SSD? If SSD, are they *enterprise* not client?
Reduced data availability: 283 pgs inactive, 464 pgs incomplete Possible data damage: 2 pgs recovery_unfound Degraded data redundancy: 42998/61175329 objects degraded (0.070%), 2 pgs degraded, 1 pg undersized 304 pgs not deep-scrubbed in time 1055 slow ops, oldest one blocked for 94042 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops.
Look for a pattern. Are these all on the same host? Send `ceph osd tree`
In the meantime run `ceph osd down 1`, wait for a couple of minutes for recovery, and see if that improves the numbers. If it does, repeat with the other OSDs above, waiting for the PGs to peer and all 31 OSDs to show up before proceeding to the next.
services: mon: 5 daemons, quorum van2-converged04n,van2-converged05n,van2-converged01n,van2-converged03n,van2-converged02n (age 17h) mgr: van2-converged05n.lybhho(active, since 22h), standbys: van2-converged04n.azmqik mds: 3/3 daemons up, 11 standby osd: 31 osds: 31 up (since 5h), 31 in (since 5h); 2 remapped pgs rbd-mirror: 2 daemons active (2 hosts)
data: volumes: 1/2 healthy, 1 recovering pools: 32 pools, 1505 pgs objects: 16.67M objects, 56 TiB usage: 152 TiB used, 234 TiB / 387 TiB avail pgs: 0.332% pgs unknown 30.897% pgs not active 42998/61175329 objects degraded (0.070%) 21469/61175329 objects misplaced (0.035%) 20/16670718 objects unfound (0.000%) 1034 active+clean 463 incomplete 5 unknown 1 remapped+incomplete 1 active+recovery_unfound+degraded 1 recovery_unfound+undersized+degraded+remapped+peered
progress: Global Recovery Event (22h) [===================.........] (remaining: 10h)
[WRN] SLOW_OPS: 1055 slow ops, oldest one blocked for 93877 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops. [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.van2.van2-converged05n.mbbzfj(mds.1): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 62279 secs [WRN] OBJECT_UNFOUND: 20/16670718 objects unfound (0.000%) pg 28.1f has 9 unfound objects pg 29.2 has 11 unfound objects [WRN] PG_AVAILABILITY: Reduced data availability: 470 pgs inactive, 464 pgs incomplete pg 16.58 is incomplete, acting [7,14,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5c is incomplete, acting [26,3,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5f is incomplete, acting [13,6,21] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.63 is incomplete, acting [9,7,15] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.64 is incomplete, acting [18,19,5] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.68 is incomplete, acting [2,5,27] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.70 is incomplete, acting [19,1,24] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.74 is incomplete, acting [8,25,14] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.78 is incomplete, acting [23,14,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7c is incomplete, acting [24,10,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7d is incomplete, acting [12,22,13] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.64 is incomplete, acting [20,11,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.66 is incomplete, acting [21,13,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6a is incomplete, acting [4,26,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6c is incomplete, acting [13,12,31] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6e is incomplete, acting [8,31,24] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.7a is incomplete, acting [8,28,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 19.54 is stuck inactive since forever, current state incomplete, last acting [10,4,20,15,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.58 is incomplete, acting [15,18,12,28,20] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.59 is incomplete, acting [6,9,18,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5a is incomplete, acting [11,31,20,17,24] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for ‘incomplceph -s Ceph
Any way to make these scrubs complete faster?
pg 19.63 not deep-scrubbed since 2025-05-24T08:18:29.738427+0000 pg 34.6d not deep-scrubbed since 2025-06-04T08:33:54.534882+0000 pg 16.5f not deep-scrubbed since 2025-05-21T10:44:09.996254+0000 pg 19.5d not deep-scrubbed since 2025-05-26T09:36:27.064154+0000 pg 19.5e not deep-scrubbed since 2025-06-05T00:52:03.859984+0000 pg 16.5c not deep-scrubbed since 2025-06-06T00:36:22.021390+0000 pg 19.5f not deep-scrubbed since 2025-06-03T16:27:42.356213+0000 pg 22.5a not deep-scrubbed since 2025-06-05T23:00:28.066065+0000 pg 34.69 not deep-scrubbed since 2025-06-03T05:07:58.209808+0000 pg 19.58 not deep-scrubbed since 2025-05-27T23:32:29.963976+0000 pg 19.59 not deep-scrubbed since 2025-05-25T11:50:44.735318+0000 pg 19.5a not deep-scrubbed since 2025-06-06T02:34:05.486126+0000 pg 16.58 not deep-scrubbed since 2025-05-13T14:10:44.570493+0000
Regards Dev _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your reply.. There were multiple OSD’s failure on different hosts and had to zap and read. Which lead us to this issue… and upgrade was running too.. Now I see three options for upgrade “Back, Pause & Stop” — is back for rollback? And which one I should use the best option…. Tried restating osd, but it seems dam slow and few are not coming up….and then need zap and add again… What to do with incomplete pigs? Here are the details/… Sorry for the wrong detail in previous emails… correct are below root@phl-host03n:~# ceph -s cluster: id: a0bd51e8-4dfc-11ee-b5a9-3b06e501a0ac health: HEALTH_WARN 8 OSD(s) experiencing slow operations in BlueStore Failed to apply 2 service(s): osd.all-available-devices,osd.iops_optimized 1 failed cephadm daemon(s) failed to probe daemons or devices noscrub,nodeep-scrub flag(s) set Degraded data redundancy: 1739710/39361366 objects degraded (4.420%), 92 pgs degraded, 103 pgs undersized 127 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum phl-host04n,phl-host02n,phl-host01n,phl-host05n,phl-host03n (age 32h) mgr: phl-host01n.pkkztj(active, since 110m), standbys: phl-host02n.wokbnx mds: 3/3 daemons up, 2 standby osd: 88 osds: 88 up (since 2h), 88 in (since 34m); 182 remapped pgs flags noscrub,nodeep-scrub rbd-mirror: 2 daemons active (2 hosts) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 3/3 healthy pools: 23 pools, 1169 pgs objects: 9.22M objects, 32 TiB usage: 68 TiB used, 239 TiB / 307 TiB avail pgs: 1739710/39361366 objects degraded (4.420%) 3592355/39361366 objects misplaced (9.127%) 982 active+clean 85 active+undersized+degraded+remapped+backfill_wait 78 active+remapped+backfill_wait 10 active+undersized+remapped+backfill_wait 7 active+undersized+degraded+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+remapped+backfilling 1 active+undersized+remapped+backfilling io: client: 140 MiB/s rd, 9.6 MiB/s wr, 528 op/s rd, 379 op/s wr recovery: 104 MiB/s, 27 objects/s root@phl-host03n:~# ceph orch upgrade status { "target_image": "quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon" ], "progress": "74/112 daemons upgraded", "message": "", "is_paused": false } # ceph config dump WHO MASK LEVEL OPTION VALUE RO global advanced client_reconnect_stale true global advanced cluster_network 10.104.5.0/24 * global basic container_image quay.io/ceph/ceph@sha256:1b9158ce28975f95def6a0ad459fa19f1336506074267a4b47c1bd914a00fec0 * global basic log_to_file true global basic mds_cache_memory_limit 10589934592 global advanced mgr/devicehealth/self_heal true global advanced mon_cluster_log_to_file true global advanced osd_pool_default_pg_autoscale_mode on global dev rbd_mirror_memory_base 1005306368 global advanced rbd_skip_partial_discard false mon advanced auth_allow_insecure_global_id_reclaim false mon basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * mon advanced mon_allow_pool_delete true mon advanced mon_mds_skip_sanity true mon advanced public_network 10.104.1.0/24 * mgr basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * mgr advanced mgr/cephadm/container_init True * mgr advanced mgr/cephadm/migration_current 7 * mgr advanced mgr/cephadm/warn_on_stray_daemons/warn_on_stray_hosts false * mgr advanced mgr/cephadm/warn_on_stray_hosts false * mgr advanced mgr/cephadm/warn_stray_daemons/warn_on_stray_hosts false * mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://pl-host04n.phl.example.com.org:9093 * mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false * mgr advanced mgr/dashboard/GRAFANA_API_URL https://pl-host02n.phl.example.com.org:3000 * mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://pl-host04n.phl.example.com.org:9095 * mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY * mgr advanced mgr/dashboard/RGW_API_SECRET_KEY * mgr advanced mgr/dashboard/ssl_server_port 8443 * mgr advanced mgr/mds_autoscaler/log_level warning mgr advanced mgr/mds_autoscaler/log_to_cluster true mgr advanced mgr/mds_autoscaler/log_to_file true mgr advanced mgr/orchestrator/orchestrator cephadm mgr advanced mgr/selftest/pl-host04n.tvzwes/testkey foo * mgr advanced mgr/selftest/rwoption1 1234 * mgr advanced mgr/selftest/rwoption2 10 * mgr advanced mgr/selftest/rwoption3 1.500000 * mgr advanced mgr/selftest/rwoption4 foo * mgr advanced mgr/selftest/rwoption5 false * mgr advanced mgr/selftest/testkey test * mgr advanced mgr/telemetry/channel_basic false * mgr advanced mgr/telemetry/channel_crash false * mgr advanced mgr/telemetry/channel_device false * mgr advanced mgr/telemetry/contact test@test.com * mgr advanced mgr/telemetry/interval 60 * mgr advanced mgr/telemetry/leaderboard true * mgr advanced mgr/zabbix/identifier ceph * mgr advanced mgr/zabbix/log_to_cluster false mgr advanced mgr/zabbix/log_to_file false mgr advanced mgr/zabbix/zabbix_host 10.0.2.110 * osd advanced bdev_aio_poll_ms 200 osd advanced bdev_ioring false osd advanced bluestore_throttle_bytes 134217728 osd advanced ms_async_op_threads 4 osd advanced osd_heartbeat_grace 70 osd advanced osd_max_backfills 1 osd advanced osd_mclock_override_recovery_settings true osd host:phl-converged01n basic osd_memory_target 19352618016 osd host:phl-converged02n basic osd_memory_target 31346288264 osd host:phl-converged03n basic osd_memory_target 19804699313 osd host:phl-converged04n basic osd_memory_target 18593027461 osd host:phl-converged05n basic osd_memory_target 20606720591 osd host:pl-host01n basic osd_memory_target 15058518075 osd host:pl-host02n basic osd_memory_target 14086545280 osd host:pl-host03n basic osd_memory_target 33342959038 osd host:pl-host04n basic osd_memory_target 53122886860 osd host:pl-host05n basic osd_memory_target 16313721223 osd advanced osd_memory_target_autotune true osd advanced osd_recovery_max_active_ssd 1 osd advanced osd_recovery_max_single_start 1 osd advanced osd_recovery_sleep 0.000000 osd advanced osd_requested_scrub_priority 1 osd advanced osd_scrub_disable_reservation_queuing true osd advanced osd_scrub_sleep 0.000000 osd advanced osd_snap_trim_cost 262144 osd.0 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.0 basic osd_mclock_max_capacity_iops_ssd 37869.100179 osd.1 basic osd_mclock_max_capacity_iops_ssd 38610.482854 osd.10 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.10 basic osd_mclock_max_capacity_iops_ssd 35297.387846 osd.100 basic osd_mclock_max_capacity_iops_ssd 29091.307131 osd.101 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.101 basic osd_mclock_max_capacity_iops_ssd 27963.050594 osd.102 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.102 basic osd_mclock_max_capacity_iops_ssd 31172.711483 osd.103 basic osd_mclock_max_capacity_iops_ssd 28664.146986 osd.104 basic osd_mclock_max_capacity_iops_ssd 14405.473296 osd.105 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.105 basic osd_mclock_max_capacity_iops_ssd 28291.779421 osd.106 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.106 basic osd_mclock_max_capacity_iops_ssd 30678.497618 osd.107 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.107 basic osd_mclock_max_capacity_iops_ssd 26799.726954 osd.108 basic osd_mclock_max_capacity_iops_ssd 27523.880430 osd.109 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.109 basic osd_mclock_max_capacity_iops_ssd 27819.225848 osd.11 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.11 basic osd_mclock_max_capacity_iops_ssd 35852.830870 osd.110 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.110 basic osd_mclock_max_capacity_iops_ssd 28671.008445 osd.111 basic osd_mclock_max_capacity_iops_ssd 28610.088976 osd.112 basic osd_mclock_max_capacity_iops_ssd 27462.279164 osd.113 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.113 basic osd_mclock_max_capacity_iops_ssd 25652.864986 osd.114 basic osd_mclock_max_capacity_iops_ssd 27469.642657 osd.115 basic osd_mclock_max_capacity_iops_ssd 26599.443843 osd.116 basic osd_mclock_max_capacity_iops_ssd 28159.154679 osd.117 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.117 basic osd_mclock_max_capacity_iops_ssd 30631.799864 osd.118 basic osd_mclock_max_capacity_iops_ssd 26888.868629 osd.119 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.119 basic osd_mclock_max_capacity_iops_ssd 27837.170185 osd.12 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.12 basic osd_mclock_max_capacity_iops_ssd 36523.110430 osd.120 basic osd_mclock_max_capacity_iops_ssd 25156.699403 osd.121 basic osd_mclock_max_capacity_iops_ssd 16854.236813 osd.122 basic osd_mclock_max_capacity_iops_ssd 29425.668046 osd.123 basic osd_mclock_max_capacity_iops_ssd 28424.967707 osd.124 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.124 basic osd_mclock_max_capacity_iops_ssd 30490.214968 osd.125 basic osd_mclock_max_capacity_iops_ssd 29875.742604 osd.126 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.126 basic osd_mclock_max_capacity_iops_ssd 29827.105097 osd.127 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.127 basic osd_mclock_max_capacity_iops_ssd 28899.535735 osd.128 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.128 basic osd_mclock_max_capacity_iops_ssd 27569.775057 osd.129 basic osd_mclock_max_capacity_iops_ssd 32820.522874 osd.13 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.13 basic osd_mclock_max_capacity_iops_ssd 38260.450852 osd.130 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.130 basic osd_mclock_max_capacity_iops_ssd 33740.656917 osd.131 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.131 basic osd_mclock_max_capacity_iops_ssd 30837.814068 osd.132 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.132 basic osd_mclock_max_capacity_iops_ssd 32357.136152 osd.133 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.133 basic osd_mclock_max_capacity_iops_ssd 33199.681376 osd.134 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.134 basic osd_mclock_max_capacity_iops_ssd 32701.919940 osd.135 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.135 basic osd_mclock_max_capacity_iops_ssd 31471.381829 osd.136 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.136 basic osd_mclock_max_capacity_iops_ssd 36040.644043 osd.137 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.137 basic osd_mclock_max_capacity_iops_ssd 34135.109403 osd.138 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.138 basic osd_mclock_max_capacity_iops_ssd 28739.646099 osd.139 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.139 basic osd_mclock_max_capacity_iops_ssd 28611.027050 osd.14 basic osd_mclock_max_capacity_iops_ssd 32587.786337 osd.15 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.15 basic osd_mclock_max_capacity_iops_ssd 33930.129246 osd.16 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.16 basic osd_mclock_max_capacity_iops_ssd 37610.362437 osd.17 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.17 basic osd_mclock_max_capacity_iops_ssd 22609.652689 osd.18 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.18 basic osd_mclock_max_capacity_iops_ssd 29116.630263 osd.19 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.19 basic osd_mclock_max_capacity_iops_ssd 34639.127606 osd.2 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.2 basic osd_mclock_max_capacity_iops_ssd 32777.963192 osd.20 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.20 basic osd_mclock_max_capacity_iops_ssd 33042.683283 osd.21 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.21 basic osd_mclock_max_capacity_iops_ssd 34608.069667 osd.22 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.22 basic osd_mclock_max_capacity_iops_ssd 33672.786827 osd.23 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.23 basic osd_mclock_max_capacity_iops_ssd 37253.942367 osd.24 basic osd_mclock_max_capacity_iops_ssd 32010.520279 osd.25 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.25 basic osd_mclock_max_capacity_iops_ssd 18731.103127 osd.26 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.26 basic osd_mclock_max_capacity_iops_ssd 33844.524796 osd.27 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.27 basic osd_mclock_max_capacity_iops_ssd 33592.404677 osd.28 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.28 basic osd_mclock_max_capacity_iops_ssd 34945.380022 osd.29 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.29 basic osd_mclock_max_capacity_iops_ssd 33041.812034 osd.3 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.3 basic osd_mclock_max_capacity_iops_ssd 30526.239088 osd.30 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.30 basic osd_mclock_max_capacity_iops_ssd 33849.746190 osd.31 basic osd_mclock_max_capacity_iops_ssd 31073.672124 osd.32 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.32 basic osd_mclock_max_capacity_iops_ssd 34475.463692 osd.33 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.33 basic osd_mclock_max_capacity_iops_ssd 35294.129689 osd.34 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.34 basic osd_mclock_max_capacity_iops_ssd 33500.582011 osd.35 basic osd_mclock_max_capacity_iops_ssd 33937.027809 osd.36 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.36 basic osd_mclock_max_capacity_iops_ssd 33363.104343 osd.37 basic osd_mclock_max_capacity_iops_ssd 33136.270826 osd.38 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.38 basic osd_mclock_max_capacity_iops_ssd 34327.772673 osd.39 basic osd_mclock_max_capacity_iops_ssd 36197.080652 osd.4 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.4 basic osd_mclock_max_capacity_iops_ssd 31292.480847 osd.40 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.40 basic osd_mclock_max_capacity_iops_ssd 30200.043883 osd.41 basic osd_mclock_max_capacity_iops_ssd 28660.094990 osd.42 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.42 basic osd_mclock_max_capacity_iops_ssd 29042.271559 osd.43 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.43 basic osd_mclock_max_capacity_iops_ssd 29802.985872 osd.44 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.44 basic osd_mclock_max_capacity_iops_ssd 29872.248343 osd.45 basic osd_mclock_max_capacity_iops_ssd 27048.306734 osd.46 basic osd_mclock_max_capacity_iops_ssd 27445.922772 osd.47 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.47 basic osd_mclock_max_capacity_iops_ssd 29463.719353 osd.48 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.48 basic osd_mclock_max_capacity_iops_ssd 27648.479917 osd.49 basic osd_mclock_max_capacity_iops_ssd 26156.443046 osd.5 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.5 basic osd_mclock_max_capacity_iops_ssd 31042.919796 osd.50 basic osd_mclock_max_capacity_iops_ssd 27126.914926 osd.51 basic osd_mclock_max_capacity_iops_ssd 24087.063752 osd.52 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.52 basic osd_mclock_max_capacity_iops_ssd 29446.685299 osd.53 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.53 basic osd_mclock_max_capacity_iops_ssd 30223.321026 osd.54 basic osd_mclock_max_capacity_iops_ssd 29041.708424 osd.55 basic osd_mclock_max_capacity_iops_ssd 31812.937889 osd.56 basic osd_mclock_max_capacity_iops_ssd 28114.742336 osd.57 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.57 basic osd_mclock_max_capacity_iops_ssd 28757.116835 osd.58 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.58 basic osd_mclock_max_capacity_iops_ssd 29569.415786 osd.59 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.59 basic osd_mclock_max_capacity_iops_ssd 29009.310857 osd.6 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.6 basic osd_mclock_max_capacity_iops_ssd 35515.197243 osd.60 basic osd_mclock_max_capacity_iops_ssd 27353.095641 osd.61 basic osd_mclock_max_capacity_iops_ssd 28033.751890 osd.62 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.62 basic osd_mclock_max_capacity_iops_ssd 28270.316050 osd.63 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.63 basic osd_mclock_max_capacity_iops_ssd 29194.558053 osd.64 basic osd_mclock_max_capacity_iops_ssd 25543.077545 osd.65 basic osd_mclock_max_capacity_iops_ssd 29257.957631 osd.66 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.66 basic osd_mclock_max_capacity_iops_ssd 27251.592177 osd.67 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.67 basic osd_mclock_max_capacity_iops_ssd 28428.290504 osd.68 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.68 basic osd_mclock_max_capacity_iops_ssd 30541.378615 osd.69 basic osd_mclock_max_capacity_iops_ssd 29164.568681 osd.7 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.7 basic osd_mclock_max_capacity_iops_ssd 35788.435091 osd.70 basic osd_mclock_max_capacity_iops_ssd 27050.432719 osd.71 basic osd_mclock_max_capacity_iops_ssd 27615.938915 osd.72 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.72 basic osd_mclock_max_capacity_iops_ssd 29912.870688 osd.73 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.73 basic osd_mclock_max_capacity_iops_ssd 30750.315068 osd.74 basic osd_mclock_max_capacity_iops_hdd 274.216582 osd.74 basic osd_mclock_max_capacity_iops_ssd 31296.317898 osd.75 basic osd_mclock_max_capacity_iops_ssd 31330.904882 osd.76 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.76 basic osd_mclock_max_capacity_iops_ssd 29541.036281 osd.77 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.77 basic osd_mclock_max_capacity_iops_ssd 27755.161947 osd.78 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.78 basic osd_mclock_max_capacity_iops_ssd 30181.440559 osd.79 basic osd_mclock_max_capacity_iops_ssd 28820.911162 osd.8 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.8 basic osd_mclock_max_capacity_iops_ssd 32704.310615 osd.80 basic osd_mclock_max_capacity_iops_ssd 30604.883487 osd.81 basic osd_mclock_max_capacity_iops_ssd 26188.393507 osd.82 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.82 basic osd_mclock_max_capacity_iops_ssd 27407.424799 osd.83 basic osd_mclock_max_capacity_iops_hdd 271.607604 osd.83 basic osd_mclock_max_capacity_iops_ssd 29270.002337 osd.84 basic osd_mclock_max_capacity_iops_ssd 29874.802768 osd.85 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.85 basic osd_mclock_max_capacity_iops_ssd 33267.445310 osd.86 basic osd_mclock_max_capacity_iops_ssd 26647.075973 osd.87 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.87 basic osd_mclock_max_capacity_iops_ssd 23232.299984 osd.88 basic osd_mclock_max_capacity_iops_ssd 30225.432154 osd.89 basic osd_mclock_max_capacity_iops_ssd 26223.229524 osd.9 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.9 basic osd_mclock_max_capacity_iops_hdd 265.536539 osd.9 basic osd_mclock_max_capacity_iops_ssd 35082.146366 osd.90 basic osd_mclock_max_capacity_iops_ssd 28870.253846 osd.91 basic osd_mclock_max_capacity_iops_ssd 28370.754122 osd.92 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.92 basic osd_mclock_max_capacity_iops_ssd 25124.799906 osd.93 basic osd_mclock_max_capacity_iops_ssd 26721.773615 osd.94 basic osd_mclock_max_capacity_iops_ssd 28325.510268 osd.95 basic osd_mclock_max_capacity_iops_ssd 27777.045030 osd.96 basic osd_mclock_max_capacity_iops_ssd 28368.353309 osd.97 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.97 basic osd_mclock_max_capacity_iops_ssd 22786.649038 osd.98 basic osd_mclock_max_capacity_iops_ssd 29639.368028 osd.99 basic osd_mclock_max_capacity_iops_ssd 27038.486473 mds.k8s-it-infra-cephfs basic mds_join_fs k8s-it-infra-cephfs mds.k8s-prod-cephfs basic mds_join_fs k8s-prod-cephfs mds.phl.prod basic mds_join_fs phl.prod client advanced client_asio_thread_count 4 client advanced rbd_op_threads 2 client.crash basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * client.rgw.phl.pl-host01n.titvzo basic rgw_frontends beast ssl_port=443 ssl_certificate=config://rgw/cert/rgw.phl * client.rgw.phl.pl-host05n.frnpbx basic rgw_frontends beast ssl_port=443 ssl_certificate=config://rgw/cert/rgw.phl * # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 307.39172 root default -19 83.83411 host pl-host01n 130 nvme 3.49309 osd.130 up 1.00000 1.00000 131 nvme 3.49309 osd.131 up 1.00000 1.00000 132 nvme 3.49309 osd.132 up 1.00000 1.00000 133 nvme 3.49309 osd.133 up 1.00000 1.00000 134 nvme 3.49309 osd.134 up 1.00000 1.00000 135 nvme 3.49309 osd.135 up 1.00000 1.00000 136 nvme 3.49309 osd.136 up 1.00000 1.00000 137 nvme 3.49309 osd.137 up 1.00000 1.00000 15 ssd 3.49309 osd.15 up 1.00000 1.00000 32 ssd 3.49309 osd.32 up 1.00000 1.00000 34 ssd 3.49309 osd.34 up 1.00000 1.00000 38 ssd 3.49309 osd.38 up 1.00000 1.00000 42 ssd 3.49309 osd.42 up 1.00000 1.00000 76 ssd 3.49309 osd.76 up 1.00000 1.00000 101 ssd 3.49309 osd.101 up 1.00000 1.00000 105 ssd 3.49309 osd.105 up 1.00000 1.00000 109 ssd 3.49309 osd.109 up 1.00000 1.00000 113 ssd 3.49309 osd.113 up 1.00000 1.00000 117 ssd 3.49309 osd.117 up 1.00000 1.00000 119 ssd 3.49309 osd.119 up 1.00000 1.00000 124 ssd 3.49309 osd.124 up 1.00000 1.00000 127 ssd 3.49309 osd.127 up 1.00000 1.00000 138 ssd 3.49309 osd.138 up 1.00000 1.00000 139 ssd 3.49309 osd.139 up 1.00000 1.00000 -25 83.83411 host pl-host02n 0 nvme 3.49309 osd.0 up 1.00000 1.00000 40 nvme 3.49309 osd.40 up 1.00000 1.00000 47 nvme 3.49309 osd.47 up 1.00000 1.00000 52 nvme 3.49309 osd.52 up 1.00000 1.00000 57 nvme 3.49309 osd.57 up 1.00000 1.00000 62 nvme 3.49309 osd.62 up 1.00000 1.00000 67 nvme 3.49309 osd.67 up 1.00000 1.00000 72 nvme 3.49309 osd.72 up 1.00000 1.00000 92 nvme 3.49309 osd.92 up 1.00000 1.00000 7 ssd 3.49309 osd.7 up 1.00000 1.00000 10 ssd 3.49309 osd.10 up 1.00000 1.00000 11 ssd 3.49309 osd.11 up 1.00000 1.00000 17 ssd 3.49309 osd.17 up 1.00000 1.00000 20 ssd 3.49309 osd.20 up 1.00000 1.00000 23 ssd 3.49309 osd.23 up 1.00000 1.00000 25 ssd 3.49309 osd.25 up 1.00000 1.00000 26 ssd 3.49309 osd.26 up 1.00000 1.00000 27 ssd 3.49309 osd.27 up 1.00000 1.00000 28 ssd 3.49309 osd.28 up 1.00000 1.00000 30 ssd 3.49309 osd.30 up 1.00000 1.00000 77 ssd 3.49309 osd.77 up 1.00000 1.00000 82 ssd 3.49309 osd.82 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 97 ssd 3.49309 osd.97 up 1.00000 1.00000 -22 38.42397 host pl-host03n 3 ssd 3.49309 osd.3 up 1.00000 1.00000 4 ssd 3.49309 osd.4 up 1.00000 1.00000 6 ssd 3.49309 osd.6 up 1.00000 1.00000 8 ssd 3.49309 osd.8 up 1.00000 1.00000 13 ssd 3.49309 osd.13 up 1.00000 1.00000 18 ssd 3.49309 osd.18 up 1.00000 1.00000 59 ssd 3.49309 osd.59 up 1.00000 1.00000 66 ssd 3.49309 osd.66 up 1.00000 1.00000 102 ssd 3.49309 osd.102 up 1.00000 1.00000 106 ssd 3.49309 osd.106 up 1.00000 1.00000 110 ssd 3.49309 osd.110 up 1.00000 1.00000 -28 20.95853 host pl-host04n 48 nvme 3.49309 osd.48 up 1.00000 1.00000 5 ssd 3.49309 osd.5 up 1.00000 1.00000 9 ssd 3.49309 osd.9 up 1.00000 1.00000 21 ssd 3.49309 osd.21 up 1.00000 1.00000 36 ssd 3.49309 osd.36 up 1.00000 1.00000 43 ssd 3.49309 osd.43 up 1.00000 1.00000 -31 80.34102 host pl-host05n 24 nvme 3.49309 osd.24 up 1.00000 1.00000 45 nvme 3.49309 osd.45 up 1.00000 1.00000 49 nvme 3.49309 osd.49 up 1.00000 1.00000 55 nvme 3.49309 osd.55 up 1.00000 1.00000 60 nvme 3.49309 osd.60 up 1.00000 1.00000 65 nvme 3.49309 osd.65 up 1.00000 1.00000 70 nvme 3.49309 osd.70 up 1.00000 1.00000 85 nvme 3.49309 osd.85 up 1.00000 1.00000 90 nvme 3.49309 osd.90 up 1.00000 1.00000 1 ssd 3.49309 osd.1 up 1.00000 1.00000 14 ssd 3.49309 osd.14 up 1.00000 1.00000 31 ssd 3.49309 osd.31 up 1.00000 1.00000 35 ssd 3.49309 osd.35 up 1.00000 1.00000 37 ssd 3.49309 osd.37 up 1.00000 1.00000 39 ssd 3.49309 osd.39 up 1.00000 1.00000 41 ssd 3.49309 osd.41 up 1.00000 1.00000 80 ssd 3.49309 osd.80 up 1.00000 1.00000 95 ssd 3.49309 osd.95 up 1.00000 1.00000 100 ssd 3.49309 osd.100 up 1.00000 1.00000 108 ssd 3.49309 osd.108 up 1.00000 1.00000 112 ssd 3.49309 osd.112 up 1.00000 1.00000 116 ssd 3.49309 osd.116 up 1.00000 1.00000 125 ssd 3.49309 osd.125 up 1.00000 1.00000
On Jun 19, 2025, at 3:25 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
hare a 'ceph config dump' (remove any passwords / confidential information from the output)
Hi Dev, Tips, real quick: - stop zapping devices :-) - I've never heard of any 'back' option on the 'ceph orch upgrade' command [1], but 'stop' could be used to allow other operations to proceed, would it be needed at some point. BTW, I will assume that you started the upgrade prior to OSDs failing to start, and not after. - I would advise that you set 'osd_op_queue' to 'wpq', to help with the current situation. Start with a single OSD (ceph osd config set osd.0 osd_op_queue wpq) and keep going with other OSDs if it started and joined the cluster without issues. - If using Podman, SSH to every hosts and run 'df -i' to check for inode exhaustion on /run partition. There's a podman bug [2] in EL 9.5 that was fixed in EL 9.6 by podman-5:5.4.0-9.el9_6.x86_64 . Could also affect other distros. In case of inode exhaustion, no containers would start. Inode exhaustion can happen really fast as the orchestrator is regularly running containers on every hosts for inventory but not only. - Choose one of the '8 OSD(s) experiencing slow operations in BlueStore' and share 'ceph tell osd.X bluefs stats' and 'ceph tell osd.0 bluestore allocator score block'. - Please share the manufacturer/model of SSDs and NVMes. Regards, Frédéric. [1] [ https://docs.ceph.com/en/latest/cephadm/upgrade/ | https://docs.ceph.com/en/latest/cephadm/upgrade/ ] [2] [ https://github.com/containers/podman/issues/25291 | https://github.com/containers/podman/issues/25291 ] ----- Le 20 Juin 25, à 5:36, Devender Singh <devender@netskrt.io> a écrit :
Thanks for your reply.. There were multiple OSD’s failure on different hosts and had to zap and read. Which lead us to this issue… and upgrade was running too.. Now I see three options for upgrade “Back, Pause & Stop” — is back for rollback? And which one I should use the best option…. Tried restating osd, but it seems dam slow and few are not coming up….and then need zap and add again… What to do with incomplete pigs?
Here are the details/…
Sorry for the wrong detail in previous emails… correct are below
root@phl-host03n:~# ceph -s
cluster:
id: a0bd51e8-4dfc-11ee-b5a9-3b06e501a0ac
health: HEALTH_WARN
8 OSD(s) experiencing slow operations in BlueStore
Failed to apply 2 service(s): osd.all-available-devices,osd.iops_optimized
1 failed cephadm daemon(s)
failed to probe daemons or devices
noscrub,nodeep-scrub flag(s) set
Degraded data redundancy: 1739710/39361366 objects degraded (4.420%), 92 pgs degraded, 103 pgs undersized
127 pgs not deep-scrubbed in time
services:
mon: 5 daemons, quorum phl-host04n,phl-host02n,phl-host01n,phl-host05n,phl-host03n (age 32h)
mgr: phl-host01n.pkkztj(active, since 110m), standbys: phl-host02n.wokbnx
mds: 3/3 daemons up, 2 standby
osd: 88 osds: 88 up (since 2h), 88 in (since 34m); 182 remapped pgs
flags noscrub,nodeep-scrub
rbd-mirror: 2 daemons active (2 hosts)
rgw: 2 daemons active (2 hosts, 1 zones)
data:
volumes: 3/3 healthy
pools: 23 pools, 1169 pgs
objects: 9.22M objects, 32 TiB
usage: 68 TiB used, 239 TiB / 307 TiB avail
pgs: 1739710/39361366 objects degraded (4.420%)
3592355/39361366 objects misplaced (9.127%)
982 active+clean
85 active+undersized+degraded+remapped+backfill_wait
78 active+remapped+backfill_wait
10 active+undersized+remapped+backfill_wait
7 active+undersized+degraded+remapped+backfilling
5 active+clean+scrubbing+deep
1 active+remapped+backfilling
1 active+undersized+remapped+backfilling
io:
client: 140 MiB/s rd, 9.6 MiB/s wr, 528 op/s rd, 379 op/s wr
recovery: 104 MiB/s, 27 objects/s
root@phl-host03n:~# ceph orch upgrade status
{
"target_image": "quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6",
"in_progress": true,
"which": "Upgrading all daemon types on all hosts",
"services_complete": [
"crash",
"mgr",
"mon"
],
"progress": "74/112 daemons upgraded",
"message": "",
"is_paused": false
}
# ceph config dump
WHO MASK LEVEL OPTION VALUE RO
global advanced client_reconnect_stale true
global advanced cluster_network 10.104.5.0/24 *
global basic container_image quay.io/ceph/ceph@sha256:1b9158ce28975f95def6a0ad459fa19f1336506074267a4b47c1bd914a00fec0 *
global basic log_to_file true
global basic mds_cache_memory_limit 10589934592
global advanced mgr/devicehealth/self_heal true
global advanced mon_cluster_log_to_file true
global advanced osd_pool_default_pg_autoscale_mode on
global dev rbd_mirror_memory_base 1005306368
global advanced rbd_skip_partial_discard false
mon advanced auth_allow_insecure_global_id_reclaim false
mon basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
mon advanced mon_allow_pool_delete true
mon advanced mon_mds_skip_sanity true
mon advanced public_network 10.104.1.0/24 *
mgr basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
mgr advanced mgr/cephadm/container_init True *
mgr advanced mgr/cephadm/migration_current 7 *
mgr advanced mgr/cephadm/warn_on_stray_daemons/warn_on_stray_hosts false *
mgr advanced mgr/cephadm/warn_on_stray_hosts false *
mgr advanced mgr/cephadm/warn_stray_daemons/warn_on_stray_hosts false *
mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://pl-host04n.phl.example.com.org:9093 *
mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false *
mgr advanced mgr/dashboard/GRAFANA_API_URL https://pl-host02n.phl.example.com.org:3000 *
mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://pl-host04n.phl.example.com.org:9095 *
mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY *
mgr advanced mgr/dashboard/RGW_API_SECRET_KEY *
mgr advanced mgr/dashboard/ssl_server_port 8443 *
mgr advanced mgr/mds_autoscaler/log_level warning
mgr advanced mgr/mds_autoscaler/log_to_cluster true
mgr advanced mgr/mds_autoscaler/log_to_file true
mgr advanced mgr/orchestrator/orchestrator cephadm
mgr advanced mgr/selftest/pl-host04n.tvzwes/testkey foo *
mgr advanced mgr/selftest/rwoption1 1234 *
mgr advanced mgr/selftest/rwoption2 10 *
mgr advanced mgr/selftest/rwoption3 1.500000 *
mgr advanced mgr/selftest/rwoption4 foo *
mgr advanced mgr/selftest/rwoption5 false *
mgr advanced mgr/selftest/testkey test *
mgr advanced mgr/telemetry/channel_basic false *
mgr advanced mgr/telemetry/channel_crash false *
mgr advanced mgr/telemetry/channel_device false *
mgr advanced mgr/telemetry/contact test@test.com *
mgr advanced mgr/telemetry/interval 60 *
mgr advanced mgr/telemetry/leaderboard true *
mgr advanced mgr/zabbix/identifier ceph *
mgr advanced mgr/zabbix/log_to_cluster false
mgr advanced mgr/zabbix/log_to_file false
mgr advanced mgr/zabbix/zabbix_host 10.0.2.110 *
osd advanced bdev_aio_poll_ms 200
osd advanced bdev_ioring false
osd advanced bluestore_throttle_bytes 134217728
osd advanced ms_async_op_threads 4
osd advanced osd_heartbeat_grace 70
osd advanced osd_max_backfills 1
osd advanced osd_mclock_override_recovery_settings true
osd host:phl-converged01n basic osd_memory_target 19352618016
osd host:phl-converged02n basic osd_memory_target 31346288264
osd host:phl-converged03n basic osd_memory_target 19804699313
osd host:phl-converged04n basic osd_memory_target 18593027461
osd host:phl-converged05n basic osd_memory_target 20606720591
osd host:pl-host01n basic osd_memory_target 15058518075
osd host:pl-host02n basic osd_memory_target 14086545280
osd host:pl-host03n basic osd_memory_target 33342959038
osd host:pl-host04n basic osd_memory_target 53122886860
osd host:pl-host05n basic osd_memory_target 16313721223
osd advanced osd_memory_target_autotune true
osd advanced osd_recovery_max_active_ssd 1
osd advanced osd_recovery_max_single_start 1
osd advanced osd_recovery_sleep 0.000000
osd advanced osd_requested_scrub_priority 1
osd advanced osd_scrub_disable_reservation_queuing true
osd advanced osd_scrub_sleep 0.000000
osd advanced osd_snap_trim_cost 262144
osd.0 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.0 basic osd_mclock_max_capacity_iops_ssd 37869.100179
osd.1 basic osd_mclock_max_capacity_iops_ssd 38610.482854
osd.10 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.10 basic osd_mclock_max_capacity_iops_ssd 35297.387846
osd.100 basic osd_mclock_max_capacity_iops_ssd 29091.307131
osd.101 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.101 basic osd_mclock_max_capacity_iops_ssd 27963.050594
osd.102 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.102 basic osd_mclock_max_capacity_iops_ssd 31172.711483
osd.103 basic osd_mclock_max_capacity_iops_ssd 28664.146986
osd.104 basic osd_mclock_max_capacity_iops_ssd 14405.473296
osd.105 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.105 basic osd_mclock_max_capacity_iops_ssd 28291.779421
osd.106 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.106 basic osd_mclock_max_capacity_iops_ssd 30678.497618
osd.107 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.107 basic osd_mclock_max_capacity_iops_ssd 26799.726954
osd.108 basic osd_mclock_max_capacity_iops_ssd 27523.880430
osd.109 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.109 basic osd_mclock_max_capacity_iops_ssd 27819.225848
osd.11 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.11 basic osd_mclock_max_capacity_iops_ssd 35852.830870
osd.110 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.110 basic osd_mclock_max_capacity_iops_ssd 28671.008445
osd.111 basic osd_mclock_max_capacity_iops_ssd 28610.088976
osd.112 basic osd_mclock_max_capacity_iops_ssd 27462.279164
osd.113 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.113 basic osd_mclock_max_capacity_iops_ssd 25652.864986
osd.114 basic osd_mclock_max_capacity_iops_ssd 27469.642657
osd.115 basic osd_mclock_max_capacity_iops_ssd 26599.443843
osd.116 basic osd_mclock_max_capacity_iops_ssd 28159.154679
osd.117 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.117 basic osd_mclock_max_capacity_iops_ssd 30631.799864
osd.118 basic osd_mclock_max_capacity_iops_ssd 26888.868629
osd.119 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.119 basic osd_mclock_max_capacity_iops_ssd 27837.170185
osd.12 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.12 basic osd_mclock_max_capacity_iops_ssd 36523.110430
osd.120 basic osd_mclock_max_capacity_iops_ssd 25156.699403
osd.121 basic osd_mclock_max_capacity_iops_ssd 16854.236813
osd.122 basic osd_mclock_max_capacity_iops_ssd 29425.668046
osd.123 basic osd_mclock_max_capacity_iops_ssd 28424.967707
osd.124 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.124 basic osd_mclock_max_capacity_iops_ssd 30490.214968
osd.125 basic osd_mclock_max_capacity_iops_ssd 29875.742604
osd.126 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.126 basic osd_mclock_max_capacity_iops_ssd 29827.105097
osd.127 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.127 basic osd_mclock_max_capacity_iops_ssd 28899.535735
osd.128 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.128 basic osd_mclock_max_capacity_iops_ssd 27569.775057
osd.129 basic osd_mclock_max_capacity_iops_ssd 32820.522874
osd.13 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.13 basic osd_mclock_max_capacity_iops_ssd 38260.450852
osd.130 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.130 basic osd_mclock_max_capacity_iops_ssd 33740.656917
osd.131 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.131 basic osd_mclock_max_capacity_iops_ssd 30837.814068
osd.132 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.132 basic osd_mclock_max_capacity_iops_ssd 32357.136152
osd.133 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.133 basic osd_mclock_max_capacity_iops_ssd 33199.681376
osd.134 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.134 basic osd_mclock_max_capacity_iops_ssd 32701.919940
osd.135 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.135 basic osd_mclock_max_capacity_iops_ssd 31471.381829
osd.136 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.136 basic osd_mclock_max_capacity_iops_ssd 36040.644043
osd.137 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.137 basic osd_mclock_max_capacity_iops_ssd 34135.109403
osd.138 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.138 basic osd_mclock_max_capacity_iops_ssd 28739.646099
osd.139 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.139 basic osd_mclock_max_capacity_iops_ssd 28611.027050
osd.14 basic osd_mclock_max_capacity_iops_ssd 32587.786337
osd.15 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.15 basic osd_mclock_max_capacity_iops_ssd 33930.129246
osd.16 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.16 basic osd_mclock_max_capacity_iops_ssd 37610.362437
osd.17 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.17 basic osd_mclock_max_capacity_iops_ssd 22609.652689
osd.18 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.18 basic osd_mclock_max_capacity_iops_ssd 29116.630263
osd.19 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.19 basic osd_mclock_max_capacity_iops_ssd 34639.127606
osd.2 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.2 basic osd_mclock_max_capacity_iops_ssd 32777.963192
osd.20 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.20 basic osd_mclock_max_capacity_iops_ssd 33042.683283
osd.21 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.21 basic osd_mclock_max_capacity_iops_ssd 34608.069667
osd.22 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.22 basic osd_mclock_max_capacity_iops_ssd 33672.786827
osd.23 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.23 basic osd_mclock_max_capacity_iops_ssd 37253.942367
osd.24 basic osd_mclock_max_capacity_iops_ssd 32010.520279
osd.25 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.25 basic osd_mclock_max_capacity_iops_ssd 18731.103127
osd.26 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.26 basic osd_mclock_max_capacity_iops_ssd 33844.524796
osd.27 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.27 basic osd_mclock_max_capacity_iops_ssd 33592.404677
osd.28 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.28 basic osd_mclock_max_capacity_iops_ssd 34945.380022
osd.29 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.29 basic osd_mclock_max_capacity_iops_ssd 33041.812034
osd.3 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.3 basic osd_mclock_max_capacity_iops_ssd 30526.239088
osd.30 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.30 basic osd_mclock_max_capacity_iops_ssd 33849.746190
osd.31 basic osd_mclock_max_capacity_iops_ssd 31073.672124
osd.32 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.32 basic osd_mclock_max_capacity_iops_ssd 34475.463692
osd.33 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.33 basic osd_mclock_max_capacity_iops_ssd 35294.129689
osd.34 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.34 basic osd_mclock_max_capacity_iops_ssd 33500.582011
osd.35 basic osd_mclock_max_capacity_iops_ssd 33937.027809
osd.36 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.36 basic osd_mclock_max_capacity_iops_ssd 33363.104343
osd.37 basic osd_mclock_max_capacity_iops_ssd 33136.270826
osd.38 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.38 basic osd_mclock_max_capacity_iops_ssd 34327.772673
osd.39 basic osd_mclock_max_capacity_iops_ssd 36197.080652
osd.4 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.4 basic osd_mclock_max_capacity_iops_ssd 31292.480847
osd.40 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.40 basic osd_mclock_max_capacity_iops_ssd 30200.043883
osd.41 basic osd_mclock_max_capacity_iops_ssd 28660.094990
osd.42 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.42 basic osd_mclock_max_capacity_iops_ssd 29042.271559
osd.43 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.43 basic osd_mclock_max_capacity_iops_ssd 29802.985872
osd.44 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.44 basic osd_mclock_max_capacity_iops_ssd 29872.248343
osd.45 basic osd_mclock_max_capacity_iops_ssd 27048.306734
osd.46 basic osd_mclock_max_capacity_iops_ssd 27445.922772
osd.47 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.47 basic osd_mclock_max_capacity_iops_ssd 29463.719353
osd.48 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.48 basic osd_mclock_max_capacity_iops_ssd 27648.479917
osd.49 basic osd_mclock_max_capacity_iops_ssd 26156.443046
osd.5 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.5 basic osd_mclock_max_capacity_iops_ssd 31042.919796
osd.50 basic osd_mclock_max_capacity_iops_ssd 27126.914926
osd.51 basic osd_mclock_max_capacity_iops_ssd 24087.063752
osd.52 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.52 basic osd_mclock_max_capacity_iops_ssd 29446.685299
osd.53 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.53 basic osd_mclock_max_capacity_iops_ssd 30223.321026
osd.54 basic osd_mclock_max_capacity_iops_ssd 29041.708424
osd.55 basic osd_mclock_max_capacity_iops_ssd 31812.937889
osd.56 basic osd_mclock_max_capacity_iops_ssd 28114.742336
osd.57 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.57 basic osd_mclock_max_capacity_iops_ssd 28757.116835
osd.58 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.58 basic osd_mclock_max_capacity_iops_ssd 29569.415786
osd.59 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.59 basic osd_mclock_max_capacity_iops_ssd 29009.310857
osd.6 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.6 basic osd_mclock_max_capacity_iops_ssd 35515.197243
osd.60 basic osd_mclock_max_capacity_iops_ssd 27353.095641
osd.61 basic osd_mclock_max_capacity_iops_ssd 28033.751890
osd.62 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.62 basic osd_mclock_max_capacity_iops_ssd 28270.316050
osd.63 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.63 basic osd_mclock_max_capacity_iops_ssd 29194.558053
osd.64 basic osd_mclock_max_capacity_iops_ssd 25543.077545
osd.65 basic osd_mclock_max_capacity_iops_ssd 29257.957631
osd.66 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.66 basic osd_mclock_max_capacity_iops_ssd 27251.592177
osd.67 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.67 basic osd_mclock_max_capacity_iops_ssd 28428.290504
osd.68 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.68 basic osd_mclock_max_capacity_iops_ssd 30541.378615
osd.69 basic osd_mclock_max_capacity_iops_ssd 29164.568681
osd.7 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.7 basic osd_mclock_max_capacity_iops_ssd 35788.435091
osd.70 basic osd_mclock_max_capacity_iops_ssd 27050.432719
osd.71 basic osd_mclock_max_capacity_iops_ssd 27615.938915
osd.72 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.72 basic osd_mclock_max_capacity_iops_ssd 29912.870688
osd.73 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.73 basic osd_mclock_max_capacity_iops_ssd 30750.315068
osd.74 basic osd_mclock_max_capacity_iops_hdd 274.216582
osd.74 basic osd_mclock_max_capacity_iops_ssd 31296.317898
osd.75 basic osd_mclock_max_capacity_iops_ssd 31330.904882
osd.76 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.76 basic osd_mclock_max_capacity_iops_ssd 29541.036281
osd.77 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.77 basic osd_mclock_max_capacity_iops_ssd 27755.161947
osd.78 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.78 basic osd_mclock_max_capacity_iops_ssd 30181.440559
osd.79 basic osd_mclock_max_capacity_iops_ssd 28820.911162
osd.8 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.8 basic osd_mclock_max_capacity_iops_ssd 32704.310615
osd.80 basic osd_mclock_max_capacity_iops_ssd 30604.883487
osd.81 basic osd_mclock_max_capacity_iops_ssd 26188.393507
osd.82 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.82 basic osd_mclock_max_capacity_iops_ssd 27407.424799
osd.83 basic osd_mclock_max_capacity_iops_hdd 271.607604
osd.83 basic osd_mclock_max_capacity_iops_ssd 29270.002337
osd.84 basic osd_mclock_max_capacity_iops_ssd 29874.802768
osd.85 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.85 basic osd_mclock_max_capacity_iops_ssd 33267.445310
osd.86 basic osd_mclock_max_capacity_iops_ssd 26647.075973
osd.87 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.87 basic osd_mclock_max_capacity_iops_ssd 23232.299984
osd.88 basic osd_mclock_max_capacity_iops_ssd 30225.432154
osd.89 basic osd_mclock_max_capacity_iops_ssd 26223.229524
osd.9 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.9 basic osd_mclock_max_capacity_iops_hdd 265.536539
osd.9 basic osd_mclock_max_capacity_iops_ssd 35082.146366
osd.90 basic osd_mclock_max_capacity_iops_ssd 28870.253846
osd.91 basic osd_mclock_max_capacity_iops_ssd 28370.754122
osd.92 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.92 basic osd_mclock_max_capacity_iops_ssd 25124.799906
osd.93 basic osd_mclock_max_capacity_iops_ssd 26721.773615
osd.94 basic osd_mclock_max_capacity_iops_ssd 28325.510268
osd.95 basic osd_mclock_max_capacity_iops_ssd 27777.045030
osd.96 basic osd_mclock_max_capacity_iops_ssd 28368.353309
osd.97 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.97 basic osd_mclock_max_capacity_iops_ssd 22786.649038
osd.98 basic osd_mclock_max_capacity_iops_ssd 29639.368028
osd.99 basic osd_mclock_max_capacity_iops_ssd 27038.486473
mds.k8s-it-infra-cephfs basic mds_join_fs k8s-it-infra-cephfs
mds.k8s-prod-cephfs basic mds_join_fs k8s-prod-cephfs
mds.phl.prod basic mds_join_fs phl.prod
client advanced client_asio_thread_count 4
client advanced rbd_op_threads 2
client.crash basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
client.rgw.phl.pl-host01n.titvzo basic rgw_frontends beast ssl_port=443 ssl_certificate=config://rgw/cert/rgw.phl *
client.rgw.phl.pl-host05n.frnpbx basic rgw_frontends beast ssl_port=443 ssl_certificate=config://rgw/cert/rgw.phl *
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 307.39172 root default
-19 83.83411 host pl-host01n
130 nvme 3.49309 osd.130 up 1.00000 1.00000
131 nvme 3.49309 osd.131 up 1.00000 1.00000
132 nvme 3.49309 osd.132 up 1.00000 1.00000
133 nvme 3.49309 osd.133 up 1.00000 1.00000
134 nvme 3.49309 osd.134 up 1.00000 1.00000
135 nvme 3.49309 osd.135 up 1.00000 1.00000
136 nvme 3.49309 osd.136 up 1.00000 1.00000
137 nvme 3.49309 osd.137 up 1.00000 1.00000
15 ssd 3.49309 osd.15 up 1.00000 1.00000
32 ssd 3.49309 osd.32 up 1.00000 1.00000
34 ssd 3.49309 osd.34 up 1.00000 1.00000
38 ssd 3.49309 osd.38 up 1.00000 1.00000
42 ssd 3.49309 osd.42 up 1.00000 1.00000
76 ssd 3.49309 osd.76 up 1.00000 1.00000
101 ssd 3.49309 osd.101 up 1.00000 1.00000
105 ssd 3.49309 osd.105 up 1.00000 1.00000
109 ssd 3.49309 osd.109 up 1.00000 1.00000
113 ssd 3.49309 osd.113 up 1.00000 1.00000
117 ssd 3.49309 osd.117 up 1.00000 1.00000
119 ssd 3.49309 osd.119 up 1.00000 1.00000
124 ssd 3.49309 osd.124 up 1.00000 1.00000
127 ssd 3.49309 osd.127 up 1.00000 1.00000
138 ssd 3.49309 osd.138 up 1.00000 1.00000
139 ssd 3.49309 osd.139 up 1.00000 1.00000
-25 83.83411 host pl-host02n
0 nvme 3.49309 osd.0 up 1.00000 1.00000
40 nvme 3.49309 osd.40 up 1.00000 1.00000
47 nvme 3.49309 osd.47 up 1.00000 1.00000
52 nvme 3.49309 osd.52 up 1.00000 1.00000
57 nvme 3.49309 osd.57 up 1.00000 1.00000
62 nvme 3.49309 osd.62 up 1.00000 1.00000
67 nvme 3.49309 osd.67 up 1.00000 1.00000
72 nvme 3.49309 osd.72 up 1.00000 1.00000
92 nvme 3.49309 osd.92 up 1.00000 1.00000
7 ssd 3.49309 osd.7 up 1.00000 1.00000
10 ssd 3.49309 osd.10 up 1.00000 1.00000
11 ssd 3.49309 osd.11 up 1.00000 1.00000
17 ssd 3.49309 osd.17 up 1.00000 1.00000
20 ssd 3.49309 osd.20 up 1.00000 1.00000
23 ssd 3.49309 osd.23 up 1.00000 1.00000
25 ssd 3.49309 osd.25 up 1.00000 1.00000
26 ssd 3.49309 osd.26 up 1.00000 1.00000
27 ssd 3.49309 osd.27 up 1.00000 1.00000
28 ssd 3.49309 osd.28 up 1.00000 1.00000
30 ssd 3.49309 osd.30 up 1.00000 1.00000
77 ssd 3.49309 osd.77 up 1.00000 1.00000
82 ssd 3.49309 osd.82 up 1.00000 1.00000
87 ssd 3.49309 osd.87 up 1.00000 1.00000
97 ssd 3.49309 osd.97 up 1.00000 1.00000
-22 38.42397 host pl-host03n
3 ssd 3.49309 osd.3 up 1.00000 1.00000
4 ssd 3.49309 osd.4 up 1.00000 1.00000
6 ssd 3.49309 osd.6 up 1.00000 1.00000
8 ssd 3.49309 osd.8 up 1.00000 1.00000
13 ssd 3.49309 osd.13 up 1.00000 1.00000
18 ssd 3.49309 osd.18 up 1.00000 1.00000
59 ssd 3.49309 osd.59 up 1.00000 1.00000
66 ssd 3.49309 osd.66 up 1.00000 1.00000
102 ssd 3.49309 osd.102 up 1.00000 1.00000
106 ssd 3.49309 osd.106 up 1.00000 1.00000
110 ssd 3.49309 osd.110 up 1.00000 1.00000
-28 20.95853 host pl-host04n
48 nvme 3.49309 osd.48 up 1.00000 1.00000
5 ssd 3.49309 osd.5 up 1.00000 1.00000
9 ssd 3.49309 osd.9 up 1.00000 1.00000
21 ssd 3.49309 osd.21 up 1.00000 1.00000
36 ssd 3.49309 osd.36 up 1.00000 1.00000
43 ssd 3.49309 osd.43 up 1.00000 1.00000
-31 80.34102 host pl-host05n
24 nvme 3.49309 osd.24 up 1.00000 1.00000
45 nvme 3.49309 osd.45 up 1.00000 1.00000
49 nvme 3.49309 osd.49 up 1.00000 1.00000
55 nvme 3.49309 osd.55 up 1.00000 1.00000
60 nvme 3.49309 osd.60 up 1.00000 1.00000
65 nvme 3.49309 osd.65 up 1.00000 1.00000
70 nvme 3.49309 osd.70 up 1.00000 1.00000
85 nvme 3.49309 osd.85 up 1.00000 1.00000
90 nvme 3.49309 osd.90 up 1.00000 1.00000
1 ssd 3.49309 osd.1 up 1.00000 1.00000
14 ssd 3.49309 osd.14 up 1.00000 1.00000
31 ssd 3.49309 osd.31 up 1.00000 1.00000
35 ssd 3.49309 osd.35 up 1.00000 1.00000
37 ssd 3.49309 osd.37 up 1.00000 1.00000
39 ssd 3.49309 osd.39 up 1.00000 1.00000
41 ssd 3.49309 osd.41 up 1.00000 1.00000
80 ssd 3.49309 osd.80 up 1.00000 1.00000
95 ssd 3.49309 osd.95 up 1.00000 1.00000
100 ssd 3.49309 osd.100 up 1.00000 1.00000
108 ssd 3.49309 osd.108 up 1.00000 1.00000
112 ssd 3.49309 osd.112 up 1.00000 1.00000
116 ssd 3.49309 osd.116 up 1.00000 1.00000
125 ssd 3.49309 osd.125 up 1.00000 1.00000
On Jun 19, 2025, at 3:25 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
hare a 'ceph config dump' (remove any passwords / confidential information from the output)
participants (3)
-
Anthony D'Atri
-
Devender Singh
-
Frédéric Nass