Thanks for your reply.. There were multiple OSD’s failure on different hosts and had to zap and read. Which lead us to this issue… and upgrade was running too.. Now I see three options for upgrade “Back, Pause & Stop” — is back for rollback? And which one I should use the best option…. Tried restating osd, but it seems dam slow and few are not coming up….and then need zap and add again… What to do with incomplete pigs? Here are the details/… Sorry for the wrong detail in previous emails… correct are below root@phl-host03n:~# ceph -s cluster: id: a0bd51e8-4dfc-11ee-b5a9-3b06e501a0ac health: HEALTH_WARN 8 OSD(s) experiencing slow operations in BlueStore Failed to apply 2 service(s): osd.all-available-devices,osd.iops_optimized 1 failed cephadm daemon(s) failed to probe daemons or devices noscrub,nodeep-scrub flag(s) set Degraded data redundancy: 1739710/39361366 objects degraded (4.420%), 92 pgs degraded, 103 pgs undersized 127 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum phl-host04n,phl-host02n,phl-host01n,phl-host05n,phl-host03n (age 32h) mgr: phl-host01n.pkkztj(active, since 110m), standbys: phl-host02n.wokbnx mds: 3/3 daemons up, 2 standby osd: 88 osds: 88 up (since 2h), 88 in (since 34m); 182 remapped pgs flags noscrub,nodeep-scrub rbd-mirror: 2 daemons active (2 hosts) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 3/3 healthy pools: 23 pools, 1169 pgs objects: 9.22M objects, 32 TiB usage: 68 TiB used, 239 TiB / 307 TiB avail pgs: 1739710/39361366 objects degraded (4.420%) 3592355/39361366 objects misplaced (9.127%) 982 active+clean 85 active+undersized+degraded+remapped+backfill_wait 78 active+remapped+backfill_wait 10 active+undersized+remapped+backfill_wait 7 active+undersized+degraded+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+remapped+backfilling 1 active+undersized+remapped+backfilling io: client: 140 MiB/s rd, 9.6 MiB/s wr, 528 op/s rd, 379 op/s wr recovery: 104 MiB/s, 27 objects/s root@phl-host03n:~# ceph orch upgrade status { "target_image": "quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon" ], "progress": "74/112 daemons upgraded", "message": "", "is_paused": false } # ceph config dump WHO MASK LEVEL OPTION VALUE RO global advanced client_reconnect_stale true global advanced cluster_network 10.104.5.0/24 * global basic container_image quay.io/ceph/ceph@sha256:1b9158ce28975f95def6a0ad459fa19f1336506074267a4b47c1bd914a00fec0 * global basic log_to_file true global basic mds_cache_memory_limit 10589934592 global advanced mgr/devicehealth/self_heal true global advanced mon_cluster_log_to_file true global advanced osd_pool_default_pg_autoscale_mode on global dev rbd_mirror_memory_base 1005306368 global advanced rbd_skip_partial_discard false mon advanced auth_allow_insecure_global_id_reclaim false mon basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * mon advanced mon_allow_pool_delete true mon advanced mon_mds_skip_sanity true mon advanced public_network 10.104.1.0/24 * mgr basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * mgr advanced mgr/cephadm/container_init True * mgr advanced mgr/cephadm/migration_current 7 * mgr advanced mgr/cephadm/warn_on_stray_daemons/warn_on_stray_hosts false * mgr advanced mgr/cephadm/warn_on_stray_hosts false * mgr advanced mgr/cephadm/warn_stray_daemons/warn_on_stray_hosts false * mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://pl-host04n.phl.example.com.org:9093 * mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false * mgr advanced mgr/dashboard/GRAFANA_API_URL https://pl-host02n.phl.example.com.org:3000 * mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://pl-host04n.phl.example.com.org:9095 * mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY * mgr advanced mgr/dashboard/RGW_API_SECRET_KEY * mgr advanced mgr/dashboard/ssl_server_port 8443 * mgr advanced mgr/mds_autoscaler/log_level warning mgr advanced mgr/mds_autoscaler/log_to_cluster true mgr advanced mgr/mds_autoscaler/log_to_file true mgr advanced mgr/orchestrator/orchestrator cephadm mgr advanced mgr/selftest/pl-host04n.tvzwes/testkey foo * mgr advanced mgr/selftest/rwoption1 1234 * mgr advanced mgr/selftest/rwoption2 10 * mgr advanced mgr/selftest/rwoption3 1.500000 * mgr advanced mgr/selftest/rwoption4 foo * mgr advanced mgr/selftest/rwoption5 false * mgr advanced mgr/selftest/testkey test * mgr advanced mgr/telemetry/channel_basic false * mgr advanced mgr/telemetry/channel_crash false * mgr advanced mgr/telemetry/channel_device false * mgr advanced mgr/telemetry/contact test@test.com * mgr advanced mgr/telemetry/interval 60 * mgr advanced mgr/telemetry/leaderboard true * mgr advanced mgr/zabbix/identifier ceph * mgr advanced mgr/zabbix/log_to_cluster false mgr advanced mgr/zabbix/log_to_file false mgr advanced mgr/zabbix/zabbix_host 10.0.2.110 * osd advanced bdev_aio_poll_ms 200 osd advanced bdev_ioring false osd advanced bluestore_throttle_bytes 134217728 osd advanced ms_async_op_threads 4 osd advanced osd_heartbeat_grace 70 osd advanced osd_max_backfills 1 osd advanced osd_mclock_override_recovery_settings true osd host:phl-converged01n basic osd_memory_target 19352618016 osd host:phl-converged02n basic osd_memory_target 31346288264 osd host:phl-converged03n basic osd_memory_target 19804699313 osd host:phl-converged04n basic osd_memory_target 18593027461 osd host:phl-converged05n basic osd_memory_target 20606720591 osd host:pl-host01n basic osd_memory_target 15058518075 osd host:pl-host02n basic osd_memory_target 14086545280 osd host:pl-host03n basic osd_memory_target 33342959038 osd host:pl-host04n basic osd_memory_target 53122886860 osd host:pl-host05n basic osd_memory_target 16313721223 osd advanced osd_memory_target_autotune true osd advanced osd_recovery_max_active_ssd 1 osd advanced osd_recovery_max_single_start 1 osd advanced osd_recovery_sleep 0.000000 osd advanced osd_requested_scrub_priority 1 osd advanced osd_scrub_disable_reservation_queuing true osd advanced osd_scrub_sleep 0.000000 osd advanced osd_snap_trim_cost 262144 osd.0 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.0 basic osd_mclock_max_capacity_iops_ssd 37869.100179 osd.1 basic osd_mclock_max_capacity_iops_ssd 38610.482854 osd.10 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.10 basic osd_mclock_max_capacity_iops_ssd 35297.387846 osd.100 basic osd_mclock_max_capacity_iops_ssd 29091.307131 osd.101 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.101 basic osd_mclock_max_capacity_iops_ssd 27963.050594 osd.102 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.102 basic osd_mclock_max_capacity_iops_ssd 31172.711483 osd.103 basic osd_mclock_max_capacity_iops_ssd 28664.146986 osd.104 basic osd_mclock_max_capacity_iops_ssd 14405.473296 osd.105 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.105 basic osd_mclock_max_capacity_iops_ssd 28291.779421 osd.106 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.106 basic osd_mclock_max_capacity_iops_ssd 30678.497618 osd.107 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.107 basic osd_mclock_max_capacity_iops_ssd 26799.726954 osd.108 basic osd_mclock_max_capacity_iops_ssd 27523.880430 osd.109 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.109 basic osd_mclock_max_capacity_iops_ssd 27819.225848 osd.11 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.11 basic osd_mclock_max_capacity_iops_ssd 35852.830870 osd.110 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.110 basic osd_mclock_max_capacity_iops_ssd 28671.008445 osd.111 basic osd_mclock_max_capacity_iops_ssd 28610.088976 osd.112 basic osd_mclock_max_capacity_iops_ssd 27462.279164 osd.113 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.113 basic osd_mclock_max_capacity_iops_ssd 25652.864986 osd.114 basic osd_mclock_max_capacity_iops_ssd 27469.642657 osd.115 basic osd_mclock_max_capacity_iops_ssd 26599.443843 osd.116 basic osd_mclock_max_capacity_iops_ssd 28159.154679 osd.117 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.117 basic osd_mclock_max_capacity_iops_ssd 30631.799864 osd.118 basic osd_mclock_max_capacity_iops_ssd 26888.868629 osd.119 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.119 basic osd_mclock_max_capacity_iops_ssd 27837.170185 osd.12 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.12 basic osd_mclock_max_capacity_iops_ssd 36523.110430 osd.120 basic osd_mclock_max_capacity_iops_ssd 25156.699403 osd.121 basic osd_mclock_max_capacity_iops_ssd 16854.236813 osd.122 basic osd_mclock_max_capacity_iops_ssd 29425.668046 osd.123 basic osd_mclock_max_capacity_iops_ssd 28424.967707 osd.124 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.124 basic osd_mclock_max_capacity_iops_ssd 30490.214968 osd.125 basic osd_mclock_max_capacity_iops_ssd 29875.742604 osd.126 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.126 basic osd_mclock_max_capacity_iops_ssd 29827.105097 osd.127 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.127 basic osd_mclock_max_capacity_iops_ssd 28899.535735 osd.128 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.128 basic osd_mclock_max_capacity_iops_ssd 27569.775057 osd.129 basic osd_mclock_max_capacity_iops_ssd 32820.522874 osd.13 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.13 basic osd_mclock_max_capacity_iops_ssd 38260.450852 osd.130 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.130 basic osd_mclock_max_capacity_iops_ssd 33740.656917 osd.131 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.131 basic osd_mclock_max_capacity_iops_ssd 30837.814068 osd.132 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.132 basic osd_mclock_max_capacity_iops_ssd 32357.136152 osd.133 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.133 basic osd_mclock_max_capacity_iops_ssd 33199.681376 osd.134 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.134 basic osd_mclock_max_capacity_iops_ssd 32701.919940 osd.135 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.135 basic osd_mclock_max_capacity_iops_ssd 31471.381829 osd.136 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.136 basic osd_mclock_max_capacity_iops_ssd 36040.644043 osd.137 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.137 basic osd_mclock_max_capacity_iops_ssd 34135.109403 osd.138 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.138 basic osd_mclock_max_capacity_iops_ssd 28739.646099 osd.139 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.139 basic osd_mclock_max_capacity_iops_ssd 28611.027050 osd.14 basic osd_mclock_max_capacity_iops_ssd 32587.786337 osd.15 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.15 basic osd_mclock_max_capacity_iops_ssd 33930.129246 osd.16 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.16 basic osd_mclock_max_capacity_iops_ssd 37610.362437 osd.17 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.17 basic osd_mclock_max_capacity_iops_ssd 22609.652689 osd.18 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.18 basic osd_mclock_max_capacity_iops_ssd 29116.630263 osd.19 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.19 basic osd_mclock_max_capacity_iops_ssd 34639.127606 osd.2 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.2 basic osd_mclock_max_capacity_iops_ssd 32777.963192 osd.20 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.20 basic osd_mclock_max_capacity_iops_ssd 33042.683283 osd.21 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.21 basic osd_mclock_max_capacity_iops_ssd 34608.069667 osd.22 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.22 basic osd_mclock_max_capacity_iops_ssd 33672.786827 osd.23 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.23 basic osd_mclock_max_capacity_iops_ssd 37253.942367 osd.24 basic osd_mclock_max_capacity_iops_ssd 32010.520279 osd.25 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.25 basic osd_mclock_max_capacity_iops_ssd 18731.103127 osd.26 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.26 basic osd_mclock_max_capacity_iops_ssd 33844.524796 osd.27 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.27 basic osd_mclock_max_capacity_iops_ssd 33592.404677 osd.28 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.28 basic osd_mclock_max_capacity_iops_ssd 34945.380022 osd.29 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.29 basic osd_mclock_max_capacity_iops_ssd 33041.812034 osd.3 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.3 basic osd_mclock_max_capacity_iops_ssd 30526.239088 osd.30 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.30 basic osd_mclock_max_capacity_iops_ssd 33849.746190 osd.31 basic osd_mclock_max_capacity_iops_ssd 31073.672124 osd.32 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.32 basic osd_mclock_max_capacity_iops_ssd 34475.463692 osd.33 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.33 basic osd_mclock_max_capacity_iops_ssd 35294.129689 osd.34 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.34 basic osd_mclock_max_capacity_iops_ssd 33500.582011 osd.35 basic osd_mclock_max_capacity_iops_ssd 33937.027809 osd.36 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.36 basic osd_mclock_max_capacity_iops_ssd 33363.104343 osd.37 basic osd_mclock_max_capacity_iops_ssd 33136.270826 osd.38 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.38 basic osd_mclock_max_capacity_iops_ssd 34327.772673 osd.39 basic osd_mclock_max_capacity_iops_ssd 36197.080652 osd.4 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.4 basic osd_mclock_max_capacity_iops_ssd 31292.480847 osd.40 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.40 basic osd_mclock_max_capacity_iops_ssd 30200.043883 osd.41 basic osd_mclock_max_capacity_iops_ssd 28660.094990 osd.42 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.42 basic osd_mclock_max_capacity_iops_ssd 29042.271559 osd.43 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.43 basic osd_mclock_max_capacity_iops_ssd 29802.985872 osd.44 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.44 basic osd_mclock_max_capacity_iops_ssd 29872.248343 osd.45 basic osd_mclock_max_capacity_iops_ssd 27048.306734 osd.46 basic osd_mclock_max_capacity_iops_ssd 27445.922772 osd.47 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.47 basic osd_mclock_max_capacity_iops_ssd 29463.719353 osd.48 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.48 basic osd_mclock_max_capacity_iops_ssd 27648.479917 osd.49 basic osd_mclock_max_capacity_iops_ssd 26156.443046 osd.5 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.5 basic osd_mclock_max_capacity_iops_ssd 31042.919796 osd.50 basic osd_mclock_max_capacity_iops_ssd 27126.914926 osd.51 basic osd_mclock_max_capacity_iops_ssd 24087.063752 osd.52 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.52 basic osd_mclock_max_capacity_iops_ssd 29446.685299 osd.53 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.53 basic osd_mclock_max_capacity_iops_ssd 30223.321026 osd.54 basic osd_mclock_max_capacity_iops_ssd 29041.708424 osd.55 basic osd_mclock_max_capacity_iops_ssd 31812.937889 osd.56 basic osd_mclock_max_capacity_iops_ssd 28114.742336 osd.57 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.57 basic osd_mclock_max_capacity_iops_ssd 28757.116835 osd.58 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.58 basic osd_mclock_max_capacity_iops_ssd 29569.415786 osd.59 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.59 basic osd_mclock_max_capacity_iops_ssd 29009.310857 osd.6 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.6 basic osd_mclock_max_capacity_iops_ssd 35515.197243 osd.60 basic osd_mclock_max_capacity_iops_ssd 27353.095641 osd.61 basic osd_mclock_max_capacity_iops_ssd 28033.751890 osd.62 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.62 basic osd_mclock_max_capacity_iops_ssd 28270.316050 osd.63 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.63 basic osd_mclock_max_capacity_iops_ssd 29194.558053 osd.64 basic osd_mclock_max_capacity_iops_ssd 25543.077545 osd.65 basic osd_mclock_max_capacity_iops_ssd 29257.957631 osd.66 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.66 basic osd_mclock_max_capacity_iops_ssd 27251.592177 osd.67 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.67 basic osd_mclock_max_capacity_iops_ssd 28428.290504 osd.68 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.68 basic osd_mclock_max_capacity_iops_ssd 30541.378615 osd.69 basic osd_mclock_max_capacity_iops_ssd 29164.568681 osd.7 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.7 basic osd_mclock_max_capacity_iops_ssd 35788.435091 osd.70 basic osd_mclock_max_capacity_iops_ssd 27050.432719 osd.71 basic osd_mclock_max_capacity_iops_ssd 27615.938915 osd.72 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.72 basic osd_mclock_max_capacity_iops_ssd 29912.870688 osd.73 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.73 basic osd_mclock_max_capacity_iops_ssd 30750.315068 osd.74 basic osd_mclock_max_capacity_iops_hdd 274.216582 osd.74 basic osd_mclock_max_capacity_iops_ssd 31296.317898 osd.75 basic osd_mclock_max_capacity_iops_ssd 31330.904882 osd.76 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.76 basic osd_mclock_max_capacity_iops_ssd 29541.036281 osd.77 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.77 basic osd_mclock_max_capacity_iops_ssd 27755.161947 osd.78 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.78 basic osd_mclock_max_capacity_iops_ssd 30181.440559 osd.79 basic osd_mclock_max_capacity_iops_ssd 28820.911162 osd.8 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.8 basic osd_mclock_max_capacity_iops_ssd 32704.310615 osd.80 basic osd_mclock_max_capacity_iops_ssd 30604.883487 osd.81 basic osd_mclock_max_capacity_iops_ssd 26188.393507 osd.82 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.82 basic osd_mclock_max_capacity_iops_ssd 27407.424799 osd.83 basic osd_mclock_max_capacity_iops_hdd 271.607604 osd.83 basic osd_mclock_max_capacity_iops_ssd 29270.002337 osd.84 basic osd_mclock_max_capacity_iops_ssd 29874.802768 osd.85 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.85 basic osd_mclock_max_capacity_iops_ssd 33267.445310 osd.86 basic osd_mclock_max_capacity_iops_ssd 26647.075973 osd.87 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.87 basic osd_mclock_max_capacity_iops_ssd 23232.299984 osd.88 basic osd_mclock_max_capacity_iops_ssd 30225.432154 osd.89 basic osd_mclock_max_capacity_iops_ssd 26223.229524 osd.9 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.9 basic osd_mclock_max_capacity_iops_hdd 265.536539 osd.9 basic osd_mclock_max_capacity_iops_ssd 35082.146366 osd.90 basic osd_mclock_max_capacity_iops_ssd 28870.253846 osd.91 basic osd_mclock_max_capacity_iops_ssd 28370.754122 osd.92 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.92 basic osd_mclock_max_capacity_iops_ssd 25124.799906 osd.93 basic osd_mclock_max_capacity_iops_ssd 26721.773615 osd.94 basic osd_mclock_max_capacity_iops_ssd 28325.510268 osd.95 basic osd_mclock_max_capacity_iops_ssd 27777.045030 osd.96 basic osd_mclock_max_capacity_iops_ssd 28368.353309 osd.97 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * osd.97 basic osd_mclock_max_capacity_iops_ssd 22786.649038 osd.98 basic osd_mclock_max_capacity_iops_ssd 29639.368028 osd.99 basic osd_mclock_max_capacity_iops_ssd 27038.486473 mds.k8s-it-infra-cephfs basic mds_join_fs k8s-it-infra-cephfs mds.k8s-prod-cephfs basic mds_join_fs k8s-prod-cephfs mds.phl.prod basic mds_join_fs phl.prod client advanced client_asio_thread_count 4 client advanced rbd_op_threads 2 client.crash basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 * client.rgw.phl.pl-host01n.titvzo basic rgw_frontends beast ssl_port=443 ssl_certificate=config://rgw/cert/rgw.phl * client.rgw.phl.pl-host05n.frnpbx basic rgw_frontends beast ssl_port=443 ssl_certificate=config://rgw/cert/rgw.phl * # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 307.39172 root default -19 83.83411 host pl-host01n 130 nvme 3.49309 osd.130 up 1.00000 1.00000 131 nvme 3.49309 osd.131 up 1.00000 1.00000 132 nvme 3.49309 osd.132 up 1.00000 1.00000 133 nvme 3.49309 osd.133 up 1.00000 1.00000 134 nvme 3.49309 osd.134 up 1.00000 1.00000 135 nvme 3.49309 osd.135 up 1.00000 1.00000 136 nvme 3.49309 osd.136 up 1.00000 1.00000 137 nvme 3.49309 osd.137 up 1.00000 1.00000 15 ssd 3.49309 osd.15 up 1.00000 1.00000 32 ssd 3.49309 osd.32 up 1.00000 1.00000 34 ssd 3.49309 osd.34 up 1.00000 1.00000 38 ssd 3.49309 osd.38 up 1.00000 1.00000 42 ssd 3.49309 osd.42 up 1.00000 1.00000 76 ssd 3.49309 osd.76 up 1.00000 1.00000 101 ssd 3.49309 osd.101 up 1.00000 1.00000 105 ssd 3.49309 osd.105 up 1.00000 1.00000 109 ssd 3.49309 osd.109 up 1.00000 1.00000 113 ssd 3.49309 osd.113 up 1.00000 1.00000 117 ssd 3.49309 osd.117 up 1.00000 1.00000 119 ssd 3.49309 osd.119 up 1.00000 1.00000 124 ssd 3.49309 osd.124 up 1.00000 1.00000 127 ssd 3.49309 osd.127 up 1.00000 1.00000 138 ssd 3.49309 osd.138 up 1.00000 1.00000 139 ssd 3.49309 osd.139 up 1.00000 1.00000 -25 83.83411 host pl-host02n 0 nvme 3.49309 osd.0 up 1.00000 1.00000 40 nvme 3.49309 osd.40 up 1.00000 1.00000 47 nvme 3.49309 osd.47 up 1.00000 1.00000 52 nvme 3.49309 osd.52 up 1.00000 1.00000 57 nvme 3.49309 osd.57 up 1.00000 1.00000 62 nvme 3.49309 osd.62 up 1.00000 1.00000 67 nvme 3.49309 osd.67 up 1.00000 1.00000 72 nvme 3.49309 osd.72 up 1.00000 1.00000 92 nvme 3.49309 osd.92 up 1.00000 1.00000 7 ssd 3.49309 osd.7 up 1.00000 1.00000 10 ssd 3.49309 osd.10 up 1.00000 1.00000 11 ssd 3.49309 osd.11 up 1.00000 1.00000 17 ssd 3.49309 osd.17 up 1.00000 1.00000 20 ssd 3.49309 osd.20 up 1.00000 1.00000 23 ssd 3.49309 osd.23 up 1.00000 1.00000 25 ssd 3.49309 osd.25 up 1.00000 1.00000 26 ssd 3.49309 osd.26 up 1.00000 1.00000 27 ssd 3.49309 osd.27 up 1.00000 1.00000 28 ssd 3.49309 osd.28 up 1.00000 1.00000 30 ssd 3.49309 osd.30 up 1.00000 1.00000 77 ssd 3.49309 osd.77 up 1.00000 1.00000 82 ssd 3.49309 osd.82 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 97 ssd 3.49309 osd.97 up 1.00000 1.00000 -22 38.42397 host pl-host03n 3 ssd 3.49309 osd.3 up 1.00000 1.00000 4 ssd 3.49309 osd.4 up 1.00000 1.00000 6 ssd 3.49309 osd.6 up 1.00000 1.00000 8 ssd 3.49309 osd.8 up 1.00000 1.00000 13 ssd 3.49309 osd.13 up 1.00000 1.00000 18 ssd 3.49309 osd.18 up 1.00000 1.00000 59 ssd 3.49309 osd.59 up 1.00000 1.00000 66 ssd 3.49309 osd.66 up 1.00000 1.00000 102 ssd 3.49309 osd.102 up 1.00000 1.00000 106 ssd 3.49309 osd.106 up 1.00000 1.00000 110 ssd 3.49309 osd.110 up 1.00000 1.00000 -28 20.95853 host pl-host04n 48 nvme 3.49309 osd.48 up 1.00000 1.00000 5 ssd 3.49309 osd.5 up 1.00000 1.00000 9 ssd 3.49309 osd.9 up 1.00000 1.00000 21 ssd 3.49309 osd.21 up 1.00000 1.00000 36 ssd 3.49309 osd.36 up 1.00000 1.00000 43 ssd 3.49309 osd.43 up 1.00000 1.00000 -31 80.34102 host pl-host05n 24 nvme 3.49309 osd.24 up 1.00000 1.00000 45 nvme 3.49309 osd.45 up 1.00000 1.00000 49 nvme 3.49309 osd.49 up 1.00000 1.00000 55 nvme 3.49309 osd.55 up 1.00000 1.00000 60 nvme 3.49309 osd.60 up 1.00000 1.00000 65 nvme 3.49309 osd.65 up 1.00000 1.00000 70 nvme 3.49309 osd.70 up 1.00000 1.00000 85 nvme 3.49309 osd.85 up 1.00000 1.00000 90 nvme 3.49309 osd.90 up 1.00000 1.00000 1 ssd 3.49309 osd.1 up 1.00000 1.00000 14 ssd 3.49309 osd.14 up 1.00000 1.00000 31 ssd 3.49309 osd.31 up 1.00000 1.00000 35 ssd 3.49309 osd.35 up 1.00000 1.00000 37 ssd 3.49309 osd.37 up 1.00000 1.00000 39 ssd 3.49309 osd.39 up 1.00000 1.00000 41 ssd 3.49309 osd.41 up 1.00000 1.00000 80 ssd 3.49309 osd.80 up 1.00000 1.00000 95 ssd 3.49309 osd.95 up 1.00000 1.00000 100 ssd 3.49309 osd.100 up 1.00000 1.00000 108 ssd 3.49309 osd.108 up 1.00000 1.00000 112 ssd 3.49309 osd.112 up 1.00000 1.00000 116 ssd 3.49309 osd.116 up 1.00000 1.00000 125 ssd 3.49309 osd.125 up 1.00000 1.00000
On Jun 19, 2025, at 3:25 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
hare a 'ceph config dump' (remove any passwords / confidential information from the output)