Hi Dev, Tips, real quick: - stop zapping devices :-) - I've never heard of any 'back' option on the 'ceph orch upgrade' command [1], but 'stop' could be used to allow other operations to proceed, would it be needed at some point. BTW, I will assume that you started the upgrade prior to OSDs failing to start, and not after. - I would advise that you set 'osd_op_queue' to 'wpq', to help with the current situation. Start with a single OSD (ceph osd config set osd.0 osd_op_queue wpq) and keep going with other OSDs if it started and joined the cluster without issues. - If using Podman, SSH to every hosts and run 'df -i' to check for inode exhaustion on /run partition. There's a podman bug [2] in EL 9.5 that was fixed in EL 9.6 by podman-5:5.4.0-9.el9_6.x86_64 . Could also affect other distros. In case of inode exhaustion, no containers would start. Inode exhaustion can happen really fast as the orchestrator is regularly running containers on every hosts for inventory but not only. - Choose one of the '8 OSD(s) experiencing slow operations in BlueStore' and share 'ceph tell osd.X bluefs stats' and 'ceph tell osd.0 bluestore allocator score block'. - Please share the manufacturer/model of SSDs and NVMes. Regards, Frédéric. [1] [ https://docs.ceph.com/en/latest/cephadm/upgrade/ | https://docs.ceph.com/en/latest/cephadm/upgrade/ ] [2] [ https://github.com/containers/podman/issues/25291 | https://github.com/containers/podman/issues/25291 ] ----- Le 20 Juin 25, à 5:36, Devender Singh <devender@netskrt.io> a écrit :
Thanks for your reply.. There were multiple OSD’s failure on different hosts and had to zap and read. Which lead us to this issue… and upgrade was running too.. Now I see three options for upgrade “Back, Pause & Stop” — is back for rollback? And which one I should use the best option…. Tried restating osd, but it seems dam slow and few are not coming up….and then need zap and add again… What to do with incomplete pigs?
Here are the details/…
Sorry for the wrong detail in previous emails… correct are below
root@phl-host03n:~# ceph -s
cluster:
id: a0bd51e8-4dfc-11ee-b5a9-3b06e501a0ac
health: HEALTH_WARN
8 OSD(s) experiencing slow operations in BlueStore
Failed to apply 2 service(s): osd.all-available-devices,osd.iops_optimized
1 failed cephadm daemon(s)
failed to probe daemons or devices
noscrub,nodeep-scrub flag(s) set
Degraded data redundancy: 1739710/39361366 objects degraded (4.420%), 92 pgs degraded, 103 pgs undersized
127 pgs not deep-scrubbed in time
services:
mon: 5 daemons, quorum phl-host04n,phl-host02n,phl-host01n,phl-host05n,phl-host03n (age 32h)
mgr: phl-host01n.pkkztj(active, since 110m), standbys: phl-host02n.wokbnx
mds: 3/3 daemons up, 2 standby
osd: 88 osds: 88 up (since 2h), 88 in (since 34m); 182 remapped pgs
flags noscrub,nodeep-scrub
rbd-mirror: 2 daemons active (2 hosts)
rgw: 2 daemons active (2 hosts, 1 zones)
data:
volumes: 3/3 healthy
pools: 23 pools, 1169 pgs
objects: 9.22M objects, 32 TiB
usage: 68 TiB used, 239 TiB / 307 TiB avail
pgs: 1739710/39361366 objects degraded (4.420%)
3592355/39361366 objects misplaced (9.127%)
982 active+clean
85 active+undersized+degraded+remapped+backfill_wait
78 active+remapped+backfill_wait
10 active+undersized+remapped+backfill_wait
7 active+undersized+degraded+remapped+backfilling
5 active+clean+scrubbing+deep
1 active+remapped+backfilling
1 active+undersized+remapped+backfilling
io:
client: 140 MiB/s rd, 9.6 MiB/s wr, 528 op/s rd, 379 op/s wr
recovery: 104 MiB/s, 27 objects/s
root@phl-host03n:~# ceph orch upgrade status
{
"target_image": "quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6",
"in_progress": true,
"which": "Upgrading all daemon types on all hosts",
"services_complete": [
"crash",
"mgr",
"mon"
],
"progress": "74/112 daemons upgraded",
"message": "",
"is_paused": false
}
# ceph config dump
WHO MASK LEVEL OPTION VALUE RO
global advanced client_reconnect_stale true
global advanced cluster_network 10.104.5.0/24 *
global basic container_image quay.io/ceph/ceph@sha256:1b9158ce28975f95def6a0ad459fa19f1336506074267a4b47c1bd914a00fec0 *
global basic log_to_file true
global basic mds_cache_memory_limit 10589934592
global advanced mgr/devicehealth/self_heal true
global advanced mon_cluster_log_to_file true
global advanced osd_pool_default_pg_autoscale_mode on
global dev rbd_mirror_memory_base 1005306368
global advanced rbd_skip_partial_discard false
mon advanced auth_allow_insecure_global_id_reclaim false
mon basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
mon advanced mon_allow_pool_delete true
mon advanced mon_mds_skip_sanity true
mon advanced public_network 10.104.1.0/24 *
mgr basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
mgr advanced mgr/cephadm/container_init True *
mgr advanced mgr/cephadm/migration_current 7 *
mgr advanced mgr/cephadm/warn_on_stray_daemons/warn_on_stray_hosts false *
mgr advanced mgr/cephadm/warn_on_stray_hosts false *
mgr advanced mgr/cephadm/warn_stray_daemons/warn_on_stray_hosts false *
mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://pl-host04n.phl.example.com.org:9093 *
mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false *
mgr advanced mgr/dashboard/GRAFANA_API_URL https://pl-host02n.phl.example.com.org:3000 *
mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://pl-host04n.phl.example.com.org:9095 *
mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY *
mgr advanced mgr/dashboard/RGW_API_SECRET_KEY *
mgr advanced mgr/dashboard/ssl_server_port 8443 *
mgr advanced mgr/mds_autoscaler/log_level warning
mgr advanced mgr/mds_autoscaler/log_to_cluster true
mgr advanced mgr/mds_autoscaler/log_to_file true
mgr advanced mgr/orchestrator/orchestrator cephadm
mgr advanced mgr/selftest/pl-host04n.tvzwes/testkey foo *
mgr advanced mgr/selftest/rwoption1 1234 *
mgr advanced mgr/selftest/rwoption2 10 *
mgr advanced mgr/selftest/rwoption3 1.500000 *
mgr advanced mgr/selftest/rwoption4 foo *
mgr advanced mgr/selftest/rwoption5 false *
mgr advanced mgr/selftest/testkey test *
mgr advanced mgr/telemetry/channel_basic false *
mgr advanced mgr/telemetry/channel_crash false *
mgr advanced mgr/telemetry/channel_device false *
mgr advanced mgr/telemetry/contact test@test.com *
mgr advanced mgr/telemetry/interval 60 *
mgr advanced mgr/telemetry/leaderboard true *
mgr advanced mgr/zabbix/identifier ceph *
mgr advanced mgr/zabbix/log_to_cluster false
mgr advanced mgr/zabbix/log_to_file false
mgr advanced mgr/zabbix/zabbix_host 10.0.2.110 *
osd advanced bdev_aio_poll_ms 200
osd advanced bdev_ioring false
osd advanced bluestore_throttle_bytes 134217728
osd advanced ms_async_op_threads 4
osd advanced osd_heartbeat_grace 70
osd advanced osd_max_backfills 1
osd advanced osd_mclock_override_recovery_settings true
osd host:phl-converged01n basic osd_memory_target 19352618016
osd host:phl-converged02n basic osd_memory_target 31346288264
osd host:phl-converged03n basic osd_memory_target 19804699313
osd host:phl-converged04n basic osd_memory_target 18593027461
osd host:phl-converged05n basic osd_memory_target 20606720591
osd host:pl-host01n basic osd_memory_target 15058518075
osd host:pl-host02n basic osd_memory_target 14086545280
osd host:pl-host03n basic osd_memory_target 33342959038
osd host:pl-host04n basic osd_memory_target 53122886860
osd host:pl-host05n basic osd_memory_target 16313721223
osd advanced osd_memory_target_autotune true
osd advanced osd_recovery_max_active_ssd 1
osd advanced osd_recovery_max_single_start 1
osd advanced osd_recovery_sleep 0.000000
osd advanced osd_requested_scrub_priority 1
osd advanced osd_scrub_disable_reservation_queuing true
osd advanced osd_scrub_sleep 0.000000
osd advanced osd_snap_trim_cost 262144
osd.0 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.0 basic osd_mclock_max_capacity_iops_ssd 37869.100179
osd.1 basic osd_mclock_max_capacity_iops_ssd 38610.482854
osd.10 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.10 basic osd_mclock_max_capacity_iops_ssd 35297.387846
osd.100 basic osd_mclock_max_capacity_iops_ssd 29091.307131
osd.101 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.101 basic osd_mclock_max_capacity_iops_ssd 27963.050594
osd.102 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.102 basic osd_mclock_max_capacity_iops_ssd 31172.711483
osd.103 basic osd_mclock_max_capacity_iops_ssd 28664.146986
osd.104 basic osd_mclock_max_capacity_iops_ssd 14405.473296
osd.105 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.105 basic osd_mclock_max_capacity_iops_ssd 28291.779421
osd.106 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.106 basic osd_mclock_max_capacity_iops_ssd 30678.497618
osd.107 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.107 basic osd_mclock_max_capacity_iops_ssd 26799.726954
osd.108 basic osd_mclock_max_capacity_iops_ssd 27523.880430
osd.109 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.109 basic osd_mclock_max_capacity_iops_ssd 27819.225848
osd.11 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.11 basic osd_mclock_max_capacity_iops_ssd 35852.830870
osd.110 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.110 basic osd_mclock_max_capacity_iops_ssd 28671.008445
osd.111 basic osd_mclock_max_capacity_iops_ssd 28610.088976
osd.112 basic osd_mclock_max_capacity_iops_ssd 27462.279164
osd.113 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.113 basic osd_mclock_max_capacity_iops_ssd 25652.864986
osd.114 basic osd_mclock_max_capacity_iops_ssd 27469.642657
osd.115 basic osd_mclock_max_capacity_iops_ssd 26599.443843
osd.116 basic osd_mclock_max_capacity_iops_ssd 28159.154679
osd.117 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.117 basic osd_mclock_max_capacity_iops_ssd 30631.799864
osd.118 basic osd_mclock_max_capacity_iops_ssd 26888.868629
osd.119 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.119 basic osd_mclock_max_capacity_iops_ssd 27837.170185
osd.12 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.12 basic osd_mclock_max_capacity_iops_ssd 36523.110430
osd.120 basic osd_mclock_max_capacity_iops_ssd 25156.699403
osd.121 basic osd_mclock_max_capacity_iops_ssd 16854.236813
osd.122 basic osd_mclock_max_capacity_iops_ssd 29425.668046
osd.123 basic osd_mclock_max_capacity_iops_ssd 28424.967707
osd.124 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.124 basic osd_mclock_max_capacity_iops_ssd 30490.214968
osd.125 basic osd_mclock_max_capacity_iops_ssd 29875.742604
osd.126 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.126 basic osd_mclock_max_capacity_iops_ssd 29827.105097
osd.127 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.127 basic osd_mclock_max_capacity_iops_ssd 28899.535735
osd.128 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.128 basic osd_mclock_max_capacity_iops_ssd 27569.775057
osd.129 basic osd_mclock_max_capacity_iops_ssd 32820.522874
osd.13 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.13 basic osd_mclock_max_capacity_iops_ssd 38260.450852
osd.130 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.130 basic osd_mclock_max_capacity_iops_ssd 33740.656917
osd.131 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.131 basic osd_mclock_max_capacity_iops_ssd 30837.814068
osd.132 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.132 basic osd_mclock_max_capacity_iops_ssd 32357.136152
osd.133 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.133 basic osd_mclock_max_capacity_iops_ssd 33199.681376
osd.134 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.134 basic osd_mclock_max_capacity_iops_ssd 32701.919940
osd.135 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.135 basic osd_mclock_max_capacity_iops_ssd 31471.381829
osd.136 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.136 basic osd_mclock_max_capacity_iops_ssd 36040.644043
osd.137 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.137 basic osd_mclock_max_capacity_iops_ssd 34135.109403
osd.138 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.138 basic osd_mclock_max_capacity_iops_ssd 28739.646099
osd.139 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.139 basic osd_mclock_max_capacity_iops_ssd 28611.027050
osd.14 basic osd_mclock_max_capacity_iops_ssd 32587.786337
osd.15 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.15 basic osd_mclock_max_capacity_iops_ssd 33930.129246
osd.16 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.16 basic osd_mclock_max_capacity_iops_ssd 37610.362437
osd.17 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.17 basic osd_mclock_max_capacity_iops_ssd 22609.652689
osd.18 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.18 basic osd_mclock_max_capacity_iops_ssd 29116.630263
osd.19 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.19 basic osd_mclock_max_capacity_iops_ssd 34639.127606
osd.2 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.2 basic osd_mclock_max_capacity_iops_ssd 32777.963192
osd.20 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.20 basic osd_mclock_max_capacity_iops_ssd 33042.683283
osd.21 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.21 basic osd_mclock_max_capacity_iops_ssd 34608.069667
osd.22 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.22 basic osd_mclock_max_capacity_iops_ssd 33672.786827
osd.23 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.23 basic osd_mclock_max_capacity_iops_ssd 37253.942367
osd.24 basic osd_mclock_max_capacity_iops_ssd 32010.520279
osd.25 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.25 basic osd_mclock_max_capacity_iops_ssd 18731.103127
osd.26 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.26 basic osd_mclock_max_capacity_iops_ssd 33844.524796
osd.27 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.27 basic osd_mclock_max_capacity_iops_ssd 33592.404677
osd.28 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.28 basic osd_mclock_max_capacity_iops_ssd 34945.380022
osd.29 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.29 basic osd_mclock_max_capacity_iops_ssd 33041.812034
osd.3 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.3 basic osd_mclock_max_capacity_iops_ssd 30526.239088
osd.30 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.30 basic osd_mclock_max_capacity_iops_ssd 33849.746190
osd.31 basic osd_mclock_max_capacity_iops_ssd 31073.672124
osd.32 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.32 basic osd_mclock_max_capacity_iops_ssd 34475.463692
osd.33 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.33 basic osd_mclock_max_capacity_iops_ssd 35294.129689
osd.34 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.34 basic osd_mclock_max_capacity_iops_ssd 33500.582011
osd.35 basic osd_mclock_max_capacity_iops_ssd 33937.027809
osd.36 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.36 basic osd_mclock_max_capacity_iops_ssd 33363.104343
osd.37 basic osd_mclock_max_capacity_iops_ssd 33136.270826
osd.38 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.38 basic osd_mclock_max_capacity_iops_ssd 34327.772673
osd.39 basic osd_mclock_max_capacity_iops_ssd 36197.080652
osd.4 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.4 basic osd_mclock_max_capacity_iops_ssd 31292.480847
osd.40 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.40 basic osd_mclock_max_capacity_iops_ssd 30200.043883
osd.41 basic osd_mclock_max_capacity_iops_ssd 28660.094990
osd.42 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.42 basic osd_mclock_max_capacity_iops_ssd 29042.271559
osd.43 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.43 basic osd_mclock_max_capacity_iops_ssd 29802.985872
osd.44 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.44 basic osd_mclock_max_capacity_iops_ssd 29872.248343
osd.45 basic osd_mclock_max_capacity_iops_ssd 27048.306734
osd.46 basic osd_mclock_max_capacity_iops_ssd 27445.922772
osd.47 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.47 basic osd_mclock_max_capacity_iops_ssd 29463.719353
osd.48 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.48 basic osd_mclock_max_capacity_iops_ssd 27648.479917
osd.49 basic osd_mclock_max_capacity_iops_ssd 26156.443046
osd.5 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.5 basic osd_mclock_max_capacity_iops_ssd 31042.919796
osd.50 basic osd_mclock_max_capacity_iops_ssd 27126.914926
osd.51 basic osd_mclock_max_capacity_iops_ssd 24087.063752
osd.52 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.52 basic osd_mclock_max_capacity_iops_ssd 29446.685299
osd.53 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.53 basic osd_mclock_max_capacity_iops_ssd 30223.321026
osd.54 basic osd_mclock_max_capacity_iops_ssd 29041.708424
osd.55 basic osd_mclock_max_capacity_iops_ssd 31812.937889
osd.56 basic osd_mclock_max_capacity_iops_ssd 28114.742336
osd.57 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.57 basic osd_mclock_max_capacity_iops_ssd 28757.116835
osd.58 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.58 basic osd_mclock_max_capacity_iops_ssd 29569.415786
osd.59 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.59 basic osd_mclock_max_capacity_iops_ssd 29009.310857
osd.6 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.6 basic osd_mclock_max_capacity_iops_ssd 35515.197243
osd.60 basic osd_mclock_max_capacity_iops_ssd 27353.095641
osd.61 basic osd_mclock_max_capacity_iops_ssd 28033.751890
osd.62 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.62 basic osd_mclock_max_capacity_iops_ssd 28270.316050
osd.63 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.63 basic osd_mclock_max_capacity_iops_ssd 29194.558053
osd.64 basic osd_mclock_max_capacity_iops_ssd 25543.077545
osd.65 basic osd_mclock_max_capacity_iops_ssd 29257.957631
osd.66 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.66 basic osd_mclock_max_capacity_iops_ssd 27251.592177
osd.67 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.67 basic osd_mclock_max_capacity_iops_ssd 28428.290504
osd.68 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.68 basic osd_mclock_max_capacity_iops_ssd 30541.378615
osd.69 basic osd_mclock_max_capacity_iops_ssd 29164.568681
osd.7 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.7 basic osd_mclock_max_capacity_iops_ssd 35788.435091
osd.70 basic osd_mclock_max_capacity_iops_ssd 27050.432719
osd.71 basic osd_mclock_max_capacity_iops_ssd 27615.938915
osd.72 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.72 basic osd_mclock_max_capacity_iops_ssd 29912.870688
osd.73 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.73 basic osd_mclock_max_capacity_iops_ssd 30750.315068
osd.74 basic osd_mclock_max_capacity_iops_hdd 274.216582
osd.74 basic osd_mclock_max_capacity_iops_ssd 31296.317898
osd.75 basic osd_mclock_max_capacity_iops_ssd 31330.904882
osd.76 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.76 basic osd_mclock_max_capacity_iops_ssd 29541.036281
osd.77 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.77 basic osd_mclock_max_capacity_iops_ssd 27755.161947
osd.78 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.78 basic osd_mclock_max_capacity_iops_ssd 30181.440559
osd.79 basic osd_mclock_max_capacity_iops_ssd 28820.911162
osd.8 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.8 basic osd_mclock_max_capacity_iops_ssd 32704.310615
osd.80 basic osd_mclock_max_capacity_iops_ssd 30604.883487
osd.81 basic osd_mclock_max_capacity_iops_ssd 26188.393507
osd.82 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.82 basic osd_mclock_max_capacity_iops_ssd 27407.424799
osd.83 basic osd_mclock_max_capacity_iops_hdd 271.607604
osd.83 basic osd_mclock_max_capacity_iops_ssd 29270.002337
osd.84 basic osd_mclock_max_capacity_iops_ssd 29874.802768
osd.85 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.85 basic osd_mclock_max_capacity_iops_ssd 33267.445310
osd.86 basic osd_mclock_max_capacity_iops_ssd 26647.075973
osd.87 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.87 basic osd_mclock_max_capacity_iops_ssd 23232.299984
osd.88 basic osd_mclock_max_capacity_iops_ssd 30225.432154
osd.89 basic osd_mclock_max_capacity_iops_ssd 26223.229524
osd.9 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.9 basic osd_mclock_max_capacity_iops_hdd 265.536539
osd.9 basic osd_mclock_max_capacity_iops_ssd 35082.146366
osd.90 basic osd_mclock_max_capacity_iops_ssd 28870.253846
osd.91 basic osd_mclock_max_capacity_iops_ssd 28370.754122
osd.92 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.92 basic osd_mclock_max_capacity_iops_ssd 25124.799906
osd.93 basic osd_mclock_max_capacity_iops_ssd 26721.773615
osd.94 basic osd_mclock_max_capacity_iops_ssd 28325.510268
osd.95 basic osd_mclock_max_capacity_iops_ssd 27777.045030
osd.96 basic osd_mclock_max_capacity_iops_ssd 28368.353309
osd.97 basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
osd.97 basic osd_mclock_max_capacity_iops_ssd 22786.649038
osd.98 basic osd_mclock_max_capacity_iops_ssd 29639.368028
osd.99 basic osd_mclock_max_capacity_iops_ssd 27038.486473
mds.k8s-it-infra-cephfs basic mds_join_fs k8s-it-infra-cephfs
mds.k8s-prod-cephfs basic mds_join_fs k8s-prod-cephfs
mds.phl.prod basic mds_join_fs phl.prod
client advanced client_asio_thread_count 4
client advanced rbd_op_threads 2
client.crash basic container_image quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 *
client.rgw.phl.pl-host01n.titvzo basic rgw_frontends beast ssl_port=443 ssl_certificate=config://rgw/cert/rgw.phl *
client.rgw.phl.pl-host05n.frnpbx basic rgw_frontends beast ssl_port=443 ssl_certificate=config://rgw/cert/rgw.phl *
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 307.39172 root default
-19 83.83411 host pl-host01n
130 nvme 3.49309 osd.130 up 1.00000 1.00000
131 nvme 3.49309 osd.131 up 1.00000 1.00000
132 nvme 3.49309 osd.132 up 1.00000 1.00000
133 nvme 3.49309 osd.133 up 1.00000 1.00000
134 nvme 3.49309 osd.134 up 1.00000 1.00000
135 nvme 3.49309 osd.135 up 1.00000 1.00000
136 nvme 3.49309 osd.136 up 1.00000 1.00000
137 nvme 3.49309 osd.137 up 1.00000 1.00000
15 ssd 3.49309 osd.15 up 1.00000 1.00000
32 ssd 3.49309 osd.32 up 1.00000 1.00000
34 ssd 3.49309 osd.34 up 1.00000 1.00000
38 ssd 3.49309 osd.38 up 1.00000 1.00000
42 ssd 3.49309 osd.42 up 1.00000 1.00000
76 ssd 3.49309 osd.76 up 1.00000 1.00000
101 ssd 3.49309 osd.101 up 1.00000 1.00000
105 ssd 3.49309 osd.105 up 1.00000 1.00000
109 ssd 3.49309 osd.109 up 1.00000 1.00000
113 ssd 3.49309 osd.113 up 1.00000 1.00000
117 ssd 3.49309 osd.117 up 1.00000 1.00000
119 ssd 3.49309 osd.119 up 1.00000 1.00000
124 ssd 3.49309 osd.124 up 1.00000 1.00000
127 ssd 3.49309 osd.127 up 1.00000 1.00000
138 ssd 3.49309 osd.138 up 1.00000 1.00000
139 ssd 3.49309 osd.139 up 1.00000 1.00000
-25 83.83411 host pl-host02n
0 nvme 3.49309 osd.0 up 1.00000 1.00000
40 nvme 3.49309 osd.40 up 1.00000 1.00000
47 nvme 3.49309 osd.47 up 1.00000 1.00000
52 nvme 3.49309 osd.52 up 1.00000 1.00000
57 nvme 3.49309 osd.57 up 1.00000 1.00000
62 nvme 3.49309 osd.62 up 1.00000 1.00000
67 nvme 3.49309 osd.67 up 1.00000 1.00000
72 nvme 3.49309 osd.72 up 1.00000 1.00000
92 nvme 3.49309 osd.92 up 1.00000 1.00000
7 ssd 3.49309 osd.7 up 1.00000 1.00000
10 ssd 3.49309 osd.10 up 1.00000 1.00000
11 ssd 3.49309 osd.11 up 1.00000 1.00000
17 ssd 3.49309 osd.17 up 1.00000 1.00000
20 ssd 3.49309 osd.20 up 1.00000 1.00000
23 ssd 3.49309 osd.23 up 1.00000 1.00000
25 ssd 3.49309 osd.25 up 1.00000 1.00000
26 ssd 3.49309 osd.26 up 1.00000 1.00000
27 ssd 3.49309 osd.27 up 1.00000 1.00000
28 ssd 3.49309 osd.28 up 1.00000 1.00000
30 ssd 3.49309 osd.30 up 1.00000 1.00000
77 ssd 3.49309 osd.77 up 1.00000 1.00000
82 ssd 3.49309 osd.82 up 1.00000 1.00000
87 ssd 3.49309 osd.87 up 1.00000 1.00000
97 ssd 3.49309 osd.97 up 1.00000 1.00000
-22 38.42397 host pl-host03n
3 ssd 3.49309 osd.3 up 1.00000 1.00000
4 ssd 3.49309 osd.4 up 1.00000 1.00000
6 ssd 3.49309 osd.6 up 1.00000 1.00000
8 ssd 3.49309 osd.8 up 1.00000 1.00000
13 ssd 3.49309 osd.13 up 1.00000 1.00000
18 ssd 3.49309 osd.18 up 1.00000 1.00000
59 ssd 3.49309 osd.59 up 1.00000 1.00000
66 ssd 3.49309 osd.66 up 1.00000 1.00000
102 ssd 3.49309 osd.102 up 1.00000 1.00000
106 ssd 3.49309 osd.106 up 1.00000 1.00000
110 ssd 3.49309 osd.110 up 1.00000 1.00000
-28 20.95853 host pl-host04n
48 nvme 3.49309 osd.48 up 1.00000 1.00000
5 ssd 3.49309 osd.5 up 1.00000 1.00000
9 ssd 3.49309 osd.9 up 1.00000 1.00000
21 ssd 3.49309 osd.21 up 1.00000 1.00000
36 ssd 3.49309 osd.36 up 1.00000 1.00000
43 ssd 3.49309 osd.43 up 1.00000 1.00000
-31 80.34102 host pl-host05n
24 nvme 3.49309 osd.24 up 1.00000 1.00000
45 nvme 3.49309 osd.45 up 1.00000 1.00000
49 nvme 3.49309 osd.49 up 1.00000 1.00000
55 nvme 3.49309 osd.55 up 1.00000 1.00000
60 nvme 3.49309 osd.60 up 1.00000 1.00000
65 nvme 3.49309 osd.65 up 1.00000 1.00000
70 nvme 3.49309 osd.70 up 1.00000 1.00000
85 nvme 3.49309 osd.85 up 1.00000 1.00000
90 nvme 3.49309 osd.90 up 1.00000 1.00000
1 ssd 3.49309 osd.1 up 1.00000 1.00000
14 ssd 3.49309 osd.14 up 1.00000 1.00000
31 ssd 3.49309 osd.31 up 1.00000 1.00000
35 ssd 3.49309 osd.35 up 1.00000 1.00000
37 ssd 3.49309 osd.37 up 1.00000 1.00000
39 ssd 3.49309 osd.39 up 1.00000 1.00000
41 ssd 3.49309 osd.41 up 1.00000 1.00000
80 ssd 3.49309 osd.80 up 1.00000 1.00000
95 ssd 3.49309 osd.95 up 1.00000 1.00000
100 ssd 3.49309 osd.100 up 1.00000 1.00000
108 ssd 3.49309 osd.108 up 1.00000 1.00000
112 ssd 3.49309 osd.112 up 1.00000 1.00000
116 ssd 3.49309 osd.116 up 1.00000 1.00000
125 ssd 3.49309 osd.125 up 1.00000 1.00000
On Jun 19, 2025, at 3:25 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
hare a 'ceph config dump' (remove any passwords / confidential information from the output)