- ceph-users - lists.ceph.io

Setting temporary CRUSH "constraint" for planned cross-datacenter downtime
by Niklas Hambüchen 06 Nov '24

06 Nov '24

My server provider usually does infrastructure maintenance and planned downtimes on a per-datacenter-building granularity, and thus I have a Ceph cluster with that set as the "datacenter" failure domain in CRUSH. However, it now has a planned maintenance that affects two buildings simultaneously. How can I tell Ceph to balance data such that DC1 and DC2 specifically should be treated as the same failure domain? (All other datacenters remain independent; ideally with minimal necessary rebalancing.) Thanks!

3 3

OSD refuse to start
by Albert Shih 06 Nov '24

06 Nov '24

Hi everyone, I'm running currently ceph reef 18.2.4 with podman with Debian 12 Today I've to reboot the cluster (firmware alerts), everything goes as the plan but....one osd on one server won't start. I try everything in https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-osd/ but no luck. I don't find any message in the dmesg. Zero message with journalctl zero message with systemctl status At the end I reboot once more time the server and everything work fine again. Is anyone encounter something like that ? Is that “normal” Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mar. 05 nov. 2024 15:11:40 CET

2 2

Ceph Steering Committee (CLT) Meeting Minutes 2024-11-04
by Neha Ojha 05 Nov '24

05 Nov '24

Hi everybody, These are the highlights from this week's meeting: - Election: - CEC Election is live: https://vote.heliosvoting.org/helios/elections/b2fa20e6-1a5d-4f49-a747-0f8f… - https://pad.ceph.com/p/css-vote-2024q4 - Feedback from the Ceph Ambassadors meeting - October 2024: - https://pad.ceph.com/p/ceph-ambassadors-meeting - 17.2.8 - ongoing discussion with CERN to help make el8 ceph packages available for 17.2.8 - QE validation runs in dev leads review phase - Ceph Tentacle Award - criteria/logistics needs to be discussed - Ceph Windows client builds - cloudbase is reviewing if they will continue this next year - Ceph Foundation had funded it in 2021/22, no longer funding it - Shaman also builds, but we don't distribute. (Signing costs $$) - Usage metrics unknown, the Ceph Foundation has it on agenda for their 2025 budget discussion Thanks, Neha

1 0

Assistance Required: Ceph OSD Out of Memory (OOM) Issue
by Md Mosharaf Hossain 03 Nov '24

03 Nov '24

Dear Ceph Community, I hope this message finds you well. I am encountering an out-of-memory (OOM) issue with one of my Ceph OSDs, which is repeatedly getting killed by the OOM killer on my system. Below are the relevant details from the log: *OOM Log*: [Wed Oct 30 13:14:48 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/system-ceph\x2dosd.slice,task=ceph-osd,pid=6213,uid=64045 [Wed Oct 30 13:14:48 2024] Out of memory: Killed process 6213 (ceph-osd) total-vm:216486528kB, anon-rss:211821164kB, file-rss:0kB, shmem-rss:0kB, UID:64045 pgtables:418836kB oom_score_adj:0 [Wed Oct 30 13:14:58 2024] oom_reaper: reaped process 6213 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB *Ceph OSD Log*: 2024-10-30T13:15:30.207+0600 7f906c74dd80 0 _get_class not permitted to load lua 2024-10-30T13:15:30.211+0600 7f906c74dd80 0 <cls> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello 2024-10-30T13:15:30.215+0600 7f906c74dd80 0 _get_class not permitted to load kvs 2024-10-30T13:15:30.219+0600 7f906c74dd80 0 _get_class not permitted to load queue 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 <cls> /build/ceph-15.2.17/src/cls/cephfs/cls_cephfs.cc:198: loading cephfs 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 osd.13 299547 crush map has features 432629239337189376, adjusting msgr requires for clients 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 osd.13 299547 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 osd.13 299547 crush map has features 3314933000854323200, adjusting msgr requires for osds 2024-10-30T13:15:30.223+0600 7f906c74dd80 1 osd.13 299547 check_osdmap_features require_osd_release unknown -> octopus 2024-10-30T13:15:31.023+0600 7f906c74dd80 0 osd.13 299547 load_pgs *Environment Details*: - Ceph Version: 15.2.17 (Octopus) - OSD: osd.13 - Kernel: Linux kernel version It seems that the OSD process is consuming a substantial amount of memory (total-vm: 216486528kB, anon-rss: 211821164kB), leading to OOM kills on the node. The OSD service restarts but continues to showing consumption excessive memory and OSD get down. Could you please provide guidance or suggestions on how to mitigate this issue? Are there any known memory management settings, configuration adjustments, or OSD-specific tuning parameters that could help prevent this from recurring? Any help would be greatly appreciated. Thank you for your time and assistance! Regards Mosharaf Hossain Manager, Product Development Bangladesh Online (BOL) Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 95757 Cell: +880 1787 680828, Web: www.bol-online.com

2 2

MDS and stretched clusters
by Sake Ceph 01 Nov '24

01 Nov '24

Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other? We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode.

5 8

KRBD: downside of setting alloc_size=4M for discard alignment?
by Friedrich Weber 31 Oct '24

31 Oct '24

Hi, Some of our Proxmox VE users have noticed that a large fstrim inside a QEMU/KVM guest does not free up as much space as expected on the backing RBD image -- if the image is mapped on the host via KRBD and passed to QEMU as a block device (checked via `rbd du --exact`). If the image is attached via QEMU's librbd integration, fstrim seems to work much better. I've found an earlier discussion [0] according to which, for fstrim to work properly, the filesystem should be aligned a object size (4M) boundaries. Indeed, in the test setups I've looked at, the filesystem is not aligned to 4M boundaries. Still, I'm wondering if there might be a solution that doesn't require a specific partitioning/filesystem layout. To have a simpler test setup, I'm not looking at VMs and instead into unaligned blkdiscard on a KRBD-backed block device (on the host). On my test cluster (for versions see [5]), I create an 1G test volume, map it with default settings, write random data to it, and then issue blkdiscard with an 1M offset (see [1] for complete commands): > # blkdiscard --offset 1M /dev/rbd/vmpool/test An `rbd du --exact` reports a size of 256M: > # rbd du --exact -p vmpool test > NAME PROVISIONED USED > test 1 GiB 256 MiB Naively I would expect a result between 1 and 4M, my reasoning being that the 1023M discard could be split into 3M (to get to 4M alignment) plus 1020M. But I've checked the kernel's discard splitting logic [2], and as far as I understand it, it aims to align the discard requests to `discard_granularity`, which is 64k here: > /sys/class/block/rbd0/queue/discard_granularity:65536 I've found I can set the `alloc_size` option [3] to 4M which sets `discard_granularity` to 4M. The result of the blkdiscard is much closer to my expectations (see [4] for complete commands). > # blkdiscard --offset 1M /dev/rbd/vmpool/test > # rbd du --exact -p vmpool test > NAME PROVISIONED USED > test 1 GiB 1 MiB However, apparently with `alloc_size` set to 4M, `minimum_io_size` is also set to 4M (it was 64k before, see [1]): > /sys/class/block/rbd0/queue/minimum_io_size:4194304 My expectation is that this could negatively impact non-discard IO performance (write amplification?). But I am unsure, as I ran a few small benchmarks and couldn't really see any difference between the two settings. Thus, my questions: - Should I expect any downside for non-discard IO after setting `alloc_size` to 4M? - If yes: would it be feasible for KRBD to decouple `discard_granularity` and `minimum_io_size`, i.e., expose an option to set only `discard_granularity` to 4M? Happy about any pointers, and let me know if I can provide any further information. Thanks and best wishes, Friedrich [0] https://www.spinics.net/lists/ceph-users/msg67740.html [1] > # rbd create -p vmpool test --size 1G > # rbd map -p vmpool test > /dev/rbd0 > # grep '' /sys/class/block/rbd0/queue/{discard_*,minimum_io_size,optimal_*} > /sys/class/block/rbd0/queue/discard_granularity:65536 > /sys/class/block/rbd0/queue/discard_max_bytes:4194304 > /sys/class/block/rbd0/queue/discard_max_hw_bytes:4194304 > /sys/class/block/rbd0/queue/discard_zeroes_data:0 > /sys/class/block/rbd0/queue/minimum_io_size:65536 > /sys/class/block/rbd0/queue/optimal_io_size:4194304 > # dd if=/dev/urandom of=/dev/rbd/vmpool/test bs=4M > dd: error writing '/dev/rbd/vmpool/test': No space left on device > 257+0 records in > 256+0 records out > 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.73227 s, 227 MB/s > # rbd du --exact -p vmpool test > NAME PROVISIONED USED > test 1 GiB 1 GiB [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/blo… [3] https://docs.ceph.com/en/reef/man/8/rbd/ [4] > # rbd map -p vmpool test -o alloc_size=4194304 > /dev/rbd0 > # grep '' /sys/class/block/rbd*/device/config_info > 10.1.1.201:6789,10.1.1.202:6789,10.1.1.203:6789 name=admin,key=client.admin,alloc_size=4194304 vmpool test - > # grep '' /sys/class/block/rbd0/queue/{discard_*,minimum_io_size,optimal_*} > /sys/class/block/rbd0/queue/discard_granularity:4194304 > /sys/class/block/rbd0/queue/discard_max_bytes:4194304 > /sys/class/block/rbd0/queue/discard_max_hw_bytes:4194304 > /sys/class/block/rbd0/queue/discard_zeroes_data:0 > /sys/class/block/rbd0/queue/minimum_io_size:4194304 > /sys/class/block/rbd0/queue/optimal_io_size:4194304 > # dd if=/dev/urandom of=/dev/rbd/vmpool/test bs=4M > dd: error writing '/dev/rbd/vmpool/test': No space left on device > 257+0 records in > 256+0 records out > 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.39016 s, 245 MB/s > # rbd du --exact -p vmpool test > NAME PROVISIONED USED > test 1 GiB 1 GiB [5] Host: Proxmox VE 8.2 but with Ubuntu mainline kernel 6.11 build (6.11.0-061100-generic from https://kernel.ubuntu.com/mainline/v6.11/) Ceph: Proxmox build of 18.2.4, but happy try a different build if needed.

3 3

"ceph orch" not working anymore
by Malte Stroem 31 Oct '24

31 Oct '24

Hi there, I have a three node cluster with latest Ceph 18 and cephadm as orchestrator. One node is dead. So the cluster is not healthy but working. Now: Working: cephadm ls or other commands with cephadm... However: ceph orch... does not work at all: Error ENOENT: No orchestrator configured (try `ceph orch set backend`) But it is set! And disabling and enabling it does not work! The cluster is running and working. All ceph commands are working fine. ceph mgr fail or restarting the MGRs did not help. Checked the configuration: ceph config get... everything is fine. Any ideas? Best, Malte

7 24

Deploy custom mgr module
by Darrell Enns 31 Oct '24

31 Oct '24

Is there a simple way to deploy a custom (in-house) mgr module to an orchestrator managed cluster? I assume the module code would need to be included in the mgr container image. However, there doesn't seem to be a straightforward way to do this without having the module merged to upstream ceph (not possible for a custom/in-house solution) or maintaining an in-house container repository and custom container images (a lot of extra maintenance overhead). Also, what's the best way to handle testing during development? Custom scripts to push the code into to a mgr container in a dev cluster?

4 4

Destroyed OSD clinging to wrong disk
by Dave Hall 30 Oct '24

30 Oct '24

Hello. The following is on a Reef Podman installation: In attempting to deal over the weekend with a failed OSD disk, I have somehow managed to have two OSDs pointing to the same HDD, as shown below. [image: image.png] To be sure, the failure occurred on OSD.12, which was pointing to /dev/sdi. I disabled the systemd unit for OSD.12 because it kept restarting. I then destroyed it. When I physically removed the failed disk and rebooted the system, the disk enumeration changed. So, before the reboot, OSD.12 was using /dev/sdi. After the reboot, OSD.9 moved to /dev/sdi. I didn't know that I had an issue until 'ceph-volume lvm prepare' failed. It was in the process of investigating this that I found the above. Right now I have reinserted the failed disk and rebooted, hoping that OSD.12 would find its old disk by some other means, but no joy. My concern is that if I run 'ceph osd rm' I could take out OSD.9. I could take the precaution of marking OSD.9 out and let it drain, but I'd rather not. I am, perhaps, more inclined to manually clear the lingering configuration associated with OSD.12 if someone could send me the list of commands. Otherwise, I'm open to suggestions. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 4

no recovery running
by Joffrey 30 Oct '24

30 Oct '24

HI, This is my cluster: cluster: id: c300532c-51fa-11ec-9a41-0050569c3b55 health: HEALTH_WARN Degraded data redundancy: 2062374/1331064781 objects degraded (0.155%), 278 pgs degraded, 40 pgs undersized 2497 pgs not deep-scrubbed in time 2497 pgs not scrubbed in time services: mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 (age 9d) mgr: hbgt-ceph1-mon3.gmfzqm(active, since 10d), standbys: hbgt-ceph1-mon2.nteihj, hbgt-ceph1-mon1.thrnnu osd: 96 osds: 96 up (since 9d), 96 in (since 45h); 1588 remapped pgs rgw: 3 daemons active (3 hosts, 2 zones) data: pools: 16 pools, 2497 pgs objects: 266.22M objects, 518 TiB usage: 976 TiB used, 808 TiB / 1.7 PiB avail pgs: 2062374/1331064781 objects degraded (0.155%) 349917519/1331064781 objects misplaced (26.289%) 1312 active+remapped+backfill_wait 864 active+clean 199 active+recovery_wait+degraded+remapped 38 active+recovery_wait+degraded 33 active+undersized+degraded+remapped+backfill_wait 33 active+recovery_wait+remapped 7 active+recovery_wait 6 active+undersized+degraded+remapped+backfilling 2 active+recovering+remapped 1 active+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+recovery_wait+undersized+degraded+remapped io: client: 683 KiB/s rd, 2.2 KiB/s wr, 51 op/s rd, 2 op/s wr No recovery is running and I don't understand why. I have free space: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1784.12231 - 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1 TiB 808 TiB 54.72 1.00 - root default -5 208.09680 - 208 TiB 142 TiB 130 TiB 51 GiB 605 GiB 66 TiB 68.14 1.25 - host hbgt-ceph1-osd01 1 hdd 17.34140 1.00000 17 TiB 11 TiB 11 TiB 33 KiB 49 GiB 5.9 TiB 66.16 1.21 136 up osd.1 3 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 23 GiB 49 GiB 6.3 TiB 63.80 1.17 139 up osd.3 5 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 139 MiB 53 GiB 4.8 TiB 72.31 1.32 142 up osd.5 7 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 51 GiB 5.6 TiB 67.97 1.24 145 up osd.7 9 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 2.2 GiB 49 GiB 6.0 TiB 65.67 1.20 140 up osd.9 11 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 329 MiB 50 GiB 5.5 TiB 68.42 1.25 145 up osd.11 13 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52 GiB 5.1 TiB 70.45 1.29 153 up osd.13 15 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 61 KiB 48 GiB 5.7 TiB 66.85 1.22 144 up osd.15 17 hdd 17.34140 1.00000 17 TiB 11 TiB 9.5 TiB 272 MiB 45 GiB 6.8 TiB 60.63 1.11 120 up osd.17 19 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 12 GiB 50 GiB 5.9 TiB 65.90 1.20 134 up osd.19 21 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.6 GiB 57 GiB 4.1 TiB 76.49 1.40 152 up osd.21 23 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 31 KiB 54 GiB 4.7 TiB 73.10 1.34 124 up osd.23 -3 208.09680 - 208 TiB 146 TiB 134 TiB 64 GiB 629 GiB 62 TiB 70.05 1.28 - host hbgt-ceph1-osd02 0 hdd 17.34140 1.00000 17 TiB 11 TiB 9.8 TiB 22 GiB 49 GiB 6.6 TiB 62.07 1.13 124 up osd.0 2 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.7 GiB 52 GiB 5.2 TiB 70.14 1.28 150 up osd.2 4 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 48 GiB 5.8 TiB 66.83 1.22 152 up osd.4 6 hdd 17.34140 0.85004 17 TiB 13 TiB 12 TiB 11 GiB 58 GiB 4.0 TiB 76.85 1.40 153 up osd.6 8 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 54 GiB 4.9 TiB 71.58 1.31 152 up osd.8 10 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 6.3 MiB 47 GiB 6.1 TiB 64.91 1.19 133 up osd.10 12 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 109 MiB 51 GiB 5.6 TiB 67.72 1.24 137 up osd.12 14 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 GiB 53 GiB 5.1 TiB 70.37 1.29 148 up osd.14 16 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 2.9 GiB 54 GiB 4.9 TiB 71.65 1.31 145 up osd.16 18 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 88 MiB 55 GiB 4.8 TiB 72.45 1.32 154 up osd.18 20 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 108 MiB 55 GiB 4.6 TiB 73.39 1.34 166 up osd.20 22 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 209 MiB 54 GiB 4.7 TiB 72.68 1.33 138 up osd.22 -7 208.09680 - 208 TiB 148 TiB 136 TiB 29 GiB 635 GiB 60 TiB 71.03 1.30 - host hbgt-ceph1-osd03 24 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.4 GiB 56 GiB 4.5 TiB 74.27 1.36 150 up osd.24 25 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 32 KiB 57 GiB 4.1 TiB 76.27 1.39 162 up osd.25 26 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 1.4 GiB 49 GiB 6.0 TiB 65.30 1.19 134 up osd.26 27 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.7 GiB 53 GiB 4.7 TiB 73.16 1.34 152 up osd.27 28 hdd 17.34140 1.00000 17 TiB 12 TiB 12 TiB 2.3 MiB 52 GiB 4.9 TiB 71.97 1.32 158 up osd.28 29 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 10 GiB 52 GiB 5.3 TiB 69.72 1.27 141 up osd.29 30 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 266 MiB 54 GiB 4.6 TiB 73.38 1.34 142 up osd.30 31 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 93 MiB 50 GiB 5.7 TiB 67.36 1.23 145 up osd.31 32 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 13 GiB 50 GiB 6.1 TiB 64.77 1.18 131 up osd.32 33 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 MiB 53 GiB 5.2 TiB 69.77 1.28 132 up osd.33 34 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 242 MiB 50 GiB 5.5 TiB 68.17 1.25 139 up osd.34 35 hdd 17.34140 1.00000 17 TiB 14 TiB 13 TiB 135 MiB 58 GiB 3.8 TiB 78.26 1.43 157 up osd.35 -9 208.09680 - 208 TiB 146 TiB 134 TiB 38 GiB 627 GiB 62 TiB 70.19 1.28 - host hbgt-ceph1-osd04 36 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 210 MiB 47 GiB 6.4 TiB 63.16 1.15 139 up osd.36 37 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 222 MiB 53 GiB 5.0 TiB 71.34 1.30 147 up osd.37 38 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 13 MiB 55 GiB 4.5 TiB 74.33 1.36 155 up osd.38 39 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 135 MiB 52 GiB 5.2 TiB 70.22 1.28 140 up osd.39 40 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 25 MiB 53 GiB 4.9 TiB 71.46 1.31 150 up osd.40 41 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 54 GiB 5.0 TiB 71.36 1.30 150 up osd.41 42 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 823 MiB 50 GiB 5.8 TiB 66.49 1.22 156 up osd.42 43 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52 GiB 5.2 TiB 69.97 1.28 160 up osd.43 44 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52 GiB 5.3 TiB 69.35 1.27 133 up osd.44 45 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 12 GiB 55 GiB 4.8 TiB 72.12 1.32 148 up osd.45 46 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52 GiB 5.1 TiB 70.74 1.29 148 up osd.46 47 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 188 MiB 54 GiB 4.9 TiB 71.74 1.31 151 up osd.47 -11 208.09680 - 208 TiB 147 TiB 135 TiB 27 GiB 627 GiB 62 TiB 70.45 1.29 - host hbgt-ceph1-osd05 48 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 49 GiB 5.7 TiB 67.11 1.23 152 up osd.48 49 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 52 GiB 5.1 TiB 70.87 1.30 155 up osd.49 50 hdd 17.34140 0.90002 17 TiB 12 TiB 11 TiB 11 GiB 54 GiB 5.2 TiB 70.01 1.28 147 up osd.50 51 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 84 MiB 54 GiB 4.6 TiB 73.24 1.34 147 up osd.51 52 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 337 MiB 53 GiB 4.8 TiB 72.26 1.32 146 up osd.52 53 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.6 GiB 54 GiB 5.1 TiB 70.56 1.29 132 up osd.53 54 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.3 GiB 56 GiB 4.5 TiB 73.77 1.35 138 up osd.54 55 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 10 GiB 53 GiB 4.8 TiB 72.59 1.33 148 up osd.55 56 hdd 17.34140 1.00000 17 TiB 11 TiB 9.6 TiB 1.8 GiB 46 GiB 6.7 TiB 61.16 1.12 127 up osd.56 57 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 3.1 MiB 50 GiB 5.6 TiB 67.96 1.24 148 up osd.57 58 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 687 KiB 52 GiB 5.0 TiB 70.93 1.30 152 up osd.58 59 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 22 KiB 55 GiB 4.4 TiB 74.88 1.37 147 up osd.59 -13 245.93866 - 246 TiB 112 TiB 106 TiB 28 GiB 498 GiB 134 TiB 45.56 0.83 - host hbgt-ceph1-osd06 60 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 10 GiB 51 GiB 9.3 TiB 54.43 0.99 134 up osd.60 61 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 216 MiB 43 GiB 11 TiB 47.06 0.86 120 up osd.61 62 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 330 MiB 41 GiB 11 TiB 44.53 0.81 116 up osd.62 63 hdd 20.49489 1.00000 20 TiB 8.9 TiB 8.4 TiB 354 MiB 39 GiB 12 TiB 43.29 0.79 113 up osd.63 64 hdd 20.49489 1.00000 20 TiB 8.1 TiB 7.7 TiB 148 MiB 36 GiB 12 TiB 39.72 0.73 117 up osd.64 65 hdd 20.49489 1.00000 20 TiB 12 TiB 12 TiB 14 KiB 53 GiB 8.4 TiB 58.90 1.08 159 up osd.65 66 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 18 MiB 49 GiB 9.8 TiB 52.33 0.96 129 up osd.66 67 hdd 20.49489 1.00000 20 TiB 498 GiB 1.5 GiB 1 KiB 745 MiB 20 TiB 2.37 0.04 6 up osd.67 68 hdd 20.49489 1.00000 20 TiB 10 TiB 9.8 TiB 13 GiB 47 GiB 10 TiB 50.12 0.92 138 up osd.68 69 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 2.0 GiB 48 GiB 9.7 TiB 52.77 0.96 135 up osd.69 70 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 1.6 GiB 50 GiB 9.2 TiB 55.24 1.01 150 up osd.70 71 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 86 MiB 42 GiB 11 TiB 45.96 0.84 130 up osd.71 -15 245.93866 - 246 TiB 114 TiB 108 TiB 62 GiB 505 GiB 132 TiB 46.18 0.84 - host hbgt-ceph1-osd07 72 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 12 GiB 49 GiB 9.6 TiB 53.39 0.98 146 up osd.72 73 hdd 20.49489 1.00000 20 TiB 8.5 TiB 8.0 TiB 194 MiB 38 GiB 12 TiB 41.31 0.75 112 up osd.73 74 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 10 GiB 42 GiB 11 TiB 46.12 0.84 124 up osd.74 75 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 12 GiB 41 GiB 11 TiB 44.16 0.81 125 up osd.75 76 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 114 MiB 41 GiB 11 TiB 45.91 0.84 111 up osd.76 77 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 1.5 GiB 42 GiB 11 TiB 47.03 0.86 127 up osd.77 78 hdd 20.49489 1.00000 20 TiB 7.7 TiB 7.3 TiB 12 GiB 36 GiB 13 TiB 37.77 0.69 105 up osd.78 79 hdd 20.49489 1.00000 20 TiB 8.2 TiB 7.7 TiB 1 KiB 36 GiB 12 TiB 39.92 0.73 91 up osd.79 80 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 371 MiB 41 GiB 11 TiB 46.13 0.84 132 up osd.80 81 hdd 20.49489 1.00000 20 TiB 9.8 TiB 9.3 TiB 353 MiB 45 GiB 11 TiB 47.95 0.88 121 up osd.81 82 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 12 GiB 51 GiB 9.2 TiB 55.23 1.01 142 up osd.82 83 hdd 20.49489 1.00000 20 TiB 10 TiB 9.6 TiB 1.9 GiB 44 GiB 10 TiB 49.28 0.90 125 up osd.83 -17 251.76105 - 252 TiB 23 TiB 11 TiB 43 MiB 64 GiB 229 TiB 8.99 0.16 - host hbgt-ceph1-osd08 84 hdd 20.98009 1.00000 21 TiB 1.0 TiB 65 GiB 1 KiB 1.4 GiB 20 TiB 4.93 0.09 5 up osd.84 85 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 813 MiB 20 TiB 4.63 0.08 7 up osd.85 86 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.0 GiB 20 TiB 4.63 0.08 2 up osd.86 87 hdd 20.98009 1.00000 21 TiB 1008 GiB 14 GiB 1 KiB 1.0 GiB 20 TiB 4.69 0.09 7 up osd.87 88 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.6 GiB 20 TiB 4.63 0.08 6 up osd.88 89 hdd 20.98009 1.00000 21 TiB 4.9 TiB 4.0 TiB 17 MiB 20 GiB 16 TiB 23.56 0.43 51 up osd.89 90 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.5 GiB 20 TiB 4.63 0.08 5 up osd.90 91 hdd 20.98009 1.00000 21 TiB 3.3 TiB 2.3 TiB 1 KiB 11 GiB 18 TiB 15.70 0.29 40 up osd.91 92 hdd 20.98009 1.00000 21 TiB 5.1 TiB 4.1 TiB 26 MiB 20 GiB 16 TiB 24.34 0.44 60 up osd.92 93 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.3 GiB 20 TiB 4.63 0.08 9 up osd.93 94 hdd 20.98009 1.00000 21 TiB 1.4 TiB 489 GiB 1 KiB 3.4 GiB 20 TiB 6.90 0.13 14 up osd.94 95 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1016 MiB 20 TiB 4.63 0.08 4 up osd.95 TOTAL 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1 TiB 808 TiB 54.72 I tried: - update backfill and recovery settings - change osd mclock profile - change pg/pgp_num The cluster does not recovery. Clients are working well. What can I do ? Thanks for your help

3 2