- ceph-users - lists.ceph.io

Re: First time configuration advice
by Robert Sander 25 Jun '25

25 Jun '25

Hi, Am 6/24/25 um 19:21 schrieb Ryan Sleeth: > Do you think there would be any performance benefits to using namespaces > over LVM partitions? I do not think there is a big difference. > Currently I am setting up the nodes with this: > > sudo ceph orch daemon add osd ceph01:/dev/mapper/ceph--nvme--01-block--01,encrypted=true > sudo ceph orch daemon add osd ceph01:data_devices=/dev/mapper/ceph--hdd--01-block--02,db_devices=/dev/mapper/ceph--nvme--01-db--02,encrypted=true > sudo ceph orch daemon add osd ceph01:data_devices=/dev/mapper/ceph--hdd--02-block--03,db_devices=/dev/mapper/ceph--nvme--01-db--03,encrypted=true This is very manual and will not benefit from the automation possibilities of the orchestrator. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Question: What's special about subvolumes?
by James Tocknell 24 Jun '25

24 Jun '25

Hi All Possibly a dumb question, but given with cephfs you can apply quotas to arbitrary directories, snapshot arbitrary directories, assign arbitrary directories to pools etc. what is special/different about subvolumes, and why would I want to use them (and more importantly, not want to use them)? I guess the only thing I can see could be around where the extended attributes are assigned (with the two-level directory structure enabling setting a quota while not allowing read access to the directory above, based on looking at the cephfs CSI?), but I feel I'm missing something obvious. The specific usecase I'm looking at is evaluating using just quotas vs subvolumes for team shares (to ensure that each team only uses the amount of storage they require), but in general I'm trying to understand subvolumes. Regards James

4 3

Force remove undeletable image on RBD pool?
by Gaël THEROND 24 Jun '25

24 Jun '25

Hi folks, I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`. This pool is a EC based pool and neither the metadata nor the data pool have any data of that image remaining. So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can. Thanks!

7 12

CEPH upgrade from 18.2.7 to 19.2.2 -- Hung from last 24h at 66%
by Devender Singh 24 Jun '25

24 Jun '25

Hello all I have a cluster where my cluster is in hung state, Some back fills are there but I reduced it to 1 but still upgrade not progressing… Please help… ```# ceph health detail HEALTH_WARN 8 OSD(s) experiencing slow operations in BlueStore; Failed to apply 2 service(s): osd.all-available-devices,osd.iops_optimized; 1 failed cephadm daemon(s); failed to probe daemons or devices; noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 1600150/39365198 objects degraded (4.065%), 93 pgs degraded, 103 pgs undersized; 127 pgs not deep-scrubbed in time [WRN] BLUESTORE_SLOW_OP_ALERT: 8 OSD(s) experiencing slow operations in BlueStore osd.5 observed slow operation indications in BlueStore osd.9 observed slow operation indications in BlueStore osd.18 observed slow operation indications in BlueStore osd.36 observed slow operation indications in BlueStore osd.59 observed slow operation indications in BlueStore osd.66 observed slow operation indications in BlueStore osd.106 observed slow operation indications in BlueStore osd.110 observed slow operation indications in BlueStore [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 2 service(s): osd.all-available-devices,osd.iops_optimized osd.all-available-devices: Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) osd.iops_optimized: Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.19 on phl-prod-host04n.example.comis in error state [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices Command "cephadm ceph-volume -- inventory" timed out on host phl-prod-converged03n.phl.netskrt.org (default 900 second timeout) [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set [WRN] PG_DEGRADED: Degraded data redundancy: 1600150/39365198 objects degraded (4.065%), 93 pgs degraded, 103 pgs undersized pg 25.0 is stuck undersized for 21h, current state active+undersized+degraded+remapped+backfill_wait, last acting [90,77,NONE,38,3] pg 25.1 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [131,1,66,NONE,72] pg 25.c is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [1,135,110,28,NONE] pg 25.e is active+undersized+degraded+remapped+backfill_wait, acting [18,20,108,101,NONE] pg 25.14 is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,13,NONE,65,97] pg 25.17 is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,65,110,23,NONE] pg 25.1a is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [97,66,NONE,112,139] pg 25.1c is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [102,136,97,NONE,45] pg 25.1f is stuck undersized for 8h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,66,7,39,109] pg 26.45 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,47] pg 26.48 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [113,55] pg 26.4b is stuck undersized for 8h, current state active+undersized+degraded+remapped+backfill_wait, last acting [105,7] pg 26.59 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [35,40] pg 26.6a is stuck undersized for 7h, current state active+undersized+degraded+remapped+backfill_wait, last acting [25,45] pg 26.74 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [112,131] pg 26.77 is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [17,105] pg 26.9c is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [47,76] pg 26.bf is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,26] pg 26.c2 is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [97,135] pg 26.ec is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [26,80] pg 26.f7 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [49,105] pg 31.12 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [47,110,131,37,NONE] pg 31.19 is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,87,3,137,39] pg 31.1b is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [108,NONE,42,102,97] pg 31.1c is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [113,66,NONE,45,72] pg 31.41 is stuck undersized for 2h, current state active+undersized+remapped+backfill_wait, last acting [14,101,NONE,67,18] pg 31.42 is stuck undersized for 7h, current state active+undersized+degraded+remapped+backfill_wait, last acting [37,134,NONE,82,8] pg 31.44 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [49,NONE,101,3,62] pg 31.46 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfilling, last acting [95,38,NONE,25,102] pg 31.47 is stuck undersized for 18h, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,NONE,130,72,110] pg 31.4a is stuck undersized for 4h, current state active+undersized+degraded+remapped+backfilling, last acting [66,NONE,135,82,14] pg 31.4c is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [34,NONE,1,82,18] pg 31.4d is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [101,13,65,30,NONE] pg 31.52 is stuck undersized for 4m, current state active+undersized+remapped+backfill_wait, last acting [26,112,66,NONE,135] pg 31.53 is stuck undersized for 17h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,24,11,42,110] pg 31.55 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [116,NONE,27,4,117] pg 31.57 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [17,15,110,NONE,1] pg 31.5a is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [13,24,124,26,NONE] pg 31.5b is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,31,8,27,42] pg 31.5c is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,60,28,18,119] pg 31.5d is stuck undersized for 17h, current state active+undersized+degraded+remapped+backfill_wait, last acting [124,NONE,85,6,11] pg 31.5e is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,4,1,23,138] pg 31.60 is stuck undersized for 18h, current state active+undersized+degraded+remapped+backfilling, last acting [37,11,102,NONE,133] pg 31.64 is stuck undersized for 4m, current state active+undersized+remapped+backfill_wait, last acting [26,106,45,34,NONE] pg 31.65 is stuck undersized for 4h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,14,127,62,3] pg 31.66 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [34,NONE,35,23,59] pg 31.67 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [67,24,127,8,NONE] pg 31.68 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,41,18,17,NONE] pg 31.70 is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfilling, last acting [106,38,1,NONE,97] pg 31.75 is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [26,31,34,8,NONE] pg 31.7f is stuck undersized for 2h, current state active+undersized+remapped+backfilling, last acting [NONE,101,110,10,45] [WRN] PG_NOT_DEEP_SCRUBBED: 127 pgs not deep-scrubbed in time pg 26.e5 not deep-scrubbed since 2025-06-07T05:59:22.612553+0000 pg 26.ca not deep-scrubbed since 2025-06-06T18:21:07.435823+0000 pg 26.bf not deep-scrubbed since 2025-06-03T07:29:02.791095+0000 pg 26.be not deep-scrubbed since 2025-06-04T07:55:54.389714+0000 pg 26.a5 not deep-scrubbed since 2025-06-07T07:28:56.878096+0000 pg 26.85 not deep-scrubbed since 2025-06-03T15:18:59.395929+0000 pg 26.7f not deep-scrubbed since 2025-06-04T21:29:28.412637+0000 pg 26.7e not deep-scrubbed since 2025-06-03T09:14:19.585388+0000 pg 26.7d not deep-scrubbed since 2025-06-03T18:37:30.931020+0000 pg 26.7c not deep-scrubbed since 2025-06-04T13:38:00.061488+0000 pg 26.73 not deep-scrubbed since 2025-06-03T06:20:15.111819+0000 pg 26.6f not deep-scrubbed since 2025-06-03T13:45:24.880397+0000 pg 26.6e not deep-scrubbed since 2025-05-26T23:15:32.099862+0000 pg 26.6d not deep-scrubbed since 2025-06-04T14:04:10.449101+0000 pg 31.62 not deep-scrubbed since 2025-06-03T13:34:49.518456+0000 pg 26.65 not deep-scrubbed since 2025-06-04T07:56:25.353411+0000 pg 31.66 not deep-scrubbed since 2025-06-03T10:32:05.364424+0000 pg 26.62 not deep-scrubbed since 2025-06-04T09:35:58.267976+0000 pg 31.65 not deep-scrubbed since 2025-06-03T16:04:40.003140+0000 pg 31.5b not deep-scrubbed since 2025-06-03T14:18:18.835477+0000 pg 26.5d not deep-scrubbed since 2025-06-04T15:14:30.870252+0000 pg 31.58 not deep-scrubbed since 2025-06-03T03:09:27.568605+0000 pg 26.5c not deep-scrubbed since 2025-06-03T01:57:27.644129+0000 pg 31.5f not deep-scrubbed since 2025-06-03T05:53:20.860393+0000 pg 31.52 not deep-scrubbed since 2025-05-27T00:01:27.040861+0000 pg 31.53 not deep-scrubbed since 2025-05-24T09:37:58.964829+0000 pg 26.55 not deep-scrubbed since 2025-06-04T21:25:34.135356+0000 pg 26.54 not deep-scrubbed since 2025-06-04T06:07:12.978734+0000 pg 31.56 not deep-scrubbed since 2025-06-04T12:58:17.599712+0000 pg 31.57 not deep-scrubbed since 2025-06-03T07:02:16.859990+0000 pg 26.51 not deep-scrubbed since 2025-06-03T05:42:22.435483+0000 pg 26.4f not deep-scrubbed since 2025-06-03T09:10:22.617328+0000 pg 31.4a not deep-scrubbed since 2025-05-28T00:54:55.246532+0000 pg 26.4e not deep-scrubbed since 2025-06-03T11:16:49.278513+0000 pg 31.4b not deep-scrubbed since 2025-06-03T10:24:24.123351+0000 pg 26.4d not deep-scrubbed since 2025-06-04T19:01:44.614410+0000 pg 31.49 not deep-scrubbed since 2025-05-28T04:56:29.368285+0000 pg 31.42 not deep-scrubbed since 2025-05-28T08:38:57.151865+0000 pg 26.41 not deep-scrubbed since 2025-06-03T05:47:35.443867+0000 pg 26.40 not deep-scrubbed since 2025-06-03T05:43:13.283668+0000 pg 25.17 not deep-scrubbed since 2025-06-03T09:26:52.253625+0000 pg 32.2e not deep-scrubbed since 2025-06-05T13:01:06.175389+0000 pg 22.1a not deep-scrubbed since 2025-06-07T01:40:45.063268+0000 pg 31.13 not deep-scrubbed since 2025-06-03T12:21:17.965218+0000 pg 22.1b not deep-scrubbed since 2025-06-04T12:22:44.947751+0000 pg 25.14 not deep-scrubbed since 2025-05-28T06:26:32.552200+0000 pg 26.17 not deep-scrubbed since 2025-06-03T18:37:26.617483+0000 pg 31.12 not deep-scrubbed since 2025-05-28T08:39:23.271194+0000 pg 31.1c not deep-scrubbed since 2025-06-03T08:17:51.230187+0000 pg 25.1f not deep-scrubbed since 2025-05-28T00:19:23.653883+0000 77 more pgs… Regards Dev

6 16

Why is it still so difficult to just dump all config and where it comes from?
by Niklas Hambüchen 22 Jun '25

22 Jun '25

Year after year I find myself confused with ceph's various methods to just show the current configuration. I just want one command that can dump all the config, and where the values come from. I used a mix of the following so far: ceph config dump "dumps the entire Monitor central configuration database" But it shows only values that are different from the default. Outputs a "WHO" column that includes entries for both "mgr" or "osd", and e.g. "osd.123". config show-with-defaults <who> Very useful: Outputs SOURCE/OVERRIDES/IGNORES Oddly, not listed in https://docs.ceph.com/en/squid/rados/configuration/ceph-conf/#commands Unfortunately, cannot dump all config, instead needs to be given <who> e.g. `config show-with-defaults osd.123`. But not all things from the WHO column from "ceph config dump" are accepted, e.g. `config show-with-defaults mgr` does not work. Confusing. Somehow misses some values: `ceph config dump` shows WHO MASK LEVEL OPTION VALUE RO mgr unknown mgr/pg_autoscaler/autoscale_profile scale-up * But `autoscale_profile` is nowhere to be found in any `ceph config show-with-defaults mgr...` outputs. ceph-conf --show-config Legacy, as explained on https://docs.ceph.com/en/squid/man/8/ceph-conf/ But I think https://docs.ceph.com/en/squid/rados/configuration/ceph-conf/#commands should list it so that from there one can easily see it's legacy. This is all very confusing. Can we please have 1 command, that can dump all config (including from config files, monitor central configuration database, all currently running daemons), and nicely point out what's set and overridden where and which value is in effect? Is there any issue I can subscribe to that tracks improving this? Thanks!

2 3

OSD network Issue -- Not using cluster Network
by Devender Singh 22 Jun '25

22 Jun '25

Hello All Found some network issue on OSD’s, why they are not using cluster network? Only using public network? My mons are on public network… How can I resolve it? Changing this network will lead any issue like need to stop all osd’s or restart cluster or rolling restart osd’s & how, please help? # ceph config get mon cluster_network 10.102.5.0/24 root@pl-host01n:/cephfs/bin# ceph config get mon public_network 10.102.1.0/24 root@pl-host01n:/cephfs/bin# ceph osd find osd.8 { "osd": 8, "addrs": { "addrvec": [ { "type": "v2", "addr": "10.102.1.122:6806", "nonce": 4172377551 }, { "type": "v1", "addr": "10.102.1.122:6807", "nonce": 4172377551 } ] }, "osd_fsid": "1b167c95-a873-4c02-aafc-24067246e65d", "host": "pl-host02n.ld6.example.”com, "crush_location": { "host": "pl-host02n", "root": "default" } } Regards Dev

2 1

Adding OSD nodes
by Sinan Polat 21 Jun '25

21 Jun '25

Hello, I am currently managing a Ceph cluster that consists of 3 racks, each with 4 OSD nodes. Each node contains 24 OSDs. I plan to add three new nodes, one to each rack, to help alleviate the high OSD utilization. The current highest OSD utilization is 85%. I am concerned about the possibility of any OSD reaching the osd_full_ratio threshold during the rebalancing process. This would cause the cluster to enter a read-only state, which I want to avoid at all costs. I am planning to execute the following commands: ceph orch host add new-node-1 ceph orch host add new-node-2 ceph orch host add new-node-3 ceph osd crush move new-node-1 rack=rack-1 ceph osd crush move new-node-2 rack=rack-2 ceph osd crush move new-node-3 rack=rack-3 ceph config set osd osd_max_backfills 1 ceph config set osd osd_recovery_max_active 1 ceph config set osd osd_recovery_sleep 0.1 ceph orch apply osd --all-available-devices Before proceeding, I would like to ask if the above steps are safe to execute in a cluster with such high utilization. My main concern is whether the rebalancing could cause any OSD to exceed the osd_full_ratio and result in unexpected failures. Any insights or advice on how to safely add these nodes without impacting cluster stability would be greatly appreciated. Thanks!

7 7

el10 plans
by Christopher Durham 20 Jun '25

20 Jun '25

Hi, When will rpms for Linux EL10 and clones such as Rocky Linux be available, and what ceph release will it be? I am not specifically asking when RedHat or IBMthemselves will release on this version, I am looking for el10 rpms at https://download.ceph.com. Thanks -Chris

2 1

Ceph reef vs squid performance
by Wissem MIMOUNA - Ceph Users 20 Jun '25

20 Jun '25

Hi ceph users, We have two ceph cluster with the same hardware/CPU(ARM)/OS(debian 12)/network , and both deployed with cephadm ( default configuration , 2 osds per device nvme ) , except that one is on reef version and the seco,d is on squid version. We noticed a big difference in our benchmark ( the Reef cluster is far better the Squid cluster ). Does mean that Ceph Reef is more performant than Ceph Squid ? Squid RandRW bench => fio -ioengine=rbd -name=test-2 -direct=1 -bs=4k -iodepth=1 -rw=randrw -rwmixwrite=100 -runtime=10 -pool=nova -rbdname=myimage test-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=2488KiB/s][w=622 IOPS][eta 00m:00s] test-2: (groupid=0, jobs=1): err= 0: pid=331890: Fri Jun 20 11:35:14 2025 write: IOPS=627, BW=2511KiB/s (2571kB/s)(24.5MiB/10002msec); 0 zone resets slat (nsec): min=4400, max=51722, avg=15476.42, stdev=4695.32 clat (usec): min=670, max=7704, avg=1574.43, stdev=341.38 lat (usec): min=676, max=7730, avg=1589.91, stdev=341.83 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1303], 10.00th=[ 1369], 20.00th=[ 1434], | 30.00th=[ 1483], 40.00th=[ 1516], 50.00th=[ 1549], 60.00th=[ 1582], | 70.00th=[ 1614], 80.00th=[ 1647], 90.00th=[ 1696], 95.00th=[ 1778], | 99.00th=[ 2769], 99.50th=[ 3458], 99.90th=[ 5800], 99.95th=[ 5932], | 99.99th=[ 7701] bw ( KiB/s): min= 2000, max= 2608, per=100.00%, avg=2512.42, stdev=131.03, samples=19 iops : min= 500, max= 652, avg=628.11, stdev=32.76, samples=19 lat (usec) : 750=0.14%, 1000=0.61% lat (msec) : 2=96.21%, 4=2.64%, 10=0.40% cpu : usr=1.34%, sys=0.93%, ctx=6280, majf=0, minf=5 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,6278,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Reef RandRW bench=> | | |fio -ioengine=rbd -name=test -direct=1 -bs=4K -iodepth=1 -rw=randrw -rwmixwrite=100 -runtime=10 -pool=nova -rbdname=image1 test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.33 | |Starting 1 process | |Jobs: 1 (f=1): [w(1)][100.0%][w=3931KiB/s][w=982 IOPS][eta 00m:00s] | |test: (groupid=0, jobs=1): err= 0: pid=1866868: Tue Jun 17 15:24:42 2025 | |write: IOPS=964, BW=3860KiB/s (3952kB/s)(37.7MiB/10001msec); 0 zone resets | |slat (nsec): min=3880, max=30402, avg=6499.36, stdev=1597.11 | |clat (usec): min=617, max=3219, avg=1027.83, stdev=161.62 | |lat (usec): min=625, max=3224, avg=1034.33, stdev=161.62 | |clat percentiles (usec): | 1.00th=[ 824], 5.00th=[ 881], 10.00th=[ 906], 20.00th=[ 938], | 30.00th=[ 963], 40.00th=[ 979], 50.00th=[ 1004], 60.00th=[ 1020], | 70.00th=[ 1045], 80.00th=[ 1074], 90.00th=[ 1123], 95.00th=[ 1221], | 99.00th=[ 1860], 99.50th=[ 1942], 99.90th=[ 2147], 99.95th=[ 2343], | 99.99th=[ 3228] | |bw ( KiB/s): min= 3152, max= 3984, per=100.00%, avg=3861.89, stdev=183.90, samples=19 | |iops : min= 788, max= 996, avg=965.47, stdev=45.98, samples=19 | |lat (usec) : 750=0.16%, 1000=49.50% | |lat (msec) : 2=50.07%, 4=0.27% | |cpu : usr=0.91%, sys=0.51%, ctx=9657, majf=0, minf=13 | |IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,9650,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 |

1 0

Cluster Not recovering after multiple disk failure
by Devender Singh 20 Jun '25

20 Jun '25

Hello all Need urgent help on below… I tried reducing min_size but still showing same… cluster: id: 15688cb4-044a-11ec-942e-516035adea04 health: HEALTH_ERR 3 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 20/16670718 objects unfound (0.000%) Reduced data availability: 283 pgs inactive, 464 pgs incomplete Possible data damage: 2 pgs recovery_unfound Degraded data redundancy: 42998/61175329 objects degraded (0.070%), 2 pgs degraded, 1 pg undersized 304 pgs not deep-scrubbed in time 1055 slow ops, oldest one blocked for 94042 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops. services: mon: 5 daemons, quorum van2-converged04n,van2-converged05n,van2-converged01n,van2-converged03n,van2-converged02n (age 17h) mgr: van2-converged05n.lybhho(active, since 22h), standbys: van2-converged04n.azmqik mds: 3/3 daemons up, 11 standby osd: 31 osds: 31 up (since 5h), 31 in (since 5h); 2 remapped pgs rbd-mirror: 2 daemons active (2 hosts) data: volumes: 1/2 healthy, 1 recovering pools: 32 pools, 1505 pgs objects: 16.67M objects, 56 TiB usage: 152 TiB used, 234 TiB / 387 TiB avail pgs: 0.332% pgs unknown 30.897% pgs not active 42998/61175329 objects degraded (0.070%) 21469/61175329 objects misplaced (0.035%) 20/16670718 objects unfound (0.000%) 1034 active+clean 463 incomplete 5 unknown 1 remapped+incomplete 1 active+recovery_unfound+degraded 1 recovery_unfound+undersized+degraded+remapped+peered progress: Global Recovery Event (22h) [===================.........] (remaining: 10h) [WRN] SLOW_OPS: 1055 slow ops, oldest one blocked for 93877 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops. [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.van2.van2-converged05n.mbbzfj(mds.1): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 62279 secs [WRN] OBJECT_UNFOUND: 20/16670718 objects unfound (0.000%) pg 28.1f has 9 unfound objects pg 29.2 has 11 unfound objects [WRN] PG_AVAILABILITY: Reduced data availability: 470 pgs inactive, 464 pgs incomplete pg 16.58 is incomplete, acting [7,14,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5c is incomplete, acting [26,3,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5f is incomplete, acting [13,6,21] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.63 is incomplete, acting [9,7,15] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.64 is incomplete, acting [18,19,5] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.68 is incomplete, acting [2,5,27] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.70 is incomplete, acting [19,1,24] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.74 is incomplete, acting [8,25,14] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.78 is incomplete, acting [23,14,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7c is incomplete, acting [24,10,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7d is incomplete, acting [12,22,13] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.64 is incomplete, acting [20,11,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.66 is incomplete, acting [21,13,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6a is incomplete, acting [4,26,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6c is incomplete, acting [13,12,31] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6e is incomplete, acting [8,31,24] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.7a is incomplete, acting [8,28,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 19.54 is stuck inactive since forever, current state incomplete, last acting [10,4,20,15,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.58 is incomplete, acting [15,18,12,28,20] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.59 is incomplete, acting [6,9,18,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5a is incomplete, acting [11,31,20,17,24] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for ‘incomplceph -s Ceph Any way to make these scrubs complete faster? pg 19.63 not deep-scrubbed since 2025-05-24T08:18:29.738427+0000 pg 34.6d not deep-scrubbed since 2025-06-04T08:33:54.534882+0000 pg 16.5f not deep-scrubbed since 2025-05-21T10:44:09.996254+0000 pg 19.5d not deep-scrubbed since 2025-05-26T09:36:27.064154+0000 pg 19.5e not deep-scrubbed since 2025-06-05T00:52:03.859984+0000 pg 16.5c not deep-scrubbed since 2025-06-06T00:36:22.021390+0000 pg 19.5f not deep-scrubbed since 2025-06-03T16:27:42.356213+0000 pg 22.5a not deep-scrubbed since 2025-06-05T23:00:28.066065+0000 pg 34.69 not deep-scrubbed since 2025-06-03T05:07:58.209808+0000 pg 19.58 not deep-scrubbed since 2025-05-27T23:32:29.963976+0000 pg 19.59 not deep-scrubbed since 2025-05-25T11:50:44.735318+0000 pg 19.5a not deep-scrubbed since 2025-06-06T02:34:05.486126+0000 pg 16.58 not deep-scrubbed since 2025-05-13T14:10:44.570493+0000 Regards Dev

3 4