- ceph-users - lists.ceph.io

SQUID ganesha failures
by Patrick Begou 02 Oct '25

02 Oct '25

Hi all, I have a Ceph Cluster using squid 19.2.2. I'm using CephFS storage with kernel mounts on clients and it works fine but for some older computers (still CentOS8 and one CentOS7 because it is running a commercial application) I need to use NFS so I've deployed Ganesha. But a few minutes after the first NFS client mount the volume (Just testing from the first client) and uses it, the Ceph NFS service falls in error on one node (not always the same node) and I have to reboot it. # ceph orch ps --daemon-type nfs NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID nfs.whitaker-nfs.0.2.whitaker04-ceph.itzyqf whitaker04-ceph *:12049 running (49m) 5m ago 3h 26.3M - 5.9 4892a7ef541b 42fd4ab7bba9 nfs.whitaker-nfs.1.0.whitaker05-ceph.vocxmr whitaker05-ceph *:12049 running (63m) 5m ago 23h 85.7M - 5.9 4892a7ef541b ab1c06be2ce7 nfs.whitaker-nfs.2.0.whitaker01-ceph.rhwuth whitaker01-ceph *:12049 running (33m) 5m ago 23h 24.6M - 5.9 4892a7ef541b eb93963f05cd nfs.whitaker-nfs.3.0.whitaker03-ceph.wqfnnq whitaker03-ceph *:12049 error 5m ago 23h - - <unknown> <unknown> <unknown> nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn whitaker02-ceph *:12049 running (49m) 4m ago 3h 28.3M - 5.9 4892a7ef541b 5cb71f9d8b89 and the client freeze on the mount point. Any idea about this or how to track the problem ? Thanks Patrick ---------------------------------------------- Details of my NFS CEPH cluster, building commands and status: ---------------------------------------------- # *ceph nfs cluster create whitaker-nfs \ whitaker01-ceph,whitaker02-ceph,whitaker03-ceph,whitaker04-ceph,whitaker05-ceph \ --ingress --virtual_ip 172.30.1.102/24 --ingress-mode haproxy-protocol* *# ceph nfs cluster info whitaker-nfs* { "whitaker-nfs": { "backend": [ { "hostname": "whitaker01-ceph", "ip": "172.30.1.97", "port": 12049 }, { "hostname": "whitaker02-ceph", "ip": "172.30.1.98", "port": 12049 }, { "hostname": "whitaker03-ceph", "ip": "172.30.1.99", "port": 12049 }, { "hostname": "whitaker04-ceph", "ip": "172.30.1.100", "port": 12049 }, { "hostname": "whitaker05-ceph", "ip": "172.30.1.101", "port": 12049 } ], "monitor_port": 9049, "port": 2049, "virtual_ip": "172.30.1.102" } } *# ceph nfs export create cephfs \ --cluster-id whitaker-nfs \ --pseudo-path /data \ --fsname data \ --client_addr 172.30.1.0/24* *# ceph nfs export ls whitaker-nfs --detailed* [ { "access_type": "none", "clients": [ { "access_type": "rw", "addresses": [ "172.30.1.0/24" ], "squash": "none" } ], "cluster_id": "whitaker-nfs", "export_id": 1, "fsal": { "cmount_path": "/", "fs_name": "data", "name": "CEPH", "user_id": "nfs.whitaker-nfs.data.79da3503" }, "path": "/", "protocols": [ 4 ], "pseudo": "/data", "security_label": true, "squash": "none", "transports": [ "TCP" ] } ]

3 6

Upgrading from Reef to Squid, "init missing pg_pool_t for deleted pool for pg please downgrade to luminous and allow pg deletion to complete before upgrading
by Daniel Williams 30 Sep '25

30 Sep '25

Some background pool 57 is a new rbd pool (12MiB used) that I was just experimenting with (performance of striped hdd rbd devices), I don't think I deleted it but can't say for sure (it appears in ceph df) since it doesn't matter. This pool was created on reef, a full deep scrub has been done several times over since moving to reef (March 2024), likely no deep scrub has been done since moving to squid since I've had lots of troubles... This error however has broken a 150TiB machine and worse I don't know that a restart won't break others.. After a host reboot I've lost half the OSDs on that host, they all refuse to start with: -725> 2025-09-25T18:02:37.157+0000 7f93d0aab8c0 -1 Falling back to public interface -2> 2025-09-25T18:02:40.033+0000 7f93d0aab8c0 -1 osd.21 2098994 init missing pg_pool_t for deleted pool 57 for pg 57.3s7; please downgrade to luminous and allow pg deletion to complete before upgrading -1> 2025-09-25T18:02:40.037+0000 7f93d0aab8c0 -1 ./src/osd/OSD.cc: In function 'int OSD::init()' thread 7f93d0aab8c0 time 2025-09-25T18:02:40.040491+0000 ./src/osd/OSD.cc: 3867: ceph_abort_msg("abort() called") ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb7) [0x560b78a7056a] 2: /usr/bin/ceph-osd(+0x385bcb) [0x560b789f0bcb] 3: main() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f93d165dd90] 5: __libc_start_main() 6: _start() 0> 2025-09-25T18:02:40.037+0000 7f93d0aab8c0 -1 *** Caught signal (Aborted) ** in thread 7f93d0aab8c0 thread_name:ceph-osd ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f93d1676520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x16a) [0x560b78a7061d] 6: /usr/bin/ceph-osd(+0x385bcb) [0x560b789f0bcb] 7: main() 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f93d165dd90] 9: __libc_start_main() 10: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -725> 2025-09-25T18:02:37.157+0000 7f93d0aab8c0 -1 Falling back to public interface -2> 2025-09-25T18:02:40.033+0000 7f93d0aab8c0 -1 osd.21 2098994 init missing pg_pool_t for deleted pool 57 for pg 57.3s7; please downgrade to luminous and allow pg deletion to complete before upgrading -1> 2025-09-25T18:02:40.037+0000 7f93d0aab8c0 -1 ./src/osd/OSD.cc: In function 'int OSD::init()' thread 7f93d0aab8c0 time 2025-09-25T18:02:40.040491+0000 ./src/osd/OSD.cc: 3867: ceph_abort_msg("abort() called") ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb7) [0x560b78a7056a] 2: /usr/bin/ceph-osd(+0x385bcb) [0x560b789f0bcb] 3: main() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f93d165dd90] 5: __libc_start_main() 6: _start() 0> 2025-09-25T18:02:40.037+0000 7f93d0aab8c0 -1 *** Caught signal (Aborted) ** in thread 7f93d0aab8c0 thread_name:ceph-osd ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f93d1676520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x16a) [0x560b78a7061d] 6: /usr/bin/ceph-osd(+0x385bcb) [0x560b789f0bcb] 7: main() 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f93d165dd90] 9: __libc_start_main() 10: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aborted Will deleting the PG help? Is there any way I can recover these OSDs? Will moving back to reef help? Daniel

2 3

Ceph GWCLI issue
by Kardos László 30 Sep '25

30 Sep '25

Hello, We have encountered the following issue in our production environment: A new RBD Image was created within an existing pool, and its status is reported as "unknown" in GWCLI. Based on our tests, this does not appear to cause operational issues, but we would like to investigate the root cause. No relevant information regarding this issue was found in the logs. GWCLI output: o- / .......................................................................... ............................................... [...] o- cluster .......................................................................... ............................... [Clusters: 1] | o- ceph .......................................................................... .................................. [HEALTH_OK] | o- pools .......................................................................... ............................... [Pools: 11] | | o- .mgr ................................................................ [(x3), Commit: 0.00Y/15591725M (0%), Used: 194124K] | | o- .nfs ................................................................. [(x3), Commit: 0.00Y/15591725M (0%), Used: 16924b] | | o- xxxx-test ............................................................. [(2+1), Commit: 0.00Y/23727198M (0%), Used: 0.00Y] | | o- xxxxx-erasure-0 ............................................ [(2+1), Commit: 0.00Y/23727198M (0%), Used: 61519257668K] | | o- xxxxxx-repl ...................................................... [(x3), Commit: 0.00Y/15591725M (0%), Used: 130084b] | | o- cephfs.cephfs-test.data ............................................ [(x3), Commit: 0.00Y/15591725M (0%), Used: 9090444K] | | o- cephfs.cephfs-test.meta .......................................... [(x3), Commit: 0.00Y/15591725M (0%), Used: 516415713b] | | o- xxxxx-data ..................................................... [(3+1), Commit: 0.00Y/9604386M (0%), Used: 7547753556K] | | o- xxxxx-rpl .......................................................... [(x3), Commit: 12.0T/4268616M (294%), Used: 85265b] | | o- xxxxx-data ................................................... [(3+1), Commit: 0.00Y/5011626M (0%), Used: 10955179612K] | | o- replicated_xxxx ............................................... [(x3), Commit: 25.0T/2280846592K (1176%), Used: 46912b] | o- topology .......................................................................... ..................... [OSDs: 42,MONs: 5] o- disks .......................................................................... ............................. [37.0T, Disks: 3] | o- xxxx-rpl .......................................................................... ................... [xxxx-rpl (12.0T)] | | o- xxxxx_lun0 ........................................................................ [xxxx-rpl/xxxxx_lun0 (Online, 12.0T)] | o- replicated_xxxx .......................................................................... ..... [replicated_xxxx (25.0T)] | o- xxxx_lun0 ............................................................... [replicated_xxxx/xxxx_lun0 (Online, 12.0T)] | o- xxxx_lun_new ........................................................ [replicated_xxxx/xxxx_lun_new (Unknown, 13.0T)] The image (xxxx_lun_new) is provisioned to multiple ESXi hosts, mounted, and formatted with VMFS6. The datastore is writable and readable by the hosts. There is a change in the block size of the RBD Image: the older RBD Images use a 4 MiB block size, while the new RBD Image uses a 512 KiB block size. RBD Image Parameters: For replicated_xxxx / xxxx_lun0 (Online status in GWCLI): rbd image 'xxxx_lun0': size 12 TiB in 3145728 objects order 22 (4 MiB objects) snapshot_count: 0 id: 5c1b5ecfdfa46 data_pool: xxxx0-data block_name_prefix: rbd_data.14.5c1b5ecfdfa46 format: 2 features: exclusive-lock, data-pool op_features: flags: create_timestamp: Tue Jul 8 13:02:11 2025 access_timestamp: Thu Sep 25 13:49:47 2025 modify_timestamp: Thu Sep 25 13:50:05 2025 For replicated_xxxx / xxxx_lun_new (Unknown status in GWCLI): rbd image 'xxxx_lun_new': size 13 TiB in 27262976 objects order 19 (512 KiB objects) snapshot_count: 0 id: 1945d9cf9f41ab data_pool: xxxx0-data block_name_prefix: rbd_data.14.1945d9cf9f41ab format: 2 features: exclusive-lock, data-pool op_features: flags: create_timestamp: Wed Sep 24 11:21:21 2025 access_timestamp: Thu Sep 25 13:50:42 2025 modify_timestamp: Thu Sep 25 13:49:48 2025 Pool Parameters: pool 14 'replicated_xxxx' replicated size 3 min_size 2 crush_rule 7 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 30743 flags hashpspool stripe_width 0 application rbd,rgw Ceph version: ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Question: What could be causing the RBD Image (xxxx_lun_new) to appear in an "unknown" state in GWCLI?

6 7

ceph 19.2.3 - how to identify the OSD disk
by Steven Vacaroaia 29 Sep '25

29 Sep '25

Hi, There must be something basic that I am missing but I have an "osd down" issue and cannot pinpoint which disk need to be replaced Steps I took 1. "Identify the problem" cepg osd stat 185 osds: 184 up (since 3d), 184 in (since 3d); epoch: e47876; 941 remapped pgs 2." identify the OSD" ceph osd tree | grep down 51 ssd_class 6.98630 osd.51 down 0 1.00000 osd.51 ceph-host-3 stopped 3m ago 9w - 26.1G <unknown> <unknown> <unknown> 3. "check the block" ls -al /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/osd.51 block -> /dev/ceph-419569a5-9a68-4688-8cf4-17ee450a8659/osd-block-75fe30c3-8cfa-49c7-80a1-6d79c3e4b4e6 4. "find the device" ls -al /dev/ceph-419569a5-9a68-4688-8cf4-17ee450a8659/osd-block-75fe30c3-8cfa-49c7-80a1-6d79c3e4b4e6 /dev/dm-7 5. "what type " blkid /dev/dm-7 /dev/dm-7: TYPE="ceph_bluestore" This is weird since osd 51 was type "ssd_class" so just a regular OSD deployed on SSD drive 5. which disk has dm-7 lsblk -o NAME,SERIAL | grep ceph-419569a5-9a68-4688-8cf4-17ee450a8659/osd-block-75fe30c3-8cfa-49c7-80a1-6d79c3e4b4e6 no results ?? blkid /dev/dm-7 /dev/dm-7: TYPE="ceph_bluestore"

2 2

Ceph MGR balancer violates EC CRUSH rules
by Denis Polom 29 Sep '25

29 Sep '25

Hi folks, on our Ceph v19.2.1 distributed across 4 failure domains we have a EC pool using following crush rule: { "rule_id": 4, "rule_name": "ec53", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_indep", "num": 4, "type": "tor" }, { "op": "choose_indep", "num": 2, "type": "osd" }, { "op": "emit" } ] } and following EC profile crush-device-class= crush-failure-domain=tor crush-root=default jerasure-per-chunk-alignment=false k=5 m=3 plugin=jerasure technique=reed_sol_van w=8 here is our crush tree: ID CLASS WEIGHT TYPE NAME -40 15.71931 root metadata -41 3.49318 tor s1-bos-e-10-ng.metadata -42 1.74660 host osd1-metadata 98 ssd 1.74660 osd.98 -49 1.74660 host osd2-metadata 99 ssd 1.74660 osd.99 -124 3.49318 tor s1-bos-f-7-ng.metadata -52 1.74660 host osd3-metadata 103 ssd 1.74660 osd.103 -76 1.74660 host osd9-metadata 106 ssd 1.74660 osd.106 -56 5.23978 tor s1-bos-f-8-ng.metadata -55 1.74660 host osd4-metadata 85 ssd 1.74660 osd.85 -61 1.74660 host osd5-metadata 93 ssd 1.74660 osd.93 -64 1.74660 host osd6-metadata 97 ssd 1.74660 osd.97 -67 3.49318 tor s1-bos-f-9-ng.metadata -70 1.74660 host osd7-metadata 104 ssd 1.74660 osd.104 -73 1.74660 host osd8-metadata 105 ssd 1.74660 osd.105 -1 2733.78516 root default -21 745.57776 tor s1-bos-e-10-ng -7 80.40544 host osd1 0 hdd 7.30959 osd.0 10 hdd 7.30959 osd.10 26 hdd 7.30959 osd.26 28 hdd 7.30959 osd.28 44 hdd 7.30959 osd.44 53 hdd 7.30959 osd.53 62 hdd 7.30959 osd.62 70 hdd 7.30959 osd.70 79 hdd 7.30959 osd.79 89 hdd 7.30959 osd.89 107 hdd 7.30959 osd.107 -88 116.95337 host osd12 128 hdd 7.30959 osd.128 139 hdd 7.30959 osd.139 142 hdd 7.30959 osd.142 146 hdd 7.30959 osd.146 150 hdd 7.30959 osd.150 152 hdd 7.30959 osd.152 153 hdd 7.30959 osd.153 154 hdd 7.30959 osd.154 191 hdd 7.30959 osd.191 192 hdd 7.30959 osd.192 193 hdd 7.30959 osd.193 194 hdd 7.30959 osd.194 195 hdd 7.30959 osd.195 196 hdd 7.30959 osd.196 199 hdd 7.30959 osd.199 200 hdd 7.30959 osd.200 -94 116.95337 host osd15 156 hdd 7.30959 osd.156 157 hdd 7.30959 osd.157 158 hdd 7.30959 osd.158 159 hdd 7.30959 osd.159 160 hdd 7.30959 osd.160 161 hdd 7.30959 osd.161 162 hdd 7.30959 osd.162 163 hdd 7.30959 osd.163 164 hdd 7.30959 osd.164 165 hdd 7.30959 osd.165 166 hdd 7.30959 osd.166 167 hdd 7.30959 osd.167 171 hdd 7.30959 osd.171 172 hdd 7.30959 osd.172 177 hdd 7.30959 osd.177 183 hdd 7.30959 osd.183 -17 80.40544 host osd2 4 hdd 7.30959 osd.4 15 hdd 7.30959 osd.15 19 hdd 7.30959 osd.19 33 hdd 7.30959 osd.33 38 hdd 7.30959 osd.38 52 hdd 7.30959 osd.52 58 hdd 7.30959 osd.58 69 hdd 7.30959 osd.69 73 hdd 7.30959 osd.73 81 hdd 7.30959 osd.81 95 hdd 7.30959 osd.95 -115 87.71503 host osd22 275 hdd 7.30959 osd.275 276 hdd 7.30959 osd.276 277 hdd 7.30959 osd.277 278 hdd 7.30959 osd.278 279 hdd 7.30959 osd.279 280 hdd 7.30959 osd.280 281 hdd 7.30959 osd.281 282 hdd 7.30959 osd.282 283 hdd 7.30959 osd.283 284 hdd 7.30959 osd.284 285 hdd 7.30959 osd.285 286 hdd 7.30959 osd.286 -118 87.71503 host osd23 287 hdd 7.30959 osd.287 288 hdd 7.30959 osd.288 289 hdd 7.30959 osd.289 290 hdd 7.30959 osd.290 291 hdd 7.30959 osd.291 292 hdd 7.30959 osd.292 293 hdd 7.30959 osd.293 294 hdd 7.30959 osd.294 295 hdd 7.30959 osd.295 296 hdd 7.30959 osd.296 297 hdd 7.30959 osd.297 298 hdd 7.30959 osd.298 -130 87.71503 host osd25 311 hdd 7.30959 osd.311 312 hdd 7.30959 osd.312 313 hdd 7.30959 osd.313 314 hdd 7.30959 osd.314 315 hdd 7.30959 osd.315 316 hdd 7.30959 osd.316 317 hdd 7.30959 osd.317 318 hdd 7.30959 osd.318 319 hdd 7.30959 osd.319 320 hdd 7.30959 osd.320 321 hdd 7.30959 osd.321 322 hdd 7.30959 osd.322 -134 87.71503 host osd29 359 hdd 7.30959 osd.359 360 hdd 7.30959 osd.360 361 hdd 7.30959 osd.361 362 hdd 7.30959 osd.362 363 hdd 7.30959 osd.363 364 hdd 7.30959 osd.364 365 hdd 7.30959 osd.365 366 hdd 7.30959 osd.366 367 hdd 7.30959 osd.367 368 hdd 7.30959 osd.368 369 hdd 7.30959 osd.369 370 hdd 7.30959 osd.370 -127 774.81604 tor s1-bos-f-7-ng -79 116.95337 host osd10 108 hdd 7.30959 osd.108 109 hdd 7.30959 osd.109 110 hdd 7.30959 osd.110 111 hdd 7.30959 osd.111 112 hdd 7.30959 osd.112 113 hdd 7.30959 osd.113 114 hdd 7.30959 osd.114 115 hdd 7.30959 osd.115 116 hdd 7.30959 osd.116 117 hdd 7.30959 osd.117 118 hdd 7.30959 osd.118 119 hdd 7.30959 osd.119 120 hdd 7.30959 osd.120 121 hdd 7.30959 osd.121 122 hdd 7.30959 osd.122 123 hdd 7.30959 osd.123 -85 116.95337 host osd11 125 hdd 7.30959 osd.125 126 hdd 7.30959 osd.126 127 hdd 7.30959 osd.127 129 hdd 7.30959 osd.129 138 hdd 7.30959 osd.138 144 hdd 7.30959 osd.144 149 hdd 7.30959 osd.149 151 hdd 7.30959 osd.151 188 hdd 7.30959 osd.188 189 hdd 7.30959 osd.189 190 hdd 7.30959 osd.190 197 hdd 7.30959 osd.197 198 hdd 7.30959 osd.198 201 hdd 7.30959 osd.201 202 hdd 7.30959 osd.202 203 hdd 7.30959 osd.203 -91 116.95337 host osd13 130 hdd 7.30959 osd.130 134 hdd 7.30959 osd.134 135 hdd 7.30959 osd.135 140 hdd 7.30959 osd.140 143 hdd 7.30959 osd.143 147 hdd 7.30959 osd.147 148 hdd 7.30959 osd.148 155 hdd 7.30959 osd.155 168 hdd 7.30959 osd.168 169 hdd 7.30959 osd.169 170 hdd 7.30959 osd.170 176 hdd 7.30959 osd.176 178 hdd 7.30959 osd.178 180 hdd 7.30959 osd.180 182 hdd 7.30959 osd.182 184 hdd 7.30959 osd.184 -121 87.71503 host osd24 299 hdd 7.30959 osd.299 300 hdd 7.30959 osd.300 301 hdd 7.30959 osd.301 302 hdd 7.30959 osd.302 303 hdd 7.30959 osd.303 304 hdd 7.30959 osd.304 305 hdd 7.30959 osd.305 306 hdd 7.30959 osd.306 307 hdd 7.30959 osd.307 308 hdd 7.30959 osd.308 309 hdd 7.30959 osd.309 310 hdd 7.30959 osd.310 -131 87.71503 host osd26 323 hdd 7.30959 osd.323 324 hdd 7.30959 osd.324 325 hdd 7.30959 osd.325 326 hdd 7.30959 osd.326 327 hdd 7.30959 osd.327 328 hdd 7.30959 osd.328 329 hdd 7.30959 osd.329 330 hdd 7.30959 osd.330 331 hdd 7.30959 osd.331 332 hdd 7.30959 osd.332 333 hdd 7.30959 osd.333 334 hdd 7.30959 osd.334 -5 80.40544 host osd3 2 hdd 7.30959 osd.2 12 hdd 7.30959 osd.12 21 hdd 7.30959 osd.21 34 hdd 7.30959 osd.34 42 hdd 7.30959 osd.42 50 hdd 7.30959 osd.50 54 hdd 7.30959 osd.54 63 hdd 7.30959 osd.63 100 hdd 7.30959 osd.100 101 hdd 7.30959 osd.101 102 hdd 7.30959 osd.102 -135 87.71503 host osd30 371 hdd 7.30959 osd.371 372 hdd 7.30959 osd.372 373 hdd 7.30959 osd.373 374 hdd 7.30959 osd.374 375 hdd 7.30959 osd.375 376 hdd 7.30959 osd.376 377 hdd 7.30959 osd.377 378 hdd 7.30959 osd.378 379 hdd 7.30959 osd.379 380 hdd 7.30959 osd.380 381 hdd 7.30959 osd.381 382 hdd 7.30959 osd.382 -13 80.40544 host osd9 8 hdd 7.30959 osd.8 13 hdd 7.30959 osd.13 24 hdd 7.30959 osd.24 32 hdd 7.30959 osd.32 39 hdd 7.30959 osd.39 48 hdd 7.30959 osd.48 56 hdd 7.30959 osd.56 64 hdd 7.30959 osd.64 75 hdd 7.30959 osd.75 84 hdd 7.30959 osd.84 94 hdd 7.30959 osd.94 -25 584.76685 tor s1-bos-f-8-ng -97 87.71503 host osd16 216 hdd 7.30959 osd.216 217 hdd 7.30959 osd.217 218 hdd 7.30959 osd.218 219 hdd 7.30959 osd.219 220 hdd 7.30959 osd.220 221 hdd 7.30959 osd.221 222 hdd 7.30959 osd.222 223 hdd 7.30959 osd.223 224 hdd 7.30959 osd.224 225 hdd 7.30959 osd.225 226 hdd 7.30959 osd.226 227 hdd 7.30959 osd.227 -100 87.71503 host osd17 204 hdd 7.30959 osd.204 205 hdd 7.30959 osd.205 206 hdd 7.30959 osd.206 207 hdd 7.30959 osd.207 208 hdd 7.30959 osd.208 209 hdd 7.30959 osd.209 210 hdd 7.30959 osd.210 211 hdd 7.30959 osd.211 212 hdd 7.30959 osd.212 213 hdd 7.30959 osd.213 214 hdd 7.30959 osd.214 215 hdd 7.30959 osd.215 -103 80.40544 host osd18 228 hdd 7.30959 osd.228 229 hdd 7.30959 osd.229 230 hdd 7.30959 osd.230 231 hdd 7.30959 osd.231 232 hdd 7.30959 osd.232 233 hdd 7.30959 osd.233 234 hdd 7.30959 osd.234 235 hdd 7.30959 osd.235 236 hdd 7.30959 osd.236 237 hdd 7.30959 osd.237 238 hdd 7.30959 osd.238 -132 87.71503 host osd27 335 hdd 7.30959 osd.335 336 hdd 7.30959 osd.336 337 hdd 7.30959 osd.337 338 hdd 7.30959 osd.338 339 hdd 7.30959 osd.339 340 hdd 7.30959 osd.340 341 hdd 7.30959 osd.341 342 hdd 7.30959 osd.342 343 hdd 7.30959 osd.343 344 hdd 7.30959 osd.344 345 hdd 7.30959 osd.345 346 hdd 7.30959 osd.346 -11 80.40544 host osd4 3 hdd 7.30959 osd.3 16 hdd 7.30959 osd.16 18 hdd 7.30959 osd.18 31 hdd 7.30959 osd.31 36 hdd 7.30959 osd.36 49 hdd 7.30959 osd.49 59 hdd 7.30959 osd.59 68 hdd 7.30959 osd.68 72 hdd 7.30959 osd.72 77 hdd 7.30959 osd.77 82 hdd 7.30959 osd.82 -19 80.40544 host osd5 6 hdd 7.30959 osd.6 11 hdd 7.30959 osd.11 25 hdd 7.30959 osd.25 27 hdd 7.30959 osd.27 41 hdd 7.30959 osd.41 46 hdd 7.30959 osd.46 60 hdd 7.30959 osd.60 66 hdd 7.30959 osd.66 80 hdd 7.30959 osd.80 88 hdd 7.30959 osd.88 91 hdd 7.30959 osd.91 -3 80.40544 host osd6 7 hdd 7.30959 osd.7 14 hdd 7.30959 osd.14 20 hdd 7.30959 osd.20 30 hdd 7.30959 osd.30 37 hdd 7.30959 osd.37 47 hdd 7.30959 osd.47 57 hdd 7.30959 osd.57 67 hdd 7.30959 osd.67 74 hdd 7.30959 osd.74 83 hdd 7.30959 osd.83 96 hdd 7.30959 osd.96 -23 628.62439 tor s1-bos-f-9-ng -82 116.95337 host osd14 124 hdd 7.30959 osd.124 131 hdd 7.30959 osd.131 132 hdd 7.30959 osd.132 133 hdd 7.30959 osd.133 136 hdd 7.30959 osd.136 137 hdd 7.30959 osd.137 141 hdd 7.30959 osd.141 145 hdd 7.30959 osd.145 173 hdd 7.30959 osd.173 174 hdd 7.30959 osd.174 175 hdd 7.30959 osd.175 179 hdd 7.30959 osd.179 181 hdd 7.30959 osd.181 185 hdd 7.30959 osd.185 186 hdd 7.30959 osd.186 187 hdd 7.30959 osd.187 -106 87.71503 host osd19 239 hdd 7.30959 osd.239 240 hdd 7.30959 osd.240 241 hdd 7.30959 osd.241 242 hdd 7.30959 osd.242 243 hdd 7.30959 osd.243 244 hdd 7.30959 osd.244 245 hdd 7.30959 osd.245 246 hdd 7.30959 osd.246 247 hdd 7.30959 osd.247 248 hdd 7.30959 osd.248 249 hdd 7.30959 osd.249 250 hdd 7.30959 osd.250 -109 87.71503 host osd20 251 hdd 7.30959 osd.251 252 hdd 7.30959 osd.252 253 hdd 7.30959 osd.253 254 hdd 7.30959 osd.254 255 hdd 7.30959 osd.255 256 hdd 7.30959 osd.256 257 hdd 7.30959 osd.257 258 hdd 7.30959 osd.258 259 hdd 7.30959 osd.259 260 hdd 7.30959 osd.260 261 hdd 7.30959 osd.261 262 hdd 7.30959 osd.262 -112 87.71503 host osd21 263 hdd 7.30959 osd.263 264 hdd 7.30959 osd.264 265 hdd 7.30959 osd.265 266 hdd 7.30959 osd.266 267 hdd 7.30959 osd.267 268 hdd 7.30959 osd.268 269 hdd 7.30959 osd.269 270 hdd 7.30959 osd.270 271 hdd 7.30959 osd.271 272 hdd 7.30959 osd.272 273 hdd 7.30959 osd.273 274 hdd 7.30959 osd.274 -133 87.71503 host osd28 347 hdd 7.30959 osd.347 348 hdd 7.30959 osd.348 349 hdd 7.30959 osd.349 350 hdd 7.30959 osd.350 351 hdd 7.30959 osd.351 352 hdd 7.30959 osd.352 353 hdd 7.30959 osd.353 354 hdd 7.30959 osd.354 355 hdd 7.30959 osd.355 356 hdd 7.30959 osd.356 357 hdd 7.30959 osd.357 358 hdd 7.30959 osd.358 -15 80.40544 host osd7 1 hdd 7.30959 osd.1 9 hdd 7.30959 osd.9 22 hdd 7.30959 osd.22 35 hdd 7.30959 osd.35 43 hdd 7.30959 osd.43 51 hdd 7.30959 osd.51 55 hdd 7.30959 osd.55 65 hdd 7.30959 osd.65 76 hdd 7.30959 osd.76 86 hdd 7.30959 osd.86 92 hdd 7.30959 osd.92 -9 80.40544 host osd8 5 hdd 7.30959 osd.5 17 hdd 7.30959 osd.17 23 hdd 7.30959 osd.23 29 hdd 7.30959 osd.29 40 hdd 7.30959 osd.40 45 hdd 7.30959 osd.45 61 hdd 7.30959 osd.61 71 hdd 7.30959 osd.71 78 hdd 7.30959 osd.78 87 hdd 7.30959 osd.87 90 hdd 7.30959 osd.90 Ceph MGR balancer is running in upmap mode, clients are "luminous". During one FD s1-bos-f-9-ng outage there were many PGs in inactive state, for example: pg 11.558 is stuck inactive for 102s, current state undersized+degraded+peered, last acting [176,169,NONE,NONE,199,NONE,227,221] this clearly shows that PG was distributed unevenly through FDs as according crush rule it should be placed on 2 OSDs per FD. # for i in 176 169 199 227 221;do echo -n "$i: "; ceph osd find $i | jq -r .crush_location.tor;done 176: s1-bos-f-7-ng 169: s1-bos-f-7-ng 199: s1-bos-e-10-ng 227: s1-bos-f-8-ng 221: s1-bos-f-8-ng # ceph osd tree | grep tor -41 3.49318 tor s1-bos-e-10-ng.metadata -124 3.49318 tor s1-bos-f-7-ng.metadata -56 5.23978 tor s1-bos-f-8-ng.metadata -67 3.49318 tor s1-bos-f-9-ng.metadata -21 745.57776 tor s1-bos-e-10-ng -127 774.81604 tor s1-bos-f-7-ng -25 584.76685 tor s1-bos-f-8-ng -23 628.62439 tor s1-bos-f-9-ng after our FD came up I did run pg-upmap-items manually to make placement correct. But after enabling MGR balancer there were attempts to move PG placement again to different OSD under different FD that breaks crush rule. Now I know this cluster FD weights are very different in CRUSH map. So I've tried to use osdmaptool to simulate balancer. Our balancer settings were: mgr advanced mgr/balancer/mode upmap mgr advanced mgr/balancer/upmap_max_deviation 1 mgr advanced mgr/balancer/upmap_max_optimizations 100 So I tried to adjust max deviation to level, that balancer would not try to move PGs to OSDs in different FD that breaks crush rule. I ended up on value 50, and honestly don't know if at this value balancer will try to do anything anytime. From my perspective MGR balancer is ignoring CRUSH rule - at least on EC pools. I observed this behaviour on our another Ceph cluster and also only on EC pool even if FD weights there were almost same. I even tried TheJJ Ceph-balancer, and it breaks crush rule also. Any ideas or similar experiences? Thank you

2 4

Seemingly stuck scrubbing and snaptrims
by Gustavo Garcia Rondina 27 Sep '25

27 Sep '25

Hi Ceph community, We have a Ceph cluster with 6 OSD nodes and 168 OSDs (28 per node each with a 18 TB data disk). We are running quincy 17.2.6. The cluster was not properly maintained for a while and it was in not the greatest shape. After some work, we got here: [ceph: root@ceph-admin1 /]# ceph -s cluster: id: 26315dca-383a-11ee-9d49-xxxxxxxxxxxx health: HEALTH_WARN 257 pgs not deep-scrubbed in time 259 pgs not scrubbed in time services: mon: 5 daemons, quorum ceph-admin1,ceph-admin2,ceph-osd1,ceph-osd2,ceph-osd3 (age 6M) mgr: ceph-admin2.sipadf(active, since 1y), standbys: ceph-admin1.nwaovh mds: 2/2 daemons up, 2 standby osd: 168 osds: 168 up (since 6w), 168 in (since 6w) data: volumes: 2/2 healthy pools: 9 pools, 2273 pgs objects: 699.66M objects, 1.5 PiB usage: 2.2 PiB used, 529 TiB / 2.7 PiB avail pgs: 1986 active+clean 125 active+clean+snaptrim_wait 110 active+clean+scrubbing+deep 47 active+clean+snaptrim 5 active+clean+scrubbing The snaptrim_wait and snaptrim numbers are stuck on these values for weeks, if not months. We also had almost all PGs not (deep-)scrubbed in time, but after tweaking a few parameters the numbers got down to what you see above. However, now they seem to not be lowering, or doing so very slowly (a couple every 2-3 days). Dumping all PGs and filtering for the SCRUB_SCHEDULING column, I can see that some PGs have been scrubbing for a long time: [root@ceph-admin1 ~]# ceph pg dump | grep -e 'scrubbing for' -e SCHED | awk '{print $1,$(NF-2)}' | sort -n -k2 | tail 5.624 4788773s 5.5f2 4790148s 5.4b 4791924s 5.686 4795526s 5.538 4796221s 5.5c8 4796573s 5.722 4796937s 5.483 4797551s 5.81 4799856s 5.233 10052851s While recent ones look reasonable, so new scrubs are starting: [root@ceph-admin1 ~]# ceph pg dump | grep -e 'scrubbing for' -e SCHED | awk '{print $1,$(NF-2)}' | sort -n -k2 | head PG_STAT SCRUB_SCHEDULING 5.14b 53s 5.382 71s 5.ff 187s 6.9 265s 5.1c3 354s 5.70a 364s 5.596 367s 5.1fc 897s 5.3c8 1420s Looking closer to PG 5.233, it has a long snaptrimq length, but it doesn't seem to be blocked: [root@ceph-admin1 ~]# ceph pg 5.233 query | grep -E 'snaptrimq_len|blocked_by|waiting_on' "snaptrimq_len": 2682, "blocked_by": [], However it does seem to be waiting on someone: [root@ceph-admin1 ~]# ceph pg 5.233 query | grep -A5 'waiting_on_whom' "waiting_on_whom": [ "164(4)" ], "schedule": "scrubbing" The primary OSD for PG 5.233 is osd.136 on node ceph-osd4, and its logs do not show anything remarkable. The HDD for osd.136 is in good shape, no SMART errors, no I/O errors anywhere on ceph-osd4. Is it the scrubbing that's blocking the snaptrim, or the other way? Also, looking at historic ops: [root@ceph-admin1 ~]# ceph tell osd.136 dump_historic_ops_by_duration { "size": 20, "duration": 600, ... "ops": [ { "description": "rep_scrubmap(6.9 e14586 from shard 151)", "initiated_at": "2025-09-25T09:57:46.150913-0500", "age": 356.3390488, "duration": 0.12764220300000001, "type_data": { "flag_point": "started", "events": [ ... { "event": "header_read", "time": "2025-09-25T09:57:46.150912-0500", "duration": 4294967295.9999995 }, This last duration above looks odd as 4294967295.9999995 is 2^32. Not sure what it means, but thought it was strange. Any clues on any of this? Thank you, Gustavo

2 1

Ceph Community Round-up: New Member, Blogs, Survey & Cephalocon!
by Anthony Middleton 26 Sep '25

26 Sep '25

Hello Ceph Friends! I hope this week has gone well for you. Before logging off for the weekend, I wanted to share a few updates and announcements. Here's what's happening in our community this week: *Welcome, Western Digital!*The Ceph Foundation is excited to welcome Western Digital as our newest member. More details soon! Meet our members - https://ceph.io/en/foundation/members/ *Why should you attend Cephalocon 2025?*Long-time Ceph Ambassador Frédéric Nass shares why he's excited for Vancouver. Read his blog - https://ceph.io/en/news/blog/2025/cephalocon-blog/. <https://ceph.io/en/news/blog/2025/cephalocon-blog/> *Tech Deep Dive*Check out the 4-part blog series by Daniel Alexander Parkes and Anthony D'Atri on Health-Aware Load Balancing for Ceph RGW with Consul. - Part 1: Building Health-Aware Load Balancing for the Ceph Object Gateway (RGW)with Consul <https://ceph.io/en/news/blog/2025/consul-lb1/> - Part 2: Seamless DNS Routing with CoreDNS and Virtual-Host Style Bucket Access <https://ceph.io/en/news/blog/2025/consul-lb2/> - Part 3. Consul-Powered Global Load Balancing for Ceph Object Gateway (RGW) <https://ceph.io/en/news/blog/2025/consul-lb3/> - Part 4. Consul-Powered Global Load Balancing for Ceph Object Gateway (RGW) <https://ceph.io/en/news/blog/2025/consul-lb4/> *Your Feedback Needed*Help guide Ceph's future by answering our one-question survey on operability. Take the survey - https://forms.gle/RmFeS2CE1BVj4LPz8 *Cephalocon 2025 is Almost Here!*We are nearly one month away from Cephalocon 2025 in Vancouver. Promote the registration link within your social media. Registration link - https://events.linuxfoundation.org/cephalocon/register/. Also, please note the Ceph New Users Workshop on October 27th - https://events.clyso.tech/event/2/registrations/2/. *Join the Cephalocon Slack Channel*Your go-to space for travel tips, schedule updates, and after-events fun. Lastly, if you are interested in becoming more active in our community, please reach out to me at community-manager(a)ceph.io. I am always happy to chat about our community and how we can support one another. Take care! -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Missing directories on download.ceph.com?
by Oliver Freyermuth 25 Sep '25

25 Sep '25

Am 25.09.25 um 17:54 schrieb Matt Vandermeulen: > Interesting, from our mirror it looks like these (and more) are absolute symlinks: > > lrwxrwxrwx 1 root root 24 Nov 14 2024 debian-15.2.16 -> /overflow/debian-15.2.16 > lrwxrwxrwx 1 root root 24 Nov 14 2024 debian-15.2.17 -> /overflow/debian-15.2.17 > lrwxrwxrwx 1 root root 24 Nov 14 2024 debian-octopus -> /overflow/debian-octopus > lrwxrwxrwx 1 root root 21 Nov 14 2024 rpm-15.2.16 -> /overflow/rpm-15.2.16 > lrwxrwxrwx 1 root root 21 Nov 14 2024 rpm-15.2.17 -> /overflow/rpm-15.2.17 > lrwxrwxrwx 1 root root 21 Nov 14 2024 rpm-octopus -> /overflow/rpm-octopus > > Looks like it has been this way for quite some time based on our zfs snapshots: > > lrwxrwxrwx 1 1109 1109 11 Aug 9 2022 .zfs/snapshot/2024-11-01-0620/rpm-octopus -> rpm-15.2.17 > lrwxrwxrwx 1 root root 21 Nov 14 2024 .zfs/snapshot/2024-12-01-0620/rpm-octopus -> /overflow/rpm-octopus > > If you need 15.2.16, a snapshot of it is accessible via: https://mirror.reenigne.net/ceph-snapshots/2024-11-01-0620/rpm-15.2.16/ Many thanks for the helpful mirroring of the snapshots! The way we noticed is since some of our older clients started to report: - Status code: 404 for https://download.ceph.com/rpm-octopus/el8/x86_64/repodata/repomd.xml (IP: 131.220.166.104) today when running DNF, so maybe the absolute symlink target "/overflow/" was only available on the download.ceph.com server and was purged today, or on a separate storage which failed... Cheers and thanks, Oliver > > --- > Thanks, > Matt Vandermeulen > > Software Engineering, and Server Solutions > Reenigne > > Ph: 716.217.0502 > > On 2025-09-25 12:41, Oliver Freyermuth wrote: >> Dear Cephalopodians, >> >> it seems that recently some directories were emptied / became inaccessible on download.ceph.com, for example (some randomly selected ones): >> >> https://download.ceph.com/rpm-octopus >> https://download.ceph.com/rpm-15.2.16 >> >> As older releases such as: >> >> https://download.ceph.com/rpm-mimic/ >> >> are still there, I presume this was not intentional, so I am alerting here in case there might for example be a file system problem on the download mirror. >> >> Cheers, >> Oliver -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

2 2

Poor cephfs kernel client performance on 512-thread AMD EPYC systems
by Vladimir Brik 25 Sep '25

25 Sep '25

Hello We have two computing clusters, a larger one with old machines running Centos 7, and a new test cluster consisting of three 512-thread AMD EPYC system running AlmaLinux 9 (5.14.0-570.42.2 kernel). Both clusters use cephfs 17.2.5 as their shared file system. Everything is fine when metadata-intensive jobs are spread-out among the compute nodes of the older cluster (no more than one or two dozen of those jobs per system). However, if a large number of metadata-intensive jobs run on our big 512-thread systems, ceph performance essentially collapses (at least 60x slowdown compared to the old cluster), with processes spending most of their time in the kernel instead of doing their work. Under load, the EPYC compute nodes have plenty of spare memory; NICs are barely utilized; average CPU utilization 50-80%, almost all of it in sys. Our cephfs metadata pool is on SSDs. It seems like local lock contention is the bottleneck, rather than the MDSes (which are busy-ish, but are not obviously overloaded). I have attached two flamegraphs illustrating the situation. * slowpath.svg shows what happens when a lot (200-500) of processes are frequently calling the stat system call on the same files. * osq_lock.svg show what happens when, I think, a lot of processes are either listing a directory or accessing files in a directory (with about 1000 files). We have also observed MDSes frequently evicting the big EPIC machines, but I haven't figured out what workloads trigger that yet. Does anybody have any advice on how to improve cephfs performance/stability on large systems? Thanks in advance Vlad

1 0

Missing directories on download.ceph.com?
by Oliver Freyermuth 25 Sep '25

25 Sep '25

Dear Cephalopodians, it seems that recently some directories were emptied / became inaccessible on download.ceph.com, for example (some randomly selected ones): https://download.ceph.com/rpm-octopus https://download.ceph.com/rpm-15.2.16 As older releases such as: https://download.ceph.com/rpm-mimic/ are still there, I presume this was not intentional, so I am alerting here in case there might for example be a file system problem on the download mirror. Cheers, Oliver -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

1 0