- ceph-users - lists.ceph.io

ceph-volume partial failure with multiple OSDs per device
by Elias Carter 22 Jul '25

22 Jul '25

I provision OSDs with the following command: ``` ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --dmcrypt --osds-per-device 2 /dev/nvme0n1 --crush-device-class nvme ``` What I have observed is that sometimes ceph-volume will provision the first OSD successfully, but then it will fail to provision the second OSD. When this happens ceph-volume does not attempt to clean up the first OSD which was created. My question is what should be done in this scenario? Can I re-run the ceph-volume command and it is smart enough to see the first OSD is already provisioned and then create the missing second OSD? Or should I zap and purge the OSD which was created and then re-run the ceph-volume command? Elias

2 1

HELP! Cluster usage increased after adding new nodes/osd's
by mhnx 22 Jul '25

22 Jul '25

Hello! Few years ago I build a "dc-a:12 + dc-b:12 = 24" node ceph cluster with Nautilus v14.2.16 A year ago the cluster upgraded to Octopus and it was running fine. Recently I added 4+4=8 new nodes with identical hardware and SSD drives. When I created OSD's with Octopus, The cluster usage increased from %50 to %78!! The weird problem is, the new OSD's become nearfull and hold more size even if they have the same or less amount of PG's. I had to reweight new OSD's to 0.9 to make them equal size usage.. I increased the PG count 8192 to 16384 and ran balancer, it became worse and I have %84 usage now! I guess OSD or PG code changed between nautilus <-> octopus and it generates this problem. Can anyone help me with experience or knowledge about this? What should I do? My solution idea: I'm thinking of destroy and re-create old OSD's as a solution but I need to re-create 144x3.8TB Sas SSD OSD's and it means 4-5 days of maintenance. Also I have 2 osd per drive because it was recommended at Nautilus times. How about this? Should I keep the config or should I use 1 osd per 3.8TB SAS SSD ? What is the recommendation for Octopus and Quincy? - Best regards.

4 4

Re: squid 19.2.2 - cannot remove 'unknown" OSD
by Steven Vacaroaia 21 Jul '25

21 Jul '25

Hi Anthony, There was a leftover folder which I renamed I have also disablled the service and been able to remove it from crush and delete auth but still is showing when using ceph orch ps and still unable to "ceph osd rm 8" because "unable to find OSDs ['8']" Anything else that I can try (other than a reboot) ? Many thanks Steven On Mon, 21 Jul 2025 at 17:25, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Look at /var/lib/ceph on ceph-host-7 for a leftover directory for osd.8 > > Also try > ceph osd crush remove osd.8 > ceph auth del osd.8 > ceph osd rm 8 > > > On Jul 21, 2025, at 5:12 PM, Steven Vacaroaia <stef97(a)gmail.com> wrote: > > > > Hi, > > > > I meant to replace a defective NVME used by osd.8 > > but I might have used the wrong approach as I ended up with a new OSD > while osd.8 show as "unknown" > > > > I tried "ceph orch osd rm " but it say "not found" > > > > I would appreciate some guidance > > > > I can remove the new one as the server is still "noout" and recreate > properly > > or > > remove osd.8 ( but how ?) > > > > Please let me know what would be the best action > > > > <image.png> > > Steven > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

squid 19.2.2 - cannot remove 'unknown" OSD
by Steven Vacaroaia 21 Jul '25

21 Jul '25

Hi, I meant to replace a defective NVME used by osd.8 but I might have used the wrong approach as I ended up with a new OSD while osd.8 show as "unknown" I tried "ceph orch osd rm " but it say "not found" I would appreciate some guidance I can remove the new one as the server is still "noout" and recreate properly or remove osd.8 ( but how ?) Please let me know what would be the best action [image: image.png] Steven

1 0

Cephfs client deadlock (OSD op state mismatch after stuck ops?)
by Hector Martin 21 Jul '25

21 Jul '25

Hi all, I'm running a small converged Ceph cluster (mons, OSDs, mds on the same 3 hosts, one of which is also a CephFS client). I noticed that sometimes the CephFS client seems to deadlock. There are other CephFS clients on the network, but they generally don't collide in file/dir accesses much with the local client. I found this old thread that describes a similar situation: https://www.mail-archive.com/search?l=ceph-users@lists.ceph.com&q=subject:%… The kernel version is 6.14.6 for the main hosts, 6.14.8 for a few other unrelated clients, and 6.3.2 for one older client. Ceph version is 19.2.2. Ceph status was clean and dmesg didn't show any warnings around the time of the hang, other than a few of these (which I always get and seem to be related to SELinux attributes, but appear to be harmless): [3447915.841214] ceph: [667e28f4-8964-454b-b777-03ce12b67742 6505232]: sync 00000000f6e49a9a 100033726fb.152 during filling trace [3447915.851171] SELinux: inode_doinit_use_xattr: getxattr returned 16 for dev=ceph ino=1099565573883 Restarting the mds (letting it failover to the standby-replay mds) did not help the situation, but after finding the above thread I noticed that all the requests were stuck on the same OSD: > root@flamingo:~ -# cat /sys/kernel/debug/ceph/*/osdc > REQUESTS 17 homeless 0 > 46841260 osd17 14.2ab058d9 14.19s0 [17,18,15,6,7,4,3,0,1]/17 [17,18,15,6,7,4,3,0,1]/17 e25593 10002e55314.0000003c 0x400014 1 read > 46841268 osd17 14.571aa108 14.8s0 [17,16,15,0,1,2,4,5,7]/17 [17,16,15,0,1,2,4,5,7]/17 e25593 100030edef7.00000039 0x400014 1 read > 46841311 osd17 14.7b4e2d92 14.12s0 [17,16,15,6,7,4,2,1,3]/17 [17,16,15,6,7,4,2,1,3]/17 e25593 1000318f566.00000179 0x400014 1 read > 46841323 osd17 14.deec222d 14.2ds0 [17,16,18,4,7,5,2,1,0]/17 [17,16,18,4,7,5,2,1,0]/17 e25593 100030edf00.00000028 0x400014 1 read > 46841334 osd17 14.eb940708 14.8s0 [17,16,15,0,1,2,4,5,7]/17 [17,16,15,0,1,2,4,5,7]/17 e25593 10003042460.000001c8 0x400014 1 read > 46841336 osd17 14.93065908 14.8s0 [17,16,15,0,1,2,4,5,7]/17 [17,16,15,0,1,2,4,5,7]/17 e25593 1000318f566.00000181 0x400014 1 read > 46841350 osd17 14.3e9eeaa5 14.25s0 [17,18,15,2,0,1,7,6,4]/17 [17,18,15,2,0,1,7,6,4]/17 e25593 10003498f94.00000000 0x400024 1 write > 46841359 osd17 14.5120499 14.19s0 [17,18,15,6,7,4,3,0,1]/17 [17,18,15,6,7,4,3,0,1]/17 e25593 10002e53590.00000000 0x400014 1 read > 46841367 osd17 14.25e874a5 14.25s0 [17,18,15,2,0,1,7,6,4]/17 [17,18,15,2,0,1,7,6,4]/17 e25593 1000318f566.00000196 0x400014 1 read > 46841376 osd17 14.cd26e465 14.25s0 [17,18,15,2,0,1,7,6,4]/17 [17,18,15,2,0,1,7,6,4]/17 e25593 1000318f566.00000199 0x400014 1 read > 46841405 osd17 14.855295f5 14.35s0 [17,18,15,3,0,2,4,7,5]/17 [17,18,15,3,0,2,4,7,5]/17 e25593 1000318f566.000001a7 0x400014 1 read > 46841421 osd17 14.57d5ef2c 14.2cs0 [17,18,16,1,3,2,4,6,5]/17 [17,18,16,1,3,2,4,6,5]/17 e25593 10002fe07b1.00000008 0x400014 1 read > 46841424 osd17 14.ea554059 14.19s0 [17,18,15,6,7,4,3,0,1]/17 [17,18,15,6,7,4,3,0,1]/17 e25593 1000318f566.000001ac 0x400014 1 read > 46841429 osd17 14.fa91b525 14.25s0 [17,18,15,2,0,1,7,6,4]/17 [17,18,15,2,0,1,7,6,4]/17 e25593 100030916b5.0000004f 0x400014 1 read > 46841447 osd17 14.88f24788 14.8s0 [17,16,15,0,1,2,4,5,7]/17 [17,16,15,0,1,2,4,5,7]/17 e25593 100032d95c7.00000011 0x400014 1 read > 46841468 osd17 12.159b3dd2 12.12 [17,5,2]/17 [17,5,2]/17 e25593 10003307266.00000013 0x400014 1 read > 46841496 osd17 12.a08092d2 12.12 [17,5,2]/17 [17,5,2]/17 e25593 10003493c3f.00000050 0x400014 1 read > LINGER REQUESTS > BACKOFFS Restarting that OSD fixed it. Looking at the OSD log at the exact time of the hang (determined by other logs), I see this: > 2025-07-18T06:39:06.992+0900 fffeed21bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.162732124s, lat = 5s cid =12.3_head oid =#12:c0ccf68d:::1000336b41c.0000009d:13d# > 2025-07-18T06:39:06.992+0900 fffeed21bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.167396545s > 2025-07-18T06:39:06.993+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.168415070s, txc = 0xaaabe3f57000 > 2025-07-18T06:39:14.834+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.340788364s, lat = 5s cid =12.14_head oid =#12:28e0c837:::10003307273.00000024:13d# > 2025-07-18T06:39:14.835+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.342024803s > 2025-07-18T06:39:14.843+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.349779129s, txc = 0xaaac19c75b00 > 2025-07-18T06:40:37.552+0900 fffeeb9ebf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.124079704s, lat = 5s cid =12.c_head oid =#12:340862a9:::100033073f1.0000002c:13d# > 2025-07-18T06:40:37.560+0900 fffeeb9ebf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.132128239s > 2025-07-18T06:40:37.568+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.140548706s, txc = 0xaaabf25b8000 > 2025-07-18T06:40:59.228+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.677710056s, lat = 5s cid =12.2_head oid =#12:40b82ac1:::1000348d274.00000008:13d# > 2025-07-18T06:40:59.229+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.679266453s > 2025-07-18T06:40:59.232+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.682381630s, txc = 0xaaac34479500 > 2025-07-18T06:41:01.799+0900 fffeeb9ebf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.117489815s, lat = 5s cid =12.c_head oid =#12:340e7251:::10003307273.00000027:13d# > 2025-07-18T06:41:01.799+0900 fffeeb9ebf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.118617058s > 2025-07-18T06:41:01.804+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.123104095s, txc = 0xaaabbf99de00 > 2025-07-18T06:42:28.863+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.191506386s, lat = 5s cid =12.3_head oid =#12:c2349a5a:::1000336b418.000000f0:13d# > 2025-07-18T06:42:28.863+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.192711353s > 2025-07-18T06:42:28.864+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.193384171s, txc = 0xaaabd1e51e00 > 2025-07-18T06:42:34.152+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.287574291s, lat = 5s cid =12.3_head oid =#12:c2bf668a:::10003307273.0000007b:13d# > 2025-07-18T06:42:34.154+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.289901257s > 2025-07-18T06:42:34.156+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.291684628s, txc = 0xaaabf9f90c00 > 2025-07-18T06:42:40.801+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:42:40.801+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) [ repeats ... ] > 2025-07-18T06:44:15.954+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:15.954+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:44:27.591+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.561402321s, lat = 5s cid =12.0_head oid =#12:064b3406:::1000348bd03.0000004a:13d# > 2025-07-18T06:44:27.594+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.565083027s > 2025-07-18T06:44:27.595+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.565894604s, txc = 0xaaabe0e98c00 > 2025-07-18T06:44:29.442+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:29.442+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) [ repeats ... ] > 2025-07-18T06:44:32.362+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:32.362+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 1 'started' : 1 ] most affected pool [ 'cephfs2_data_hr3' : 1 ]) > 2025-07-18T06:44:33.314+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:33.314+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 1 'started' : 1 ] most affected pool [ 'cephfs2_data_hr3' : 1 ]) > 2025-07-18T06:44:34.328+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:34.328+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 1 'started' : 1 ] most affected pool [ 'cephfs2_data_hr3' : 1 ]) > 2025-07-18T06:44:38.361+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841376 14.25s0 14.cd26e465 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:38.361+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs2_data_hec5.4' : 2 ]) > 2025-07-18T06:44:39.369+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841376 14.25s0 14.cd26e465 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:39.369+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs2_data_hec5.4' : 2 ]) > 2025-07-18T06:44:40.356+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.6505232.0:46841260 14.19s0 14.2ab058d9 (undecoded) ondisk+read+known_if_redirected e25583) > 2025-07-18T06:44:40.356+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 5 slow requests (by type [ 'queued for pg' : 5 ] most affected pool [ 'cephfs2_data_hec5.4' : 5 ]) > 2025-07-18T06:44:41.324+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:41.324+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) [ repeats ... ] > 2025-07-18T06:45:14.197+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:14.197+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:15.825+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.245476246s, lat = 5s cid =12.b_head oid =#12:d12c2386:::1000336b41a.00000095:13d# > 2025-07-18T06:45:15.826+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.247073650s > 2025-07-18T06:45:15.831+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.252201080s, txc = 0xaaabd1e50c00 > 2025-07-18T06:45:19.037+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:19.037+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:20.056+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:20.056+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:21.010+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:21.010+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:21.963+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:21.963+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:22.973+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841376 14.25s0 14.cd26e465 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:22.973+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs2_data_hec5.4' : 2 ]) > 2025-07-18T06:45:23.958+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841376 14.25s0 14.cd26e465 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:23.958+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs2_data_hec5.4' : 2 ]) > 2025-07-18T06:45:32.861+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) [ repeats ... ] > 2025-07-18T06:49:20.553+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:49:21.547+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:49:21.547+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:49:22.596+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:49:22.596+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:49:48.204+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.027801514s, lat = 5s cid =12.1a_head oid =#12:5b980d7b:::100033073d8.0000002a:13d# > 2025-07-18T06:49:48.205+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.031510353s > 2025-07-18T06:49:48.214+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.040008068s, txc = 0xaaabe5c0cf00 Note that osd17 is on a different host (but some of the shards of the affected PGs are on the same host). After this 10-minute period, the logs are clean. So it kind of looks like one op (46841429) above was a slow op for about 10 minutes, but then it stopped being considered slow by the OSD, yet the kernel client never considered the op completed and it got stuck there? Other ops from that client are also mentioned, and this doesn't seem right to me. Op 46841429 is the first one to become "slow", but then other ops like 46841376 are reported as the "oldest" op, and then it goes back to 46841429. Unless this is about different phases for the same op, it doesn't make sense to me that a younger op could be reported as slow, then an older op, then back to a younger op. If the older op is older, it should always be reported as slow first, right? I checked all the other OSD logs and there's nothing interesting other than a couple other OSDs that also got slow ops at around the same time (but much shorter duration). Grafana stats show a snaptrim operation started at exactly that time and lasted 10 minutes. The CPU logs show that this did repro the CPU usage/page cache thrashing anomaly I previously mentioned on the list (high sys >> user CPU usage), but this time the thrashing only lasted 10 minutes, so I would say this is unlikely to be directly caused by that bug, though it could be that the high load triggering slow ops in general then tickled another bug causing the ops to get stuck. Happy to help debug this if it happens again if anyone has pointers (I think this might not be the first time, but it's the first time I found that thread and learned about this issue specifically, and the debug file to check the culprit OSD). - Hector

1 0

Test Cluster / Performance Degradation After Adding Private Network
by Anthony Fecarotta 19 Jul '25

19 Jul '25

Hello, I have a test cluster of some mini-PCs. This one, in particular, runs Proxmox, and has two Ceph RBDs (LXCs & VMs). The purpose of this test cluster was to test Docker Swarm. I wanted to get the feel for orchestration - our five-node production cluster is very simple and Kubernetes would be overkill. Each node boots off of NVMe, and each node has one OSD, PCIe Gen 4 M.2 NVMe. I understand this equipment is not optimal, but please keep in mind this is a test cluster. All things considered, it was running fine for two months, I even made some of our non-critical BETA programs available for internal use within our organization. Yesterday, I connected a 2.5GBe unmanaged switch, to the second 2.5GBe NIC of each node cluster, creating a Private/Cluster Network for Ceph. Since then, each node, VM, LXC, etc. are moving at a glacial pace. Just to give an example, a sudo apt update or just logging in via SSH can take sixty seconds. [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 172.16.1.0/24 fsid = 3c395d5c-7d46-4dc7-ad4b-8a6761f167b0 mon_allow_pool_delete = true mon_host = 192.168.128.156 192.168.128.150 192.168.128.158 ms_bind_ipv4 = true ms_bind_ipv6 = false osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 192.168.128.156/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [client.crash] keyring = /etc/pve/ceph/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mon.asusNuc1] public_addr = 192.168.128.150 [mon.chyna2gb] public_addr = 192.168.128.156 [mon.chyna4tb] public_addr = 192.168.128.158 # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class nvme device 1 osd.1 class nvme device 2 osd.2 class nvme device 3 osd.3 class nvme # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root # buckets host chyna2gb { id -3 # do not change unnecessarily id -4 class nvme # do not change unnecessarily # weight 1.86299 alg straw2 hash 0 # rjenkins1 item osd.0 weight 1.86299 } host chyna4tb { id -5 # do not change unnecessarily id -6 class nvme # do not change unnecessarily # weight 3.63869 alg straw2 hash 0 # rjenkins1 item osd.1 weight 3.63869 } host nuc { id -7 # do not change unnecessarily id -8 class nvme # do not change unnecessarily # weight 0.90970 alg straw2 hash 0 # rjenkins1 item osd.2 weight 0.90970 } host asusNuc1 { id -9 # do not change unnecessarily id -10 class nvme # do not change unnecessarily # weight 3.63869 alg straw2 hash 0 # rjenkins1 item osd.3 weight 3.63869 } root default { id -1 # do not change unnecessarily id -2 class nvme # do not change unnecessarily # weight 10.05006 alg straw2 hash 0 # rjenkins1 item chyna2gb weight 1.86299 item chyna4tb weight 3.63869 item nuc weight 0.90970 item asusNuc1 weight 3.63869 } # rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } # end crush map root@asusNuc1:~# ceph osd perf osd commit_latency(ms) apply_latency(ms) 0 6 6 3 15 15 2 23 23 1 4 4 To be clear, the "dumb switch" is isolated and not connected to the rest of the network. Regards, [image] Anthony Fecarotta Founder & President [image] anthony(a)linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>

2 1

Reef: cephadm tries to apply specs frequently
by Eugen Block 18 Jul '25

18 Jul '25

Good morning, I noticed something strange on a 18.2.7 cluster, running on Ubuntu 22.04, deployed by cephadm. There are 10 hosts in total, 5 of them are all-flash and those aren't affected. The other 5 hosts are hdd-only, and only 4 of those are affected: The /var/log/ceph/{FSID}/ceph-volume.log is flooded with attempts to apply the osd spec: [2025-07-17 05:40:01,994][ceph_volume.main][INFO ] Running command: ceph-volume lvm batch --no-auto /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw --yes --no-systemd [2025-07-17 05:42:00,216][ceph_volume.main][INFO ] Running command: ceph-volume lvm batch --no-auto /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw --yes --no-systemd [2025-07-17 05:43:50,521][ceph_volume.main][INFO ] Running command: ceph-volume lvm batch --no-auto /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw --yes --no-systemd So that log file alone grows to more than 1 GB per day, as a consequence other logs like syslog grow as well. But for some reason, storage08 is skipped, the mgr reports: [cephadm DEBUG root] skipping apply of storage08 on DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd This is the current hdd-only spec: # ceph orch ls osd --export service_type: osd service_id: hdd-only service_name: osd.hdd-only placement: hosts: - storage06 - storage07 - storage08 - storage09 - storage10 spec: data_devices: rotational: 1 size: '1T:' filter_logic: AND objectstore: bluestore I verified that all disks are deployed as OSDs, so there's no orphan lying around or anything. I failed the mgr (of course), rebooted one host because storage08 was recently rebooted as well. I don't know for how long this has been going on, unfortunately. So I started to look at the code [0], [1], it seems like the mgr cache is not properly updated: if not self.mgr.cache.osdspec_needs_apply(host, drive_group): self.mgr.log.debug("skipping apply of %s on %s (no change)" % ( host, drive_group)) So I looked at all these values: def osdspec_needs_apply(self, host: str, spec: ServiceSpec) -> bool: if ( host not in self.devices or host not in self.last_device_change or host not in self.last_device_update or host not in self.osdspec_last_applied or spec.service_name() not in self.osdspec_last_applied[host] ): but all keys are populated with similar values as on storage08, for example: root@storage01:~# ceph config-key get mgr/cephadm/host.storage10 | jq -r '.last_device_change,.last_device_update,.osdspec_last_applied' 2025-02-12T16:27:21.979015Z 2025-07-17T05:35:18.852618Z { "osd.hdd-only": "2025-07-17T06:03:38.860971Z" } root@storage01:~# ceph config-key get mgr/cephadm/host.storage08 | jq -r '.last_device_change,.last_device_update,.osdspec_last_applied' 2025-03-11T08:23:02.851969Z 2025-07-17T05:43:47.521682Z { "osd.hdd-only": "2025-03-11T08:23:21.494004Z" } Can anyone make sense of it? I'd appreciate any pointers! Thanks! Eugen [0] https://github.com/ceph/ceph/blob/v18.2.7/src/pybind/mgr/cephadm/services/o… [1] https://github.com/ceph/ceph/blob/v18.2.7/src/pybind/mgr/cephadm/inventory.…

1 1

Hardware recommendation
by GLE, Vivien 18 Jul '25

18 Jul '25

Hi, My mission is to make a ceph cluster production ready. We already have a mock-up of 6 nodes deployed via cephadm to test ceph's features We want to install a ceph cluster to get off our dependencies to SAN and to store around 400 TB of data. We'll use mostly block storage for VMs/k8 and a bit of CephFS and Object storage Are those specs good for our purpose ? - 3 MGR/MON/RGW 128 GB RAM 32 cores (is it recommended to put RGW daemon in the same node as MGR/MON ?) - 2 MDS 16 GB RAM 8 cores - * OSD 8 GB RAM 8 cores (with high IOPS per core) We are pretty new to ceph so any advice would be appreciate ! Thanks ! Vivien

2 2

Re: squid 19.2.2 - discrepancies between GUI and CLI
by Steven Vacaroaia 17 Jul '25

17 Jul '25

Thanks for the suggestion Unfortunately "ceph mgr fail" did not solve the issue Is there a better way to "fail" ? Steven On Thu, 17 Jul 2025 at 14:04, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Try failing the mgr > > > On Jul 17, 2025, at 1:48 PM, Steven Vacaroaia <stef97(a)gmail.com> wrote: > > > > Hi, > > > > The GUI is reporting PGs still being backfilled > > but CLI is not > > > > Who is right ;-) and how can I get them to "agree" ? > > > > <image.png> > > > > <image.png> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

squid 19.2.2 - discrepancies between GUI and CLI
by Steven Vacaroaia 17 Jul '25

17 Jul '25

Hi, The GUI is reporting PGs still being backfilled but CLI is not Who is right ;-) and how can I get them to "agree" ? [image: image.png] [image: image.png]

1 0