[ceph-users] Slow osd ops on large arm cluster

8 Jul 2024

      Hello,

we are having issues with slow ops on our large ARM hpc ceph cluster.

Cluster runs on 18.2.0 and ubutnu 20.04
MONs, MGRs and MDSs had to be moved to intel servers because of poor 
single core performance on our arm servers.
Our main cephfs data pool is on 54 serwers in 9 racks with 1458 HDDs in 
total. (OSDs without block.db on ssd)
Cephfs data pool is configured as erasure coded pool with k=6 m=2 and 
rack level replication. Pool has about 16k PGs with average pg per osd 
at ~90.

We have had good experience with EC cephfs on 3,5 times smaller intel 
ceph cluster. But this arm deployment is becoming problematic. We 
started experiencing issues since one of the users started to generate 
sequential RW traffic at at about 5GiB/s. Single OSD with slow ops was 
enough to create laggy PG and crash application generating this traffic.
We've even had issue where osd with slow ops was lagged for 6 hours and 
required manual restart.

Now we are experiencing slow ops even at much lower read only traffic 
~400MiB/s

Here is an example of slow ops on OSD:
{
     "ops": [
         {
             "description": "osd_op(client.255949991.0:92728602 4.d22s0 
4:44b3390a:::1000b640ddc.0000039b:head [read 3633152~8192] snapc 0=[] 
ondisk+read+known_if_redirected e1117246)",
             "initiated_at": "2024-07-08T10:19:58.469537+0000",
             "age": 507.242936848,
             "duration": 507.24298854800003,
             "type_data": {
                 "flag_point": "started",
                 "client_info": {
                     "client": "client.255949991",
                     "client_addr": "x.x.x.x:0/887459214",
                     "tid": 92728602
                 },
                 "events": [
                     {
                         "event": "initiated",
                         "time": "2024-07-08T10:19:58.469537+0000",
                         "duration": 0
                     },
                     {
                         "event": "throttled",
                         "time": "2024-07-08T10:19:58.469537+0000",
                         "duration": 0
                     },
                     {
                         "event": "header_read",
                         "time": "2024-07-08T10:19:58.469535+0000",
                         "duration": 4294967295.9999981
                     },
                     {
                         "event": "all_read",
                         "time": "2024-07-08T10:19:58.469571+0000",
                         "duration": 3.5859999999999999e-05
                     },
                     {
                         "event": "dispatched",
                         "time": "2024-07-08T10:19:58.469573+0000",
                         "duration": 2.08e-06
                     },
                     {
                         "event": "queued_for_pg",
                         "time": "2024-07-08T10:19:58.469586+0000",
                         "duration": 1.2721000000000001e-05
                     },
                     {
                         "event": "reached_pg",
                         "time": "2024-07-08T10:19:58.485132+0000",
                         "duration": 0.015546048999999999
                     },
                     {
                         "event": "started",
                         "time": "2024-07-08T10:19:58.485147+0000",
                         "duration": 1.5160000000000001e-05
                     }
                 ]
             }
         },
HDD with this OSD is not busy. Arm cores on these servers are slow but 
no process reaches full 100% core usage.

I think we may have the same issue as one described here: 
https://www.mail-archive.com/ceph-users@ceph.io/msg13273.html

I've tried to reduce osd_pool_default_read_lease_ratio form 0.8 to 0.2
I've tried to reduce osd_heartbeat_grace from 20 to 10.
It should lower read_lease_interval from 16 to 2 but it didn't help. 
Still see a lot of slow ops.

Could you give me tips what I could tune to fix this issue?

Could this be an issue with large number of EC PGs on large cluster with 
weak CPUs?

Best regards
Adam Prycki