Hi everyone, I'm managing a Ceph Quincy 17.2.5 cluster, waiting to upgrade it to version 17.2.7, composed and configured as follows: - 16 identical nodes 256 GB RAM, 32 CPU Cores (64 threads), 12 x rotary HDD (BLOCK) + 4 x Sata SSD (RocksDB/WAL) - Erasure Code 11+4 (Jerasure) - 10 x S3 RGW on dedicated nodes (5 physical nodes) - 3 x full SSD dedicated nodes for replicated S3 pools - 2 x 10 Gbit Public network (LACP) + 2 x 10 Gbit cluster network (LACP) - On all nodes: Ubuntu 20.04.4 LTS Operating System updated - Ceph deployed on containers on Docker CE (docker-ce 5:20.10.17~3-0~ubuntu-focal). All pools, except the EC data pool, are configured with replication 3 and stored in dedicated SSD devices on 3 dedicated nodes to guarantee the necessary performance. We encountered a constant and random problem relating to the availability of access to bucket data and many slow_ops relating to rotary OSDs (data-pools) not caused by saturation of physical devices nor by the availability of CPU/RAM on all nodes. It sometimes happens that slow_ops caused by some requests to some PCs in a random way are reported. The cluster is currently in the recovery/rebalance phase for the reconstruction of 3 HDDs that we had to recreate from scratch (all 3 HDDs are physically on the same node). By doing some analysis of the events we verified the following from the status of some OSDs impacted by slow_ops: 20/05/2024 10:19 • "description": "osd_op(client.186021790.0:57620 29.258s0 29:1a5928ea:::31497ca8-e7d6-4e53-b150-91f9ac02ac67.246100.6329_storage%2ffirstMemories%2f1010543%2f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e481205)", "initiated_at": "2024-05-16T15:03:06.015956+0000", "age": 963.83795990199997, "duration": 963.83819621700002, "type_data": { "flag_point": "delayed", "client_info": { "client": "client.186021790", "client_addr": "10.151.11.11:0/3913909849", "tid": 57620 }, "events": [ { "event": "initiated", "time": "2024-05-16T15:03:06.015956+0000", "duration": 0 }, { "event": "throttled", "time": "2024-05-16T15:03:06.015956+0000", "duration": 0 }, { "event": "header_read", "time": "2024-05-16T15:03:06.015954+0000", "duration": 4294967295.9999986 }, { "event": "all_read", "time": "2024-05-16T15:03:06.015961+0000", "duration": 7.2300000000000002e-06 }, { "event": "dispatched", "time": "2024-05-16T15:03:06.015962+0000", "duration": 1.063e-06 }, { "event": "queued_for_pg", "time": "2024-05-16T15:03:06.015966+0000", "duration": 3.332e-06 }, { "event": "reached_pg", "time": "2024-05-16T15:03:06.015992+0000", "duration": 2.6078e-05 }, { "event": "waiting for readable", "time": "2024-05-16T15:03:06.016002+0000", "duration": 1.0348e-05 } ] } } ], "num_ops": 6 } ########### { "event": "reached_pg", "time": "2024-05-16T12:43:11.694220+0000", "duration": 480.97258642200001 }, In essence the operations remain suspended in this condition: "event": "waiting for readable" trying to access some PGs. We intend to update the version as soon as the recovery/rebalance is completed. Does anyone have any idea what checks I could do to analyze the problem more thoroughly? I can't define whether the problem could be the use of the EC or whether the data written in some buckets is in a "non-standard" condition, which causes the access to wait for some reason. Thank you all for your kindness. Greetings Andrea Martra -- -- Andrea Martra +39 393 9048451