Hi everyone, I’m reaching out for some guidance on troubleshooting periodic latency spikes we’re seeing in our Ceph cluster. At times simple file operations are occasionally taking several seconds to complete and this is impacting our users. We suspect the issue lies with our MDS load balancing, specifically an imbalance where our rank 0 MDS consistently has ~3-4 billion open caps (which seems to be about where its RAM maxes out), while the rank 1 and rank 2 sit around 3-7 million. Cluster Details - Ceph Version: 18.2.4 - OS: QuantaStor 6.6.9 - MDS Configuration: 3 active, 5 standby. - Storage Pools: - Data: 16 node HDD pool (8k+3m erasure coding). - Metadata: NVMe pool (triple replication). - We have ~150 nodes in an HPC cluster that use this storage in som We’ve tried tuning some ceph config parameters to try and make the MDS balance better, but this didn’t seem to help much: - mds mds_recall_max_caps 50000 - mds mds_max_caps_per_client 100000 - mds mds_bal_split_size 5000 - mds mds_bal_interval 5 Additionally, we’ve also attempted pinning our highest-load user to the rank 2 MDS. While this did successfully increase utilization on that rank, we are still seeing significant latency spikes for small file read/write operations across the board. If it helps, this user has a volume of 505TB, we have ten other volumes in the 10-100TB range, and then 175 other volumes under 10TB in usage. Is our diagnosis correct that the MDS is what’s causing the poor latency with file operations? If so, is it possible to have our MDS services distribute load more evenly? And as another question, does the size of a volume in ceph matter, meaning should we limit volume sizes to under 100TB for example? Thanks in advance for any guidance in diagnosing and resolving this issue. Best regards, Greg -- Gregory Ballantine System Administrator for Research and Enterprise Computing UMBC - DoIT