Hi Mark, thank you for prompt answer.
The fact that changing the pg_num for the index pool drops the latency
back down might be a clue. Do you have a lot of deletes happening on this cluster? If you have a lot of deletes and long pauses between writes, you could be accumulating tombstones that you have to keep iterating over during bucket listing.
What you describe looks very close to our case of periodic creation of checkpoints. Now it sounds like it can be our issue. Those get cleaned up during
compaction. If there are no writes, you might not be compacting the tombstones away enough. Just a theory, but when you rearrange the PG counts, Ceph does a bunch of writes to move the data around, triggering compaction, and deleting the tombstones.
In v17.2.7 we enabled a feature that automatically performs a compaction if too many tombstones are present during iteration in RocksDB. It might be worth upgrading to see if it helps (you might have to try tweaking the settings if the defaults aren't helping enough). The PR is here:
https://github.com/ceph/ceph/pull/50893
Thank you very much for this idea! We'll upgrade cluster to v17.2.7 and will check if it helped. If not - we'll try to tune options you are referring to. Anyway I'll update the thread with result.
Thank you once again for well-explained suggestion, Mark! -- Thank you, Roman