Hey Frédéric,
Can I ask what symptoms made you interested in tombstones?
Mostly poor index performance due to slow rocksdb iterators (the cause being excessive tombstone accumulation).
Do you think this phenomenon could be related to tombstones? And that enabling rocksdb_cf_compact_on_deletion would help to avoid this situation?
I don't have experience with rocksdb spill, unfortunately. If compaction is helping with spill, then more frequent compaction sounds like it could help, though I don't know whether this setting in particular will help.
Also, could ttl_option be an alternative to rocksdb_cf_compact_on_deletion that wasn't backported to v16.2.10 (RHCS 5.3z7)?
Yes, we've successfully used ttl to mitigate tombstone problems before the introduction of rocksdb_cf_compact_on_deletion, and it may be more relevant to your situation. You can add ttl=<seconds> to your rocksdb options and restart OSDs to give it a try. Josh