[ceph-users] RocksDB degradation / manual compaction vs. snaptrim operations choking Ceph to a halt

Hello ceph-users, after an upgrade from Ceph Nautilus to Octopus we ran into extreme performance issues leading to an unusable cluster when doing a larger snapshot delete and the cluster doing snaptrims, see i.e. https://tracker.ceph.com/issues/50511#note-13. Since this was not an issue prior to the upgrade, maybe the conversion of the OSD to OMAP caused this degradation of the RocksDB data structures, maybe not. (We were running bluefs_buffered_io=true, so that was NOT the issue here). But I've noticed there are a few reports of such issues which boil down to RocksDB being in a somewhat degraded state and running a simple compact fixed those issues, see:  * https://tracker.ceph.com/issues/50511  * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XSEBOIT43TG…  * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/BTWAQIEXBBE…  * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Z4ADQFTGC5H…  * Maybe also: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/N74C4U2POOS… I know improvements in this regard are actively worked on for pg removal, i.e.  * https://tracker.ceph.com/issues/47174  ** https://github.com/ceph/ceph/pull/37314  ** https://github.com/ceph/ceph/pull/37496 but am wondering if this will help with snaptrims as well? In any case I was just wondering if any of you also experienced this condition with RocksDB and am wondering what you do to monitor or to actively mitigate this prior to having flapping OSDs and queuing up (snaptrim) operations? With Ceph Pacific it's possible to enable offline compaction on every start of an OSD (osd_compact_on_start), but is this really sufficient then? Regards Christian