RocksDB degradation / manual compaction vs. snaptrim operations choking Ceph to a halt
Hello ceph-users, after an upgrade from Ceph Nautilus to Octopus we ran into extreme performance issues leading to an unusable cluster when doing a larger snapshot delete and the cluster doing snaptrims, see i.e. https://tracker.ceph.com/issues/50511#note-13. Since this was not an issue prior to the upgrade, maybe the conversion of the OSD to OMAP caused this degradation of the RocksDB data structures, maybe not. (We were running bluefs_buffered_io=true, so that was NOT the issue here). But I've noticed there are a few reports of such issues which boil down to RocksDB being in a somewhat degraded state and running a simple compact fixed those issues, see: * https://tracker.ceph.com/issues/50511 * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XSEBOIT43TGI... * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/BTWAQIEXBBEG... * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Z4ADQFTGC5HM... * Maybe also: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/N74C4U2POOSH... I know improvements in this regard are actively worked on for pg removal, i.e. * https://tracker.ceph.com/issues/47174 ** https://github.com/ceph/ceph/pull/37314 ** https://github.com/ceph/ceph/pull/37496 but am wondering if this will help with snaptrims as well? In any case I was just wondering if any of you also experienced this condition with RocksDB and am wondering what you do to monitor or to actively mitigate this prior to having flapping OSDs and queuing up (snaptrim) operations? With Ceph Pacific it's possible to enable offline compaction on every start of an OSD (osd_compact_on_start), but is this really sufficient then? Regards Christian
Hi Christian, On 7/7/2021 11:31 AM, Christian Rohmann wrote:
Hello ceph-users,
after an upgrade from Ceph Nautilus to Octopus we ran into extreme performance issues leading to an unusable cluster when doing a larger snapshot delete and the cluster doing snaptrims, see i.e. https://tracker.ceph.com/issues/50511#note-13. Since this was not an issue prior to the upgrade, maybe the conversion of the OSD to OMAP caused this degradation of the RocksDB data structures, maybe not. (We were running bluefs_buffered_io=true, so that was NOT the issue here).
It's hard to say what exactly caused the issue this time. Indeed OMAP conversion could have some impact since it had performed bulk removal along the upgrade process - so DB could gain critical mass to start lagging. But I presume this is a one-time effect - it should vaporize after DB compaction. Which doesn't mean that snaptrims or any other bulk removals are absolutely safe since then though.
But I've noticed there are a few reports of such issues which boil down to RocksDB being in a somewhat degraded state and running a simple compact fixed those issues, see:
* https://tracker.ceph.com/issues/50511 * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XSEBOIT43TGI... * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/BTWAQIEXBBEG... * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Z4ADQFTGC5HM... * Maybe also: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/N74C4U2POOSH...
I know improvements in this regard are actively worked on for pg removal, i.e.
* https://tracker.ceph.com/issues/47174 ** https://github.com/ceph/ceph/pull/37314 ** https://github.com/ceph/ceph/pull/37496
but am wondering if this will help with snaptrims as well?
I'm aware of snaptrim causing bulk removal and hence being a potential issue to DB performance. Unfortunately I haven't found any good enough solution after a brief research. Hence this part of the problem is pending solution for now - the above PRs wouldn't fix it but they might make a bit less likely to happen - since other bulk removals would be handled differently and even trigger partial DB compaction on their own..
In any case I was just wondering if any of you also experienced this condition with RocksDB and am wondering what you do to monitor or to actively mitigate this prior to having flapping OSDs and queuing up (snaptrim) operations?
I would suggest to perform full DB compaction for the first time and monitor whether the issue reappears again - it could be OMAP upgrade which brought additional disturbance which finally broke DB. As the cluster was able to do that before good chances that it still could deal with the overhead when snaptrim removal is the only "bad guy".
With Ceph Pacific it's possible to enable offline compaction on every start of an OSD (osd_compact_on_start), but is this really sufficient then?
Hopefully this should be enough given you provide restarts often enough, e.g. once a day. But surely there isn't a 100% warranty ...
Regards
Christian
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Igor, On 07/07/2021 14:59, Igor Fedotov wrote:
after an upgrade from Ceph Nautilus to Octopus we ran into extreme performance issues leading to an unusable cluster when doing a larger snapshot delete and the cluster doing snaptrims, see i.e. https://tracker.ceph.com/issues/50511#note-13. Since this was not an issue prior to the upgrade, maybe the conversion of the OSD to OMAP caused this degradation of the RocksDB data structures, maybe not. (We were running bluefs_buffered_io=true, so that was NOT the issue here).
It's hard to say what exactly caused the issue this time. Indeed OMAP conversion could have some impact since it had performed bulk removal along the upgrade process - so DB could gain critical mass to start lagging.
But I presume this is a one-time effect - it should vaporize after DB compaction. Which doesn't mean that snaptrims or any other bulk removals are absolutely safe since then though.
Thank you very much for your quick and extensive reply! If OMAP conversion could have this effect, maybe it's sensible to trigger either an an immediate online compaction to the end of the conversion or at least add this to the upgrade notes. I suppose with the EoL of Nautilus more and more clusters will now make the jump to the Octopus release and convert their OSDs to OMAP in the process. Even if not all clusters RocksDBs would go over the edge, in any case running a compaction should not hurt right? Thanks again, Christian
Hi Christian, yeah, came to the same idea to trigger compaction on upgrade completetion yesterday. See https://github.com/ceph/ceph/pull/42218 Thanks, Igor On 7/8/2021 10:21 AM, Christian Rohmann wrote:
Hey Igor,
On 07/07/2021 14:59, Igor Fedotov wrote:
after an upgrade from Ceph Nautilus to Octopus we ran into extreme performance issues leading to an unusable cluster when doing a larger snapshot delete and the cluster doing snaptrims, see i.e. https://tracker.ceph.com/issues/50511#note-13. Since this was not an issue prior to the upgrade, maybe the conversion of the OSD to OMAP caused this degradation of the RocksDB data structures, maybe not. (We were running bluefs_buffered_io=true, so that was NOT the issue here).
It's hard to say what exactly caused the issue this time. Indeed OMAP conversion could have some impact since it had performed bulk removal along the upgrade process - so DB could gain critical mass to start lagging.
But I presume this is a one-time effect - it should vaporize after DB compaction. Which doesn't mean that snaptrims or any other bulk removals are absolutely safe since then though.
Thank you very much for your quick and extensive reply!
If OMAP conversion could have this effect, maybe it's sensible to trigger either an an immediate online compaction to the end of the conversion or at least add this to the upgrade notes. I suppose with the EoL of Nautilus more and more clusters will now make the jump to the Octopus release and convert their OSDs to OMAP in the process. Even if not all clusters RocksDBs would go over the edge, in any case running a compaction should not hurt right?
Thanks again,
Christian
If your PG is not served millions of objects in on PG - is not your problem... k Sent from my iPhone
On 7 Jul 2021, at 11:32, Christian Rohmann <christian.rohmann@inovex.de> wrote:
I know improvements in this regard are actively worked on for pg removal, i.e.
* https://tracker.ceph.com/issues/47174 ** https://github.com/ceph/ceph/pull/37314 ** https://github.com/ceph/ceph/pull/37496
but am wondering if this will help with snaptrims as well?
participants (3)
-
Christian Rohmann
-
Igor Fedotov
-
Konstantin Shalygin