Re: PSA: sqlite3 databases now available for ceph-mgr modules

On Wed, Jun 16, 2021 at 10:23 PM Patrick Donnelly <pdonnell(a)redhat.com> wrote: Hi Patrick, my concern is that, without carefully planning on the segmentation of the pool for storing the healthy data and the pools being monitored, we could interfere with the system being monitored by mutating its status. for instance, if a cluster is experiencing large-scale slow ops, and pumping lots of warning messages and/or structured performance related metrics, some mgr module might want to collect this information from the health monitoring subsystem, and persist them into the sqlite3 database. but it is in turn backed by the same cluster. without carefully planning, the objects stored in .mgr pool could be mapped to the same set of OSDs and monitors which are suffering from the performance issue. in the worst case, this could in turn even worsen the situation. but to allocate dedicated OSDs and create a CRUSH map picking them just for the .mgr pool might be difficult or overkill from the maintainability point of view. we actually had the same issue when adding the cluster log back to OSD for recording the slow requests. the large amount of clog puts more burden on the shoulder of the monitors. if the slow requests is caused by monitor, these clogs actually in turn slow down the monitors further. shall we switch to a (local) backup sqlite backend if we identify a performance issue, and restore / backfill the records once the issue is resolved? cheers, -- Regards Kefu Chai