PSA: sqlite3 databases now available for ceph-mgr modules
Introduced by [1] for Quincy release. This builds on work in [2] to add RADOS-backed sqlite3 support to Ceph (available in Pacific). The MgrModule API for accessing your module's database is introduced in [3]. An example of a module ("devicehealth") using the API can be seen in [4]. Please let me know if you have any questions or feedback. [1] https://github.com/ceph/ceph/pull/40740 [2] https://github.com/ceph/ceph/pull/28822 [3] https://github.com/ceph/ceph/commit/e3d771702da3bb858064b67eb6c710a659bfb08d [4] https://github.com/ceph/ceph/commit/abd35d47696c208990355395d48c1c1e261de95c -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Wed, Jun 16, 2021 at 10:23 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
Introduced by [1] for Quincy release. This builds on work in [2] to add RADOS-backed sqlite3 support to Ceph (available in Pacific).
The MgrModule API for accessing your module's database is introduced in [3]. An example of a module ("devicehealth") using the API can be seen in [4].
Please let me know if you have any questions or feedback.
Hi Patrick, my concern is that, without carefully planning on the segmentation of the pool for storing the healthy data and the pools being monitored, we could interfere with the system being monitored by mutating its status. for instance, if a cluster is experiencing large-scale slow ops, and pumping lots of warning messages and/or structured performance related metrics, some mgr module might want to collect this information from the health monitoring subsystem, and persist them into the sqlite3 database. but it is in turn backed by the same cluster. without carefully planning, the objects stored in .mgr pool could be mapped to the same set of OSDs and monitors which are suffering from the performance issue. in the worst case, this could in turn even worsen the situation. but to allocate dedicated OSDs and create a CRUSH map picking them just for the .mgr pool might be difficult or overkill from the maintainability point of view. we actually had the same issue when adding the cluster log back to OSD for recording the slow requests. the large amount of clog puts more burden on the shoulder of the monitors. if the slow requests is caused by monitor, these clogs actually in turn slow down the monitors further. shall we switch to a (local) backup sqlite backend if we identify a performance issue, and restore / backfill the records once the issue is resolved? cheers,
[1] https://github.com/ceph/ceph/pull/40740 [2] https://github.com/ceph/ceph/pull/28822 [3] https://github.com/ceph/ceph/commit/e3d771702da3bb858064b67eb6c710a659bfb08d [4] https://github.com/ceph/ceph/commit/abd35d47696c208990355395d48c1c1e261de95c
-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Regards Kefu Chai
Hi Kefu, On Thu, Jun 17, 2021 at 9:24 PM kefu chai <tchaikov@gmail.com> wrote:
On Wed, Jun 16, 2021 at 10:23 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
Introduced by [1] for Quincy release. This builds on work in [2] to add RADOS-backed sqlite3 support to Ceph (available in Pacific).
The MgrModule API for accessing your module's database is introduced in [3]. An example of a module ("devicehealth") using the API can be seen in [4].
Please let me know if you have any questions or feedback.
Hi Patrick,
my concern is that, without carefully planning on the segmentation of the pool for storing the healthy data and the pools being monitored, we could interfere with the system being monitored by mutating its status.
for instance, if a cluster is experiencing large-scale slow ops, and pumping lots of warning messages and/or structured performance related metrics, some mgr module might want to collect this information from the health monitoring subsystem, and persist them into the sqlite3 database. but it is in turn backed by the same cluster. without carefully planning, the objects stored in .mgr pool could be mapped to the same set of OSDs and monitors which are suffering from the performance issue. in the worst case, this could in turn even worsen the situation. but to allocate dedicated OSDs and create a CRUSH map picking them just for the .mgr pool might be difficult or overkill from the maintainability point of view.
we actually had the same issue when adding the cluster log back to OSD for recording the slow requests. the large amount of clog puts more burden on the shoulder of the monitors. if the slow requests is caused by monitor, these clogs actually in turn slow down the monitors further.
shall we switch to a (local) backup sqlite backend if we identify a performance issue, and restore / backfill the records once the issue is resolved?
Thanks for bringing this up. I think it would be reasonable to decide this depending on what the mgr module is doing. For example, I think devicehealth and snap_schedule are innocuous enough that we don't need to give special consideration for the system potentially being under load. Also these modules' mutations of the databases do not depend on the cluster state, healthy or degraded. OTOH, a module that is collecting large streams of data into the database might first ingest that data into a local in-memory database and only backup [1] that in-memory database to RADOS when the cluster is healthy. If the database is very large then a backup would not be desirable as the in-memory database would be too large. In that case I would suggest streaming batch updates in large transactions. What do you think? [1] https://www.sqlite.org/backup.html -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Not that anyone asked for my zwei pfennig ...
but it is in turn backed by the same cluster. without carefully planning, the objects stored in .mgr pool could be mapped to the same set of OSDs and monitors which are suffering from the performance issue.
In other words, a circular dependency of sorts?
in the worst case, this could in turn even worsen the situation. but to allocate dedicated OSDs and create a CRUSH map picking them just for the .mgr pool might be difficult or overkill from the maintainability point of view.
Certainly there are wrinkles. The first time I interacted with a devicehealth pool was because it was causing HEALTH_ERR. I found a pool with 1 PG, with an empty acting set. Couldn’t figure out how the heck it got that way, so I just removed the pool. Dedicated OSDs could be problematic if they decrease cluster capacity and uniformity by co-opting drive bays that otherwise would hold user data. With LVM, maybe a small slice of each drive? But then how would we size that slice? Would that complicate operations for operators for whom one drive == one OSDs is deeply ingrained? Beyond maintainability and zero-sum drive bays, though, is media suitability. With an HDD cluster, would .mgr pragmatically need to be on faster storage, a la RGW index? If so that feeds into the above drive bay quandary. If the only flash available in the systems is something like Optane, is there enough capacity? I’m not super familiar with sqlite, but I wonder if the access pattern would be problematic from a drive durability standpoint too.
OTOH, a module that is collecting large streams of data into the database might first ingest that data into a local in-memory database and only backup [1] that in-memory database to RADOS when the cluster is healthy. If the database is very large then a backup would not be desirable as the in-memory database would be too large. In that case I would suggest streaming batch updates in large transactions.
For at least some of these purposes, might it be feasible to just use memstore and memstore alone? Staging to persistent storage seems fraught with corner cases and atomicity concerns. — aad
On Sun, Jun 20, 2021 at 6:06 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Not that anyone asked for my zwei pfennig ...
but it is in turn backed by the same cluster. without carefully planning, the objects stored in .mgr pool could be mapped to the same set of OSDs and monitors which are suffering from the performance issue.
In other words, a circular dependency of sorts?
Negative feedback.
in the worst case, this could in turn even worsen the situation. but to allocate dedicated OSDs and create a CRUSH map picking them just for the .mgr pool might be difficult or overkill from the maintainability point of view.
Certainly there are wrinkles.
The first time I interacted with a devicehealth pool was because it was causing HEALTH_ERR. I found a pool with 1 PG, with an empty acting set. Couldn’t figure out how the heck it got that way, so I just removed the pool.
Dedicated OSDs could be problematic if they decrease cluster capacity and uniformity by co-opting drive bays that otherwise would hold user data. With LVM, maybe a small slice of each drive? But then how would we size that slice? Would that complicate operations for operators for whom one drive == one OSDs is deeply ingrained?
Beyond maintainability and zero-sum drive bays, though, is media suitability. With an HDD cluster, would .mgr pragmatically need to be on faster storage, a la RGW index? If so that feeds into the above drive bay quandary. If the only flash available in the systems is something like Optane, is there enough capacity? I’m not super familiar with sqlite, but I wonder if the access pattern would be problematic from a drive durability standpoint too.
I would not go so far as to suggest that the .mgr pool be given dedicated OSDs. libcephsqlite does repeatedly overwrite the same objects but I don't believe that would cause undue wear on drives. There is no use of omap.
OTOH, a module that is collecting large streams of data into the database might first ingest that data into a local in-memory database and only backup [1] that in-memory database to RADOS when the cluster is healthy. If the database is very large then a backup would not be desirable as the in-memory database would be too large. In that case I would suggest streaming batch updates in large transactions.
For at least some of these purposes, might it be feasible to just use memstore and memstore alone? Staging to persistent storage seems fraught with corner cases and atomicity concerns.
What corner cases / atomicity concerns? Transactions are ACID in sqlite even when backed by RADOS. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Fri, Jun 18, 2021 at 11:53 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
Hi Kefu,
On Thu, Jun 17, 2021 at 9:24 PM kefu chai <tchaikov@gmail.com> wrote:
On Wed, Jun 16, 2021 at 10:23 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
Introduced by [1] for Quincy release. This builds on work in [2] to add RADOS-backed sqlite3 support to Ceph (available in Pacific).
The MgrModule API for accessing your module's database is introduced in [3]. An example of a module ("devicehealth") using the API can be seen in [4].
Please let me know if you have any questions or feedback.
Hi Patrick,
my concern is that, without carefully planning on the segmentation of the pool for storing the healthy data and the pools being monitored, we could interfere with the system being monitored by mutating its status.
for instance, if a cluster is experiencing large-scale slow ops, and pumping lots of warning messages and/or structured performance related metrics, some mgr module might want to collect this information from the health monitoring subsystem, and persist them into the sqlite3 database. but it is in turn backed by the same cluster. without carefully planning, the objects stored in .mgr pool could be mapped to the same set of OSDs and monitors which are suffering from the performance issue. in the worst case, this could in turn even worsen the situation. but to allocate dedicated OSDs and create a CRUSH map picking them just for the .mgr pool might be difficult or overkill from the maintainability point of view.
we actually had the same issue when adding the cluster log back to OSD for recording the slow requests. the large amount of clog puts more burden on the shoulder of the monitors. if the slow requests is caused by monitor, these clogs actually in turn slow down the monitors further.
shall we switch to a (local) backup sqlite backend if we identify a performance issue, and restore / backfill the records once the issue is resolved?
Thanks for bringing this up. I think it would be reasonable to decide this depending on what the mgr module is doing. For example, I think devicehealth and snap_schedule are innocuous enough that we don't need to give special consideration for the system potentially being under load. Also these modules' mutations of the databases do not depend on the cluster state, healthy or degraded. OTOH, a module that is
okay, that's a relief. just for future developers, we should not use the sqlite backend for storing the alerts created when the whole cluster is not healthy.
collecting large streams of data into the database might first ingest that data into a local in-memory database and only backup [1] that in-memory database to RADOS when the cluster is healthy. If the database is very large then a backup would not be desirable as the in-memory database would be too large. In that case I would suggest streaming batch updates in large transactions.
thanks. glad that we have a plan B in that case.
What do you think?
[1] https://www.sqlite.org/backup.html
-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
-- Regards Kefu Chai
participants (3)
-
Anthony D'Atri
-
kefu chai
-
Patrick Donnelly