Hi All,
We are looking for inputs on a new feature to be implemented to move clog
messages storage from monstore db, refer trello card [1] for more details
around this topic.
Currently, every clog message goes to monstore db as well as debug/warning
messages generates clog messages 1000s of times per seconds which leads to
monstore db growing at an exponential rate in a catastrophic failure
situation.
The primary use cases for the logm entries in monstore db are :
- For "ceph log last" commands to get historical clog entries
- Ceph dashboard (mgr is subscriber of log-info which propagate clog to
dashboard module)
@Patrick Donnelly <pdonnell(a)redhat.com> suggested a viable solution to move
the cluster log storage to a new mgr module which handles the "ceph log
last" command. The clog data can be stored in the .mgr pool via
libcephsqlite.
Alternatively, if we donot want to get rid of logm storage from monstore db
then the other solutions would be :
- Stop writing logm entries to mon db if there are excessive entries
getting generated
- Filter out clog DBG entries and only log WRN/INF/ERR entries.
Looking forward to additional perspectives arounds this topic. Feel free to
add your inputs to trello card [1] or reply to this email-thread.
[1]
https://trello.com/c/oCGGFfTs/822-better-handling-of-cluster-log-messages-f…
Regards,
Prashant