Hi All,
We are looking for inputs on a new feature to be implemented to move clog messages storage from monstore db, refer trello card [1] for more details around this topic.
Currently, every clog message goes to monstore db as well as debug/warning messages generates clog messages 1000s of times per seconds which leads to monstore db growing at an exponential rate in a catastrophic failure situation.
The primary use cases for the logm entries in monstore db are :
- For "ceph log last" commands to get historical clog entries
- Ceph dashboard (mgr is subscriber of log-info which propagate clog to dashboard module)
@Patrick Donnelly suggested a viable solution to move the cluster log storage to a new mgr module which handles the "ceph
log last" command. The clog data can be stored in the .mgr pool via
libcephsqlite.
Alternatively, if we donot want to get rid of logm storage from monstore db then the other solutions would be :
- Stop writing logm entries to mon db if there are excessive entries getting generated
- Filter out clog DBG entries and only log WRN/INF/ERR entries.
Looking forward to additional perspectives arounds this topic. Feel free to add your inputs to trello card [1] or reply to this email-thread.