Ceph Logging Configuration and "Large omap objects found"
Hi, I have a bunch of long-standing struggles with the way Ceph handles logging and I cannot figured out how to solve them. These issues are basically the following: - The log config options are utterly confusing and very badly documented - Mon file logs are spammed with DBG-level cluster logs, no matter what I configure - syslog/journald gets only part of the messages, but file logs must be turned off due to the spam above - "audit" channel logs cannot be configured at all From this, my following needs and questions arise, perhaps you can help: - I want to receive cluster and audit logs on the MONs with level "warn" or at most "info" - I want everything to go to journald - Where to cluster logs go if "clog_to_monitors" is off? - What's the difference between the "mon_cluster_log_*" and "clog_*" settings? - What the hell does "mon_cluster_log_to_syslog_facility" do and what does "audit=local0" mean or do? A very annoying symptom of the wonky logging config is that I cannot debug the infamous "LARGE_OMAP_OBJECTS" warning. It says "Search the cluster log for 'Large omap object found' for more details.", but I cannot do that, because without enabling the file-logging flood gates, I never receive the required cluster log info at the monitors and there seems to be no other way to debug this than to grep the cluster log (why??). My current log config is the following: global advanced clog_to_monitors true global basic err_to_syslog true global basic log_to_file false global basic log_to_stderr false global basic log_to_syslog true mon advanced mon_cluster_log_file /dev/null mon advanced mon_cluster_log_to_file false mon advanced mon_cluster_log_to_stderr false mon advanced mon_cluster_log_to_syslog true mon advanced mon_cluster_log_to_syslog_facility daemon mon advanced mon_cluster_log_to_syslog_level warn Any help solving this conundrum is much appreciated. Thanks! Janek
Hi, I don't have much to comment about logging, I feel you though. I just wanted to point out that the details about the large omap object should be in the (primary) OSD log, not in the MON log: grep -i "large omap" /var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.* /var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.1.log:2024-08-09T11:21:23.943+0000 7ffb66d10700 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 3:592df674:::file:head PG: 3.2e6fb49a (3.0) Key count: 363 Size (bytes): 2070 Regards, Eugen Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
Hi,
I have a bunch of long-standing struggles with the way Ceph handles logging and I cannot figured out how to solve them. These issues are basically the following:
- The log config options are utterly confusing and very badly documented - Mon file logs are spammed with DBG-level cluster logs, no matter what I configure - syslog/journald gets only part of the messages, but file logs must be turned off due to the spam above - "audit" channel logs cannot be configured at all
From this, my following needs and questions arise, perhaps you can help:
- I want to receive cluster and audit logs on the MONs with level "warn" or at most "info" - I want everything to go to journald - Where to cluster logs go if "clog_to_monitors" is off? - What's the difference between the "mon_cluster_log_*" and "clog_*" settings? - What the hell does "mon_cluster_log_to_syslog_facility" do and what does "audit=local0" mean or do?
A very annoying symptom of the wonky logging config is that I cannot debug the infamous "LARGE_OMAP_OBJECTS" warning. It says "Search the cluster log for 'Large omap object found' for more details.", but I cannot do that, because without enabling the file-logging flood gates, I never receive the required cluster log info at the monitors and there seems to be no other way to debug this than to grep the cluster log (why??).
My current log config is the following:
global advanced clog_to_monitors true global basic err_to_syslog true global basic log_to_file false global basic log_to_stderr false global basic log_to_syslog true mon advanced mon_cluster_log_file /dev/null mon advanced mon_cluster_log_to_file false mon advanced mon_cluster_log_to_stderr false mon advanced mon_cluster_log_to_syslog true mon advanced mon_cluster_log_to_syslog_facility daemon mon advanced mon_cluster_log_to_syslog_level warn
Any help solving this conundrum is much appreciated. Thanks!
Janek
I forgot to add this one to get the info from any admin node: ceph log last 10 warn cluster 2024-08-09T11:21:23.949916+0000 osd.1 (osd.1) 6 : cluster [WRN] Large omap object found. Object: 3:592df674:::file:head PG: 3.2e6fb49a (3.0) Key count: 363 Size (bytes): 2070 2024-08-09T11:21:27.723959+0000 mon.soc9-ceph (mon.0) 11905 : cluster [WRN] Health check failed: 1 large omap objects (LARGE_OMAP_OBJECTS) Then you don't have to search each node for logs. Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I don't have much to comment about logging, I feel you though. I just wanted to point out that the details about the large omap object should be in the (primary) OSD log, not in the MON log:
grep -i "large omap" /var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.* /var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.1.log:2024-08-09T11:21:23.943+0000 7ffb66d10700 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 3:592df674:::file:head PG: 3.2e6fb49a (3.0) Key count: 363 Size (bytes): 2070
Regards, Eugen
Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
Hi,
I have a bunch of long-standing struggles with the way Ceph handles logging and I cannot figured out how to solve them. These issues are basically the following:
- The log config options are utterly confusing and very badly documented - Mon file logs are spammed with DBG-level cluster logs, no matter what I configure - syslog/journald gets only part of the messages, but file logs must be turned off due to the spam above - "audit" channel logs cannot be configured at all
From this, my following needs and questions arise, perhaps you can help:
- I want to receive cluster and audit logs on the MONs with level "warn" or at most "info" - I want everything to go to journald - Where to cluster logs go if "clog_to_monitors" is off? - What's the difference between the "mon_cluster_log_*" and "clog_*" settings? - What the hell does "mon_cluster_log_to_syslog_facility" do and what does "audit=local0" mean or do?
A very annoying symptom of the wonky logging config is that I cannot debug the infamous "LARGE_OMAP_OBJECTS" warning. It says "Search the cluster log for 'Large omap object found' for more details.", but I cannot do that, because without enabling the file-logging flood gates, I never receive the required cluster log info at the monitors and there seems to be no other way to debug this than to grep the cluster log (why??).
My current log config is the following:
global advanced clog_to_monitors true global basic err_to_syslog true global basic log_to_file false global basic log_to_stderr false global basic log_to_syslog true mon advanced mon_cluster_log_file /dev/null mon advanced mon_cluster_log_to_file false mon advanced mon_cluster_log_to_stderr false mon advanced mon_cluster_log_to_syslog true mon advanced mon_cluster_log_to_syslog_facility daemon mon advanced mon_cluster_log_to_syslog_level warn
Any help solving this conundrum is much appreciated. Thanks!
Janek
Thanks all.
ceph log last 10 warn cluster
That outputs nothing for me. Any docs about this?
I don't have much to comment about logging, I feel you though. I just wanted to point out that the details about the large omap object should be in the (primary) OSD log, not in the MON log:
The message says cluster log. But even if it were the OSD logs, am I supposed to grep every single OSD log for it?
If you’re getting much volume to mon logs maybe you aren’t setting the level in a way that’s taking effect. Should mostly be quorum status and compaction results. I have set log levels, but it's largely ignored for the cluster log. There was an issue about this a few years back, I just can't find it right now.
Hi,
ceph log last 10 warn cluster
That outputs nothing for me. Any docs about this?
not any good docs, I'm afraid. At some point I stumbled across 'ceph log last cephadm' and played around a bit to see what else you can get from that. The help command shows some useful information:
log last [<num:int>] [<level:debug|info|sec|warn|error>] [<channel:*|cluster|audit|cephadm>] print last few lines of the cluster log
But I agree, there should be a section in the docs for that. I'm adding Zac in CC, maybe there is already some work going on in that regard.
The message says cluster log. But even if it were the OSD logs, am I supposed to grep every single OSD log for it?
That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something? BTW, which ceph version are we talking about here? Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
Thanks all.
ceph log last 10 warn cluster
That outputs nothing for me. Any docs about this?
I don't have much to comment about logging, I feel you though. I just wanted to point out that the details about the large omap object should be in the (primary) OSD log, not in the MON log:
The message says cluster log. But even if it were the OSD logs, am I supposed to grep every single OSD log for it?
If you’re getting much volume to mon logs maybe you aren’t setting the level in a way that’s taking effect. Should mostly be quorum status and compaction results. I have set log levels, but it's largely ignored for the cluster log. There was an issue about this a few years back, I just can't find it right now.
That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something?
BTW, which ceph version are we talking about here?
reef. I tried ceph log last 100 debug cluster and that gives me the usual DBG spam that I otherwise see in the MON logs. But there are no messages above that level.
I just played a bit more with the 'ceph log last' command, it doesn't have a large retention time, the messages get cleared out quickly, I suppose because they haven't changed. I'll take a closer look if and how that can be handled properly. Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something?
BTW, which ceph version are we talking about here?
reef.
I tried ceph log last 100 debug cluster and that gives me the usual DBG spam that I otherwise see in the MON logs. But there are no messages above that level.
Interesting, apparently the number one provides in the 'ceph log last <num>' command is not the number of lines to display but the number of lines to search for a match. So in your case you should still see your osd log output about the large omap if you pick a large enough number. My interpretation was that the number of lines you provide is the number of lines to be displayed in the selected log level. This needs to be documented. Zitat von Eugen Block <eblock@nde.ag>:
I just played a bit more with the 'ceph log last' command, it doesn't have a large retention time, the messages get cleared out quickly, I suppose because they haven't changed. I'll take a closer look if and how that can be handled properly.
Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something?
BTW, which ceph version are we talking about here?
reef.
I tried ceph log last 100 debug cluster and that gives me the usual DBG spam that I otherwise see in the MON logs. But there are no messages above that level.
Thanks. I increased the number even further and got a (literal) handful of non-debug messages. Unfortunately, none were relevant for the problem I'm trying to debug. On 13/08/2024 14:03, Eugen Block wrote:
Interesting, apparently the number one provides in the 'ceph log last <num>' command is not the number of lines to display but the number of lines to search for a match. So in your case you should still see your osd log output about the large omap if you pick a large enough number. My interpretation was that the number of lines you provide is the number of lines to be displayed in the selected log level. This needs to be documented.
Zitat von Eugen Block <eblock@nde.ag>:
I just played a bit more with the 'ceph log last' command, it doesn't have a large retention time, the messages get cleared out quickly, I suppose because they haven't changed. I'll take a closer look if and how that can be handled properly.
Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something?
BTW, which ceph version are we talking about here?
reef.
I tried ceph log last 100 debug cluster and that gives me the usual DBG spam that I otherwise see in the MON logs. But there are no messages above that level.
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de
Hm, then I don't see another way than to scan each OSD host for the omap message. Do you have a centralized logging or some configuration management like salt where you can target all hosts with a command? Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
Thanks. I increased the number even further and got a (literal) handful of non-debug messages. Unfortunately, none were relevant for the problem I'm trying to debug.
On 13/08/2024 14:03, Eugen Block wrote:
Interesting, apparently the number one provides in the 'ceph log last <num>' command is not the number of lines to display but the number of lines to search for a match. So in your case you should still see your osd log output about the large omap if you pick a large enough number. My interpretation was that the number of lines you provide is the number of lines to be displayed in the selected log level. This needs to be documented.
Zitat von Eugen Block <eblock@nde.ag>:
I just played a bit more with the 'ceph log last' command, it doesn't have a large retention time, the messages get cleared out quickly, I suppose because they haven't changed. I'll take a closer look if and how that can be handled properly.
Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something?
BTW, which ceph version are we talking about here?
reef.
I tried ceph log last 100 debug cluster and that gives me the usual DBG spam that I otherwise see in the MON logs. But there are no messages above that level.
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany
Phone: +49 3643 58 3577 www.webis.de
I do, but it's a lot of OSDs (1393 to be precise). On 14/08/2024 13:58, Eugen Block wrote:
Hm, then I don't see another way than to scan each OSD host for the omap message. Do you have a centralized logging or some configuration management like salt where you can target all hosts with a command?
Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
Thanks. I increased the number even further and got a (literal) handful of non-debug messages. Unfortunately, none were relevant for the problem I'm trying to debug.
On 13/08/2024 14:03, Eugen Block wrote:
Interesting, apparently the number one provides in the 'ceph log last <num>' command is not the number of lines to display but the number of lines to search for a match. So in your case you should still see your osd log output about the large omap if you pick a large enough number. My interpretation was that the number of lines you provide is the number of lines to be displayed in the selected log level. This needs to be documented.
Zitat von Eugen Block <eblock@nde.ag>:
I just played a bit more with the 'ceph log last' command, it doesn't have a large retention time, the messages get cleared out quickly, I suppose because they haven't changed. I'll take a closer look if and how that can be handled properly.
Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something?
BTW, which ceph version are we talking about here?
reef.
I tried ceph log last 100 debug cluster and that gives me the usual DBG spam that I otherwise see in the MON logs. But there are no messages above that level.
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany
Phone: +49 3643 58 3577 www.webis.de
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de
participants (2)
-
Eugen Block
-
Janek Bevendorff