Cephadm flooding /var/log/ceph/cephadm.log
Hi everyone. My company has paid Ceph support. The tech is saying: "...cephadm package of ceph 5 is having bug. So It generate debug logs even its set for "info" logs..." I have two clusters running, one Ceph 5 and the other Ceph 6 (Quincy). Both of them are sending "DEBUG ---------" messages to cephadm.log. I checked everywhere I can think of and all the logs are set to INFO. Is there really a bug? I can't find any mention of one anywhere on the internet, besides, it seems that if there really is a bug it hasn't been fixed in version 6 either. I tried setting " mgr/cephadm/log_to_cluster_level" and "mgr/cephadm/log_level" to WARN and noting changed. /etc/ceph/ceph.conf has no mention of log level. Any help would be appreciated, Thanks!
I think it's just hard-coded in the cephadm binary [0]: _log_file_handler = { 'level': 'DEBUG', 'class': 'logging.handlers.WatchedFileHandler', 'formatter': 'cephadm', 'filename': '%s/cephadm.log' % LOG_DIR, } I haven't looked to deep yet if it can be overridden, but didn't find anything on a quick search. [0] https://github.com/ceph/ceph/blob/46ba365307312980303de311cacbb49974dbc74f/s... Zitat von Alex <mr.alexey@gmail.com>:
Hi everyone.
My company has paid Ceph support. The tech is saying: "...cephadm package of ceph 5 is having bug. So It generate debug logs even its set for "info" logs..." I have two clusters running, one Ceph 5 and the other Ceph 6 (Quincy). Both of them are sending "DEBUG ---------" messages to cephadm.log. I checked everywhere I can think of and all the logs are set to INFO.
Is there really a bug? I can't find any mention of one anywhere on the internet, besides, it seems that if there really is a bug it hasn't been fixed in version 6 either.
I tried setting " mgr/cephadm/log_to_cluster_level" and "mgr/cephadm/log_level" to WARN and noting changed.
/etc/ceph/ceph.conf has no mention of log level.
Any help would be appreciated, Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Interesting. So it's like that for everybody? Meaning cephadm.log logs debug messages.
Hey all, Just confirming that the same debug level has been in Reef and Squid. We got so used to it that just decided not to care anymore. Best, Laimis J.
On 8 Apr 2025, at 14:21, Alex <mr.alexey@gmail.com> wrote:
Interesting. So it's like that for everybody? Meaning cephadm.log logs debug messages. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is this bit of code responsible for hardcoding DEBUG to cephadm.log? 'loggers': { '': { 'level': 'DEBUG', 'handlers': ['console', 'log_file'], } } in /var/lib/ceph/<fsid>/cephadm.* ?
I mean this bit 'log_file': { 'level': 'DEBUG', 'class': 'logging.handlers.WatchedFileHandler', 'formatter': 'cephadm', 'filename': '%s/cephadm.log' % LOG_DIR, }
Thanks Eugen! I think you're right since support had me grep for the same code. Seems crazy though that it's hardcoded doesn't it? I guess we can mod the Python file, but you'd think that wouldn't be necessary. Should we make a feature request or modify the code ourselves and make a pull request?
I guess it's debatable as almost everything. ;-) One of the advantages is that you usually see immediately what is failing, you don't have to turn on debug first, retry the deployment or whatever again to reproduce. The file doesn't really grow to huge sizes (around 2 MB per day or so) and gets logrotated. Most people seem okay with it, at least I haven't seen many complaints. But of course, you're always welcome to propose a feature or PR. Zitat von Alex <mr.alexey@gmail.com>:
Thanks Eugen!
I think you're right since support had me grep for the same code. Seems crazy though that it's hardcoded doesn't it? I guess we can mod the Python file, but you'd think that wouldn't be necessary.
Should we make a feature request or modify the code ourselves and make a pull request?
Can someone paste in here their copy of logrotate? The trick always with rotating logs is that the service writing to it needs to be restarted or told to stop writing so the file handle gets closed. Otherwise it stays open and the free disc space isn't recovered.
Official IBM and RH "fix" is to replace DEBUG with INFO in /var/lib/ceph/<FSID>/cephadm.hash ¯\_ (ツ) _/¯
Haha yeah think that's what we're doing. I'm just going to add it to logrotate. Do you use copytruncate option or postrotate to restart ceph?
I haven't modified the logrotate config from the cephadm package. There's no cephadm process running, it's invoked everytime the orchestrator does one of its checks (host, network, osd specs etc.). So I don't see a reason to tweak anything here, I stick to the defaults. Zitat von Alex <mr.alexey@gmail.com>:
Haha yeah think that's what we're doing. I'm just going to add it to logrotate. Do you use copytruncate option or postrotate to restart ceph?
I have at times used admin sockets to tell daemons to re-open log files after running a truncate
On Apr 9, 2025, at 3:51 PM, Eugen Block <eblock@nde.ag> wrote:
I haven't modified the logrotate config from the cephadm package. There's no cephadm process running, it's invoked everytime the orchestrator does one of its checks (host, network, osd specs etc.). So I don't see a reason to tweak anything here, I stick to the defaults.
Zitat von Alex <mr.alexey@gmail.com>:
Haha yeah think that's what we're doing. I'm just going to add it to logrotate. Do you use copytruncate option or postrotate to restart ceph?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I did have to add "su root root" to the log rotate script to fix the permissions issue. There's a RH KB article and Ceph github pull requests to fix it.
This is from a openSUSE base OS: # cat /etc/logrotate.d/cephadm # created by cephadm /var/log/ceph/cephadm.log { rotate 7 daily compress missingok notifempty su root root } Zitat von Alex <mr.alexey@gmail.com>:
I did have to add "su root root" to the log rotate script to fix the permissions issue. There's a RH KB article and Ceph github pull requests to fix it.
I made a Pull Request for cephadm.log set DEBUG. Not sure if I should merge it.
Link please.
On Apr 10, 2025, at 10:59 PM, Alex <mr.alexey@gmail.com> wrote:
I made a Pull Request for cephadm.log set DEBUG. Not sure if I should merge it. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thursday, April 10, 2025 10:42:50 PM Eastern Daylight Time Alex wrote:
I made a Pull Request for cephadm.log set DEBUG. Not sure if I should merge it.
Please, no. Even if github allows you to (I think it won't) you should not merge your own PRs unless you are a component lead and it is an emergency (IMO - but I think many would agree). I have left a comment on what I assume is your pr: https://github.com/ceph/ ceph/pull/62789 PS. If you meant creating merge commits in your own PR branch, that is unsightly but harmless. If you meant what I think you meant - merge the PR into ceph main - that's what I object to. You need to follow the contribution process and get reviews, and have the PR get tested, before the component lead would possibly merge the changes into the main branch.
participants (5)
-
Alex
-
Anthony D'Atri
-
Eugen Block
-
John Mulligan
-
Laimis Juzeliūnas