I do, but it's a lot of OSDs (1393 to be precise). On 14/08/2024 13:58, Eugen Block wrote:
Hm, then I don't see another way than to scan each OSD host for the omap message. Do you have a centralized logging or some configuration management like salt where you can target all hosts with a command?
Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
Thanks. I increased the number even further and got a (literal) handful of non-debug messages. Unfortunately, none were relevant for the problem I'm trying to debug.
On 13/08/2024 14:03, Eugen Block wrote:
Interesting, apparently the number one provides in the 'ceph log last <num>' command is not the number of lines to display but the number of lines to search for a match. So in your case you should still see your osd log output about the large omap if you pick a large enough number. My interpretation was that the number of lines you provide is the number of lines to be displayed in the selected log level. This needs to be documented.
Zitat von Eugen Block <eblock@nde.ag>:
I just played a bit more with the 'ceph log last' command, it doesn't have a large retention time, the messages get cleared out quickly, I suppose because they haven't changed. I'll take a closer look if and how that can be handled properly.
Zitat von Janek Bevendorff <janek.bevendorff@uni-weimar.de>:
That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something?
BTW, which ceph version are we talking about here?
reef.
I tried ceph log last 100 debug cluster and that gives me the usual DBG spam that I otherwise see in the MON logs. But there are no messages above that level.
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany
Phone: +49 3643 58 3577 www.webis.de
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de