One host down osd status error
Hi all, We are running a ceph cluster with filesystem that contains 5 servers. Ceph version: 19.2.0 squid If I run: ceph osd status when all hosts are online and in the output is the way it should and it prints status for all osds. If just a couple of osds are down status is printed and specific osds are stated as down. One of the servers went down and we ended up with a health warning. If I run: ceph osd stat I get the information that 64 out of 80 osd is in. If I try to run: ceph osd status I get an python error: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1864, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/status/module.py", line 337, in handle_osd_status assert metadata AssertionError I suppose this is some type of bug when one host is down? Thanks! Marcus
Based on my experience, that error comes from 1 of 3 possible causes: 1. The machine in question doesn't have proper security keys 2. The machine in question is short on resources - especially RAM 3. The machine in question has its brains scrambled. Cosmic rays flipping critical RAM bits, bugs in OS software, whatever. Rebooting is the only fix for that. On 3/20/25 07:10, Marcus wrote:
Hi all, We are running a ceph cluster with filesystem that contains 5 servers. Ceph version: 19.2.0 squid
If I run: ceph osd status when all hosts are online and in the output is the way it should and it prints status for all osds. If just a couple of osds are down status is printed and specific osds are stated as down.
One of the servers went down and we ended up with a health warning. If I run: ceph osd stat I get the information that 64 out of 80 osd is in.
If I try to run: ceph osd status I get an python error: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1864, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/status/module.py", line 337, in handle_osd_status assert metadata AssertionError
I suppose this is some type of bug when one host is down?
Thanks! Marcus
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, this works for me in multiple virtual test clusters across different Ceph versions, including 19.2.0. Both within a cephadm shell as well as outside of it. Maybe do a 'ceph mgr fail' and retry? Zitat von Marcus <marcus@marcux.org>:
Hi all, We are running a ceph cluster with filesystem that contains 5 servers. Ceph version: 19.2.0 squid
If I run: ceph osd status when all hosts are online and in the output is the way it should and it prints status for all osds. If just a couple of osds are down status is printed and specific osds are stated as down.
One of the servers went down and we ended up with a health warning. If I run: ceph osd stat I get the information that 64 out of 80 osd is in.
If I try to run: ceph osd status I get an python error: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1864, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/status/module.py", line 337, in handle_osd_status assert metadata AssertionError
I suppose this is some type of bug when one host is down?
Thanks! Marcus
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Thanks for the tip Eugen!! I stopped the active systemd mgr so the cluster failed over to another mgr. After this it all worked fine! Started the systemd mgr again and it came up as a standby again. Suppose the mgr got som hickup somehow, did not found any specific in the log.
I know, unfortunately, this has been an issue for two or three years now. The first thing I (and many others) suggest if anything stopped working is to fail the mgr. My impression is that in the past years, more and more features were added to the mgr while the default configs haven't changed, causing it to silently fail or at least misbehave. I created a tracker [0] for one specific issue I saw on a customer cluster last year. My theory is that due to too low defaults, the mgr communication between MONs, OSDs, MGRs etc. gets flooded and some messages get lost. But I haven't found a way to reproduce it in test clusters yet, so it's still only a theory. [0] https://tracker.ceph.com/issues/66310 Zitat von Marcus <marcus@marcux.org>:
Hi, Thanks for the tip Eugen!!
I stopped the active systemd mgr so the cluster failed over to another mgr. After this it all worked fine! Started the systemd mgr again and it came up as a standby again. Suppose the mgr got som hickup somehow, did not found any specific in the log.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Marcus
-
Tim Holloway