Hello, I am running a Ceph cluster installed with cephadm. Version is 18.2.2 reef (stable). I am moving the DB/WAL from my HDDs to SSD, and have been doing fine on all the OSDs until I got to one in particular (osd.14) From the cephadm shell, when I run 'ceph orch daemon stop osd.14', nothing happens: it does not get marked as Down. If I mark is as Down in the GUI, it does show as Down then a few seconds later it gets marks as Up again. I was running 'journalctl -xf | grep osd.14' to see if any errors came up, but nothing did. Not sure where to check next to try to sort this out?
Hi, if the daemon is up, marking it down will only have a temporary effect: mon.nautilus2@0(leader).osd e16065 definitely_dead 0 mon.nautilus2@0(leader).osd e16065 do_prune osdmap full prune enabled log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN) mon.nautilus2@0(leader).osd e16066 e16066: 9 total, 8 up, 9 in log_channel(audit) log [INF] : from='mgr.2794107 IP:0/2317957193' entity='mgr.nautilus2' cmd='[{"prefix": "osd down", "format": "json", "ids": ["0"]}]': finished log_channel(cluster) log [DBG] : osdmap e16066: 9 total, 8 up, 9 in log_channel(cluster) log [INF] : osd.0 marked itself dead as of e16066 mon.nautilus2@0(leader).osd e16066 do_prune osdmap full prune enabled log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down) log_channel(cluster) log [INF] : Cluster is now healthy What's the output of ceph osd ok-to-stop 14 If it should be safe to stop it, inspect the mgr log why it refuses to stop it. You can always fail the mgr (ceph mgr fail) and retry. Zitat von Alan Murrell <Alan@t-net.ca>:
Hello,
I am running a Ceph cluster installed with cephadm. Version is 18.2.2 reef (stable).
I am moving the DB/WAL from my HDDs to SSD, and have been doing fine on all the OSDs until I got to one in particular (osd.14)
From the cephadm shell, when I run 'ceph orch daemon stop osd.14', nothing happens: it does not get marked as Down. If I mark is as Down in the GUI, it does show as Down then a few seconds later it gets marks as Up again.
I was running 'journalctl -xf | grep osd.14' to see if any errors came up, but nothing did.
Not sure where to check next to try to sort this out? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
What's the output of
ceph osd ok-to-stop 14
Here it is: --- START --- root@cephnode01:/# ceph osd ok-to-stop 14 {"ok_to_stop":true,"osds":[14],"num_ok_pgs":10,"num_not_ok_pgs":0,"ok_become_degraded":["18.0","18.2","18.3","18.f","18.1b","18.26","18.27","18.29","18.2e","18.3a"]} --- END --- It appears to be OK to stop. Here is the output in /var/log/ceph/ceph-mgr.cephnode01.kefvmh.log (I was grepping for just "osd.14" as there is a lot of info that goes through it): --- START --- 2025-02-04T16:54:14.221+0000 7f6b5766a700 0 log_channel(audit) log [DBG] : from='client.86159195 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "stop", "name": "osd.14", "target": ["mon-mgr", ""]}]: dispatch 2025-02-04T16:54:14.221+0000 7f6b4964e700 0 [cephadm INFO root] Schedule stop daemon osd.14 2025-02-04T16:54:14.221+0000 7f6b4964e700 0 log_channel(cephadm) log [INF] : Schedule stop daemon osd.14 --- END --- No other entries for osd.14 come up.
You can always fail the mgr (ceph mgr fail) and retry.
I failed the current active mgr (cephnode01) and the secondary took over, but running the 'ceph orch daemon stop osd.14' still did not work. Something *seems* to wonky with this OSD< but everything appears healthy. I even ran a SMART test against the HDD itself and it reported healthy.
Is there anything in the OSD logs? I might misremember, but when I fiddle with my test clusters, the podman/systemd control sometimes breaks, but after a reboot it's usually fine. Does it stop if you simply run 'systemctl stop ceph-{CEPH_FSID}@osd.14'? I would also inspect the 'podman ps' or 'docker ps' output on that node, maybe the daemon is in some error state although it's running? Zitat von Alan Murrell <Alan@t-net.ca>:
You can always fail the mgr (ceph mgr fail) and retry.
I failed the current active mgr (cephnode01) and the secondary took over, but running the 'ceph orch daemon stop osd.14' still did not work.
Something *seems* to wonky with this OSD< but everything appears healthy. I even ran a SMART test against the HDD itself and it reported healthy.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is there anything in the OSD logs?
When I issue 'ceph orch daemon stop osd.14', nothing is showing up in ' /var/log/ceph/474264fe-b00e-11ee-b586-ac1f6b0ff21a/ceph-osd.14.log' (I was doing a 'tail -f' on the log file at the time I issued to daemon stop command and absolutely nothing showed up in it)
when I fiddle with my test clusters, the podman/systemd control sometimes breaks, but after a reboot it's usually fine.
I just rebooted the node and all the OSDs came up OK, but osd.14 will still not stop when I issue the 'ceph orch daemon stop osd.14' command.
Does it stop if you simply run 'systemctl stop ceph-{CEPH_FSID}@osd.14'?
That *does* indeed appear to work (and running 'systemctl start ceph-{CEPH_FSID}@osd.14' starts it back up). Using those commands, I have been able to get the DB/WAL for osd.14 migrated off. Weird that there appears to be some sort of disconnect with the OSD when using the 'ceph orch daemon' command, but as it seems to be just this one OSD (for now...), and doesn't *really* impact anything, I think I can live with it. Thanks (again!) for your assistance and guidance.
I would try one more thing (locally on that node): cephadm unit stop --name osd.14 Does that work? If not, you could check the cephadm.log for more hints. If it works, you can start the OSD again with: cephadm unit start --name osd.14 Zitat von Alan Murrell <Alan@t-net.ca>:
Is there anything in the OSD logs?
When I issue 'ceph orch daemon stop osd.14', nothing is showing up in ' /var/log/ceph/474264fe-b00e-11ee-b586-ac1f6b0ff21a/ceph-osd.14.log' (I was doing a 'tail -f' on the log file at the time I issued to daemon stop command and absolutely nothing showed up in it)
when I fiddle with my test clusters, the podman/systemd control sometimes breaks, but after a reboot it's usually fine.
I just rebooted the node and all the OSDs came up OK, but osd.14 will still not stop when I issue the 'ceph orch daemon stop osd.14' command.
Does it stop if you simply run 'systemctl stop ceph-{CEPH_FSID}@osd.14'?
That *does* indeed appear to work (and running 'systemctl start ceph-{CEPH_FSID}@osd.14' starts it back up). Using those commands, I have been able to get the DB/WAL for osd.14 migrated off. Weird that there appears to be some sort of disconnect with the OSD when using the 'ceph orch daemon' command, but as it seems to be just this one OSD (for now...), and doesn't *really* impact anything, I think I can live with it.
Thanks (again!) for your assistance and guidance. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
cephadm unit stop --name osd.14
Does that work? If not, you could check the cephadm.log for more hints. If it works, you can start the OSD again with:
Yep! That does indeed work! I will add that to my notes as an alternate command should an OSD not respond to the 'ceph orch daemon' command.
participants (2)
-
Alan Murrell
-
Eugen Block