Hi Igor, the ceph-osd.2.log remains empty on the node where this osd is located. This is what I get when manualy restarting the osd. [root@mostha1 250f9864-0142-11ee-8e5f-00266cf8869c]# systemctl restart ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service Job for ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service failed because a timeout was exceeded. See "systemctl status ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service" and "journalctl -xe" for details. [root@mostha1 250f9864-0142-11ee-8e5f-00266cf8869c]# journalctl -xe sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 5728 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 5882 (bash) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 5884 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 6031 (bash) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 6033 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 6185 (bash) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 6187 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 14627 (bash) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 14629 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 14776 (bash) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 14778 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 15169 (bash) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 15171 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 15646 (bash) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 15648 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 15792 (bash) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 15794 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 25561 (bash) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 25563 (podman) in control group while starting unit. Ignoring. sept. 21 13:22:39 mostha1.legi.grenoble-inp.fr systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Patrick Le 21/09/2023 à 12:44, Igor Fedotov a écrit :
Hi Patrick,
please share osd restart log to investigate that.
Thanks,
Igor
On 21/09/2023 13:41, Patrick Begou wrote:
Hi,
After a power outage on my test ceph cluster, 2 osd fail to restart. The log file show:
8e5f-00266cf8869c@osd.2.service: Failed with result 'timeout'. Sep 21 11:55:02 mostha1 systemd[1]: Failed to start Ceph osd.2 for 250f9864-0142-11ee-8e5f-00266cf8869c. Sep 21 11:55:12 mostha1 systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Service RestartSec=10s expired, scheduling restart. Sep 21 11:55:12 mostha1 systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Scheduled restart job, restart counter is at 2. Sep 21 11:55:12 mostha1 systemd[1]: Stopped Ceph osd.2 for 250f9864-0142-11ee-8e5f-00266cf8869c. Sep 21 11:55:12 mostha1 systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 1858 (bash) in control group while starting unit. Ignoring. Sep 21 11:55:12 mostha1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Sep 21 11:55:12 mostha1 systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c@osd.2.service: Found left-over process 2815 (podman) in control group while starting unit. Ignoring.
This is not critical as it is a test cluster and it is actually rebalancing on other osd but I would like to know how to return to HEALTH_OK status.
Smartctl show the HDD are OK.
So is there a way to recover the osd from this state ? Version is 15.2.17 (juste moved from 15.2.13 to 15.2.17 yesterday, will try to move to latest versions as soon as this problem is solved)
Thanks
Patrick
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io