Hi Adam! I guess you only want the output for the 9100 port? [root@darkside1]# ss -tulpn | grep 9100 tcp LISTEN 0 128 [::]:9100 [::]:* users:(("node_exporter",pid=9103,fd=3)) Also, this: [root@darkside1 ~]# ps aux | grep 9103 nfsnobo+ 9103 38.4 0.0 152332 105760 ? Ssl 10:12 82:35 /bin/node_exporter --no-collector.timex Cordially, Renata. On 7/25/23 13:22, Adam King wrote:
okay, not much info on the mon failure. The other one at least seems to be a simple port conflict. What does `sudo netstat -tulpn` give you on that host?
On Tue, Jul 25, 2023 at 12:00 PM Renata Callado Borges <renato.callado@incor.usp.br> wrote:
Hi Adam!
Thank you for your response, but I am still trying to figure out the issue. I am pretty sure the problem occurs "inside" the container, and I don´t know how to get logs from there.
Just in case, this is what systemd sees:
Jul 25 12:36:32 darkside1 systemd[1]: Stopped Ceph mon.darkside1 for 920740ee-cf2d-11ed-9097-08c0eb320eda. Jul 25 12:36:32 darkside1 systemd[1]: Starting Ceph mon.darkside1 for 920740ee-cf2d-11ed-9097-08c0eb320eda... Jul 25 12:36:33 darkside1 bash[52271]: ceph-920740ee-cf2d-11ed-9097-08c0eb320eda-mon.darkside1 Jul 25 12:36:33 darkside1 podman[52311]: 2023-07-25 12:36:33.32233695 -0300 -03 m=+0.131005321 container create 7cf1d340e0a9658b2c9dac3a039a6f5b4fd2a5581bb60213b0ab708d24b69f54 (image=quay.io/ceph/ceph:v15 <http://quay.io/ceph/ceph:v15>, name=ceph-920740ee-cf2d-11ed-9097-08c0eb320eda-mon.darkside1) Jul 25 12:36:33 darkside1 podman[52311]: 2023-07-25 12:36:33.526241218 -0300 -03 m=+0.334909578 container init 7cf1d340e0a9658b2c9da c3a039a6f5b4fd2a5581bb60213b0ab708d24b69f54 (image=quay.io/ceph/ceph:v15 <http://quay.io/ceph/ceph:v15>, name=ceph-920740ee-cf2d-11ed-9097-08c0eb320eda-mon.darksid e1) Jul 25 12:36:33 darkside1 podman[52311]: 2023-07-25 12:36:33.556646854 -0300 -03 m=+0.365315225 container start 7cf1d340e0a9658b2c9d ac3a039a6f5b4fd2a5581bb60213b0ab708d24b69f54 (image=quay.io/ceph/ceph:v15 <http://quay.io/ceph/ceph:v15>, name=ceph-920740ee-cf2d-11ed-9097-08c0eb320eda-mon.darksi de1) Jul 25 12:36:33 darkside1 bash[52271]: 7cf1d340e0a9658b2c9dac3a039a6f5b4fd2a5581bb60213b0ab708d24b69f54 Jul 25 12:36:33 darkside1 systemd[1]: Started Ceph mon.darkside1 for 920740ee-cf2d-11ed-9097-08c0eb320eda. Jul 25 12:36:43 darkside1 systemd[1]: ceph-920740ee-cf2d-11ed-9097-08c0eb320eda@mon.darkside1.service: main process exited, code=exi ted, status=1/FAILURE Jul 25 12:36:43 darkside1 systemd[1]: Unit ceph-920740ee-cf2d-11ed-9097-08c0eb320eda@mon.darkside1.service entered failed state. Jul 25 12:36:43 darkside1 systemd[1]: ceph-920740ee-cf2d-11ed-9097-08c0eb320eda@mon.darkside1.service failed. Jul 25 12:36:53 darkside1 systemd[1]: ceph-920740ee-cf2d-11ed-9097-08c0eb320eda@mon.darkside1.service holdoff time over, scheduling restart. Jul 25 12:36:53 darkside1 systemd[1]: Stopped Ceph mon.darkside1 for 920740ee-cf2d-11ed-9097-08c0eb320eda. Jul 25 12:36:53 darkside1 systemd[1]: start request repeated too quickly for ceph-920740ee-cf2d-11ed-9097-08c0eb320eda@mon.darkside1 .service Jul 25 12:36:53 darkside1 systemd[1]: Failed to start Ceph mon.darkside1 for 920740ee-cf2d-11ed-9097-08c0eb320eda. Jul 25 12:36:53 darkside1 systemd[1]: Unit ceph-920740ee-cf2d-11ed-9097-08c0eb320eda@mon.darkside1.service entered failed state. Jul 25 12:36:53 darkside1 systemd[1]: ceph-920740ee-cf2d-11ed-9097-08c0eb320eda@mon.darkside1.service failed.
Also, I get the following error every 10 minutes or so on "ceph -W cephadm --watch-debug":
2023-07-25T12:35:38.115146-0300 mgr.darkside3.ujjyun [INF] Deploying daemon node-exporter.darkside1 on darkside1 2023-07-25T12:35:38.612569-0300 mgr.darkside3.ujjyun [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter. darkside1 ... Verifying port 9100 ... Cannot bind to IP 0.0.0.0 port 9100: [Errno 98] Address already in use ERROR: TCP Port(s) '9100' required for node-exporter already in use Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 1029, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1185, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.darkside1 ... Verifying port 9100 ... Cannot bind to IP 0.0.0.0 port 9100: [Errno 98] Address already in use ERROR: TCP Port(s) '9100' required for node-exporter already in use
And finally I get this error on the first line of output for my "ceph mon dump":
2023-07-25T12:46:17.008-0300 7f145f59e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only suppor t [2,1]
Cordially,
Renata.
On 7/24/23 10:57, Adam King wrote: > The logs you probably really want to look at here are the journal logs > from the mgr and mon. If you have a copy of the cephadm tool on the > host, you can do a "cephadm ls --no-detail | grep systemd" to list out > the systemd unit names for the ceph daemons on the host, or just look > find the systemd unit names in the standard way you would for any > other systemd unit (e.g. "systemctl -l | grep mgr'' will probably > include the mgr one) and then take a look at "journalctl -eu > <systemd-unit-name>" for the systemd unit for both the mgr and the > mon. I'd expect near the end of the log it would include a reason for > going down. > > As for the debug_ms (I think that's what you want over "debug mon") > stuff, I think that would need to be a command line option for the > mgr/mon process. For cephadm deployments, the systemd unit is run > through a "unit.run" file in > /var/lib/ceph/<cluster-fsid>/<daemon-name>/unit.run. If you go to the > very end of that file, which will be a very long podman or docker run > command, add in the "--debug_ms 20" and then restart the systemd unit > for that daemon, it should cause the extra debug logging to happen > from that daemon. I would say first check if there are useful errors > in the journal logs mentioned above before trying that though. > > On Mon, Jul 24, 2023 at 9:47 AM Renata Callado Borges > <renato.callado@incor.usp.br> wrote: > > Dear all, > > > How are you? > > I have a cluster on Pacific with 3 hosts, each one with 1 mon, 1 mgr > and 12 OSDs. > > One of the hosts, darkside1, has been out of quorum according to ceph > status. > > Systemd showed 4 services dead, two mons and two mgrs. > > I managed to systemctl restart one mon and one mgr, but even after > several attempts, the remaining mon and mgr services, when asked to > restart, keep returning to a failed state after a few seconds. > They try > to auto-restart and then go into a failed state where systemd > requires > me to manually set them to "reset-failed" before trying to start > again. > But they never stay up. There are no clear messages about the > issue in > /var/log/ceph/cephadm.log. > > The host is still out of quorum. > > > I have failed to "turn on debug" as per > https://docs.ceph.com/en/pacific/rados/troubleshooting/log-and-debug/. > > It seems I do not know the proper incantantion for "ceph daemon X > config > show", no string for X seems to satisfy this command. I have tried > adding this: > > [mon] > > debug mon = 20 > > > To my ceph.conf, but no additional lines of log are sent to > /var/log/cephadm.log > > > so I'm sorry I can´t provide more details. > > > Could someone help me debug this situation? I am sure that if just > reboot the machine, it will start up the services properly, as it > always > has done, but I would prefer to fix this without this action. > > > Cordially, > > Renata. > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io