Hi, I'm in the process of upgrading my cluster from 17.2.5 to 17.2.6 but the following problem existed when I was still everywhere on 17.2.5 . I had a major issue in my cluster which could be solved with a lot of your help and even more trial and error. Right now it seems that most is already fixed but I can't rule out that there's still some problem hidden. The very issue I'm asking about started during the repair. When I want to orchestrate the cluster, it logs the command but it doesn't do anything. No matter if I use ceph dashboard or "ceph orch" in "cephadm shell". I don't get any error message when I try to deploy new services, redeploy them etc. The log only says "scheduled" and that's it. Same when I change placement rules. Usually I use tags. But since they don't work anymore, too, I tried host and umanaged. No success. The only way I can actually start and stop containers is via systemctl from the host itself. When I run "ceph orch ls" or "ceph orch ps" I see services I deployed for testing being deleted (for weeks now). Ans especially a lot of old MDS are listed as "error" or "starting". The list doesn't match reality at all because I had to start them by hand. I tried "ceph mgr fail" and even a complete shutdown of the whole cluster with all nodes including all mgs, mds even osd - everything during a maintenance window. Didn't change anything. Could you help me? To be honest I'm still rather new to Ceph and since I didn't find anything in the logs that caught my eye I would be thankful for hints how to debug. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

Thanks. I set the log level to debug, try a few steps and then come back. On 04.05.23 14:48, Eugen Block wrote: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

This is sort of similar to what I said in a previous email, but the only way I've seen this happen in other setups is through hanging cephadm commands. The debug process has been, do a mgr failover, wait a few minutes, see in "ceph orch ps" and "ceph orch device ls" which hosts have and have not been refreshed (the REFRESHED column should be some lower value on the hosts where it refreshed), go to the hosts where it did not refresh and check "ps aux | grep cephadm" looking for long running (and therefore most likely hung) processes. I would still expect that's the

likely thing you're experiencing here. I haven't seen any other causes

cephadm to not refresh unless the module crashed, but that would be explicitly stated in the cluster health. On Mon, May 15, 2023 at 11:44 AM Thomas Widhalm <widhalmt(a)widhalm.or.at> wrote: > Hi, > > I tried a lot of different approaches but I didn't have any success so

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

If it persisted through a full restart, it's possible the conditions that caused the hang are still present after the fact. The two known causes I'm aware of are lack of space in the root partition and hanging mount points. Both would show up as processes in "ps aux | grep cephadm" though. The latter could possibly be related to cephfs pool issues if you have something mounted on one of the host hosts. Still hard to say without knowing what exactly got stuck. For clarity, without restarting or changing anything else, can you verify if "ps aux | grep cephadm" shows anything on the nodes. I know I'm a bit of a broken record on mentioning the hanging processes stuff, but outside of module crashes which don't appear to be present here, 100% of other cases of this type of thing happening I've looked at before have had those processes sitting around. On Mon, May 15, 2023 at 3:10 PM Thomas Widhalm &lt;widhalmt(a)widhalm.or.at <mailto:widhalmt@widhalm.or.at>> wrote: This is why I even tried a full cluster shutdown. All Hosts were out, so there's not a possibility that there's any process hanging. After I started the nodes, it's just the same as before. All refresh times

"4 weeks". Like it stopped simoultanously on all nodes. Some time ago we had a small change in name resolution so I thought, maybe the orchestrator can't connect via ssh anymore. But I tried all the steps in

The only thing that's slightly suspicous is that, it said, it added

host key to known hosts. But since I tried via "cephadm shell" I

the known hosts are just not replicated to these containers. ssh

too. (And I would have suspected that I get a warning if that failed) I don't see any information about the orchestrator module having crashed. It's running as always. From the the prior problem I had some issues in my cephfs pools.

maybe there's something broken in the .mgr pool? Could that be a

for this behaviour? I googled a while but didn't find any way how to check that explicitly. On 15.05.23 19:15, Adam King wrote: the only > way I've seen this happen in other setups is through hanging

> commands. The debug process has been, do a mgr failover, wait a

hosts have lower did not running (and that's the most causes for > cephadm to not refresh unless the module crashed, but that would

&lt;widhalmt(a)widhalm.or.at <mailto:widhalmt@widhalm.or.at>> success so far. - 4w ago 4w ago - 4w ago 4w ago 4w ago 4w ago 4w ago 1376 : 1380 : 1392 : 1394 : 2629 : >> cephadm [INF] Saving service mds.mds01 spec with placement

2780 : >> cephadm [INF] Saving service mds.fs_name spec with placement

2795 : >> cephadm [INF] Saving service mds.mds01 spec with placement

information unmanaged mgr. I daemons run >> the same version. (If you remember, upgrade failed underway). >> >> So my only way of getting daemons only is manually. I added two

>> hosts, tagged them. But there isn't a single daemon started

>> >> Could you help me again with how to debug orchestration not

>> >> >> On 04.05.23 15:12, Thomas Widhalm wrote: >>> Thanks. >>> >>> I set the log level to debug, try a few steps and then come

where it's between actually <mailto:widhalmt@widhalm.or.at>>: 17.2.6 but 17.2.5 . a lot of that most problem repair. but it "ceph orch" >>>>> in "cephadm shell". I don't get any error message when I try

"scheduled" >>>>> and that's it. Same when I change placement rules. Usually I

>>>>> tags. But since they don't work anymore, too, I tried host and >>>>> umanaged. No success. The only way I can actually start and

deployed lot of match >>>>> reality at all because I had to start them by hand. >>>>> >>>>> I tried "ceph mgr fail" and even a complete shutdown of the

everything >>>>> during a maintenance window. Didn't change anything. >>>>> >>>>> Could you help me? To be honest I'm still rather new to Ceph

would <mailto:widhalmt@widhalm.or.at> <mailto:ceph-users@ceph.io> <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users@ceph.io> <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users@ceph.io> <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users@ceph.io> <mailto:ceph-users-leave@ceph.io>

Okay, thanks for verifying that bit, sorry to have gone about it so long. I guess we could look at connection issues next. I wrote a short python script that  tries to connect to hosts using asyncssh closely to how cephadm does it ( https://github.com/adk3798/testing_scripts/blob/main/asyncssh-connect.py). Maybe if you try that within a shell it will give some insights on if these connections, specifically through asyncssh using the keys/settings as cephadm does, is working alright. I used it in a shell with --no-hosts from the host with the active mgr to try and be as close to internal cephadm operations as possible. connect.py is the same as the linked file on github here. [root@vm-00 ~]# cephadm shell --no-hosts --mount connect.py Inferring fsid 9dcee730-f32f-11ed-ba89-52540033ae03 Using recent ceph image quay.io/adk3798/ceph@sha256:7168a3334fa16cd9d091462327c73f20548f113f9f404a316322aa4779a7639c [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# python3 -m pip install asyncssh WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Collecting asyncssh    Using cached https://files.pythonhosted.org/packages/1e/9f/ad61867b12823f6e2c0ef2b80a704… Requirement already satisfied: cryptography>=3.1 in /usr/lib64/python3.6/site-packages (from asyncssh) Collecting typing-extensions>=3.6 (from asyncssh)    Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fa… Requirement already satisfied: six>=1.4.1 in /usr/lib/python3.6/site-packages (from cryptography>=3.1->asyncssh) Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/lib64/python3.6/site-packages (from cryptography>=3.1->asyncssh) Requirement already satisfied: pycparser in /usr/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=3.1->asyncssh) Installing collected packages: typing-extensions, asyncssh Successfully installed asyncssh-2.13.1 typing-extensions-4.1.1 [ceph: root@vm-00 /]# ceph config-key get mgr/cephadm/ssh_identity_key > identity_key [ceph: root@vm-00 /]# ceph config-key get mgr/cephadm/ssh_identity_pub > identity_pub [ceph: root@vm-00 /]# ceph cephadm get-ssh-config > ssh_config [ceph: root@vm-00 /]# ceph orch host ls HOST   ADDR             LABELS  STATUS vm-00  192.168.122.121  _admin vm-01  192.168.122.209 vm-02  192.168.122.128 3 hosts in cluster [ceph: root@vm-00 /]# python3 /mnt/connect.py connect --address 192.168.122.209 --priv-key-file identity_key --pub-key-file identity_pub --ssh-config-file ssh_config return code: 0 stdout: stderr: [ceph: root@vm-00 /]# The output won't look nice in an email, but basically it was just installing the asyncssh library, gathering the ssh keys/config from cephadm, picking an IP from the "ceph orch host ls" output and then running the script. By default it just runs "true" on the host, so in a success case it's just return code 0 and no output, but that could obviously be changed by modifying the python script to do something else for the cmd. If it fails, you get a traceback [ceph: root@vm-00 /]# python3 /mnt/connect.py connect --address 192.168.122.201 --priv-key-file identity_key --pub-key-file identity_pub --ssh-config-file ssh_config Traceback (most recent call last):    File "/mnt/connect.py", line 100, in <module>      main()    File "/mnt/connect.py", line 93, in main      r = args.func(args)    File "/mnt/connect.py", line 8, in try_connection      async_run(_connect(args))    File "/mnt/connect.py", line 14, in async_run      return loop.run_until_complete(coro)    File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete      return future.result()    File "/mnt/connect.py", line 27, in _connect      preferred_auth=['publickey'], options=ssh_options)    File "/usr/local/lib/python3.6/site-packages/asyncssh/connection.py", line 8045, in connect      timeout=new_options.connect_timeout)    File "/usr/lib64/python3.6/asyncio/tasks.py", line 358, in wait_for      return fut.result()    File "/usr/local/lib/python3.6/site-packages/asyncssh/connection.py", line 432, in _connect      flags=flags, local_addr=local_addr)    File "/usr/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection      raise exceptions[0]    File "/usr/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection      yield from self.sock_connect(sock, address)    File "/usr/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect      return (yield from fut)    File "/usr/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb      raise OSError(err, 'Connect call failed %s' % (address,)) OSError: [Errno 113] Connect call failed ('192.168.122.201', 22) By trying this for connecting to each host in the cluster, given how close it is to how cephadm is operating, it should help verify with relative certainty if this is connection related or not. Will add the important bit that I'm using the user "root". If you're using a non-root user, the script takes a "--user" option. On Mon, May 15, 2023 at 3:36 PM Thomas Widhalm &lt;widhalmt(a)widhalm.or.at&gt; wrote: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi, I'm in the process of upgrading my cluster from 17.2.5 to 17.2.6 but the following problem existed when I was still everywhere on 17.2.5 . I had a major issue in my cluster which could be solved with a lot of your help and even more trial and error. Right now it seems that most is already fixed but I can't rule out that there's still some problem hidden. The very issue I'm asking about started during the repair. When I want to orchestrate the cluster, it logs the command but it doesn't do anything. No matter if I use ceph dashboard or "ceph orch" in "cephadm shell". I don't get any error message when I try to deploy new services, redeploy them etc. The log only says "scheduled" and that's it. Same when I change placement rules. Usually I use tags. But since they don't work anymore, too, I tried host and umanaged. No success. The only way I can actually start and stop containers is via systemctl from the host itself. When I run "ceph orch ls" or "ceph orch ps" I see services I deployed for testing being deleted (for weeks now). Ans especially a lot of old MDS are listed as "error" or "starting". The list doesn't match reality at all because I had to start them by hand. I tried "ceph mgr fail" and even a complete shutdown of the whole cluster with all nodes including all mgs, mds even osd - everything during a maintenance window. Didn't change anything. Could you help me? To be honest I'm still rather new to Ceph and since I didn't find anything in the logs that caught my eye I would be thankful for hints how to debug. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

To completely rule out hung processes, I managed to get another short shutdown. Now I'm seeing lots of: mgr.server handle_open ignoring open from mds.mds01.ceph01.usujbi v2:192.168.23.61:6800/2922006253; not ready for session (expect reconnect) mgr finish mon failed to return metadata for mds.mds01.ceph02.otvipq: (2) No such file or directory log lines. Seems like it now realises that some of these informations are stale. But it looks like it's just waiting for it to come back and not do anything about it. On 04.05.23 14:48, Eugen Block wrote: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

what does specifically `ceph log last 200 debug cephadm` spit out? The log lines you've posted so far I don't think are generated by the orchestrator so curious what the last actions it took was (and how long ago). On Thu, May 4, 2023 at 10:35 AM Thomas Widhalm &lt;widhalmt(a)widhalm.or.at <mailto:widhalmt@widhalm.or.at>> wrote:     To completely rule out hung processes, I managed to get another short     shutdown.     Now I'm seeing lots of:     mgr.server handle_open ignoring open from mds.mds01.ceph01.usujbi     v2:192.168.23.61:6800/2922006253     <http://192.168.23.61:6800/2922006253>; not ready for session     (expect reconnect)     mgr finish mon failed to return metadata for mds.mds01.ceph02.otvipq:     (2) No such file or directory     log lines. Seems like it now realises that some of these informations     are stale. But it looks like it's just waiting for it to come back and     not do anything about it.     On 04.05.23 14:48, Eugen Block wrote:     it's     between a actually     <mailto:widhalmt@widhalm.or.at>>:     but     17.2.5 .     lot of     most     problem repair. but it     orch"     deploy and     tags. But     is via     deployed     of old everything     since     <mailto:widhalmt@widhalm.or.at>     <mailto:ceph-users@ceph.io>     <mailto:ceph-users-leave@ceph.io>     _______________________________________________     ceph-users mailing list -- ceph-users(a)ceph.io     <mailto:ceph-users@ceph.io>     To unsubscribe send an email to ceph-users-leave(a)ceph.io     <mailto:ceph-users-leave@ceph.io>