Thank you Adam for your response, I tried all your comments and the troubleshooting link you sent. From the Quincy mgrs containers, they can ssh into all other Pacific nodes successfully by running the exact command in the log output and vice versa. Here are some debug logs from the cephadm while updating: 2023-04-12T11:35:56.260958+0000 mgr.host8.jukgqm (mgr.4468627) 103 : cephadm [DBG] Opening connection to cephadmin(a)x.x.x.x with ssh options '-F /tmp/cephadm-conf-2bbfubub -i /tmp/cephadm-identity-7x2m8gvr' 2023-04-12T11:35:56.525091+0000 mgr.host8.jukgqm (mgr.4468627) 144 : cephadm [DBG] _run_cephadm : command = ls 2023-04-12T11:35:56.525406+0000 mgr.host8.jukgqm (mgr.4468627) 145 : cephadm [DBG] _run_cephadm : args = [] 2023-04-12T11:35:56.525571+0000 mgr.host8.jukgqm (mgr.4468627) 146 : cephadm [DBG] mon container image my-private-repo/quay-io/ceph/ceph@sha256 :1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add 2023-04-12T11:35:56.525619+0000 mgr.host8.jukgqm (mgr.4468627) 147 : cephadm [DBG] args: --image my-private-repo/quay-io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add ls 2023-04-12T11:35:56.525738+0000 mgr.host8.jukgqm (mgr.4468627) 148 : cephadm [DBG] Running command: sudo which python3 2023-04-12T11:35:56.534227+0000 mgr.host8.jukgqm (mgr.4468627) 149 : cephadm [DBG] Connection to host1 failed. Process exited with non-zero exit status 3 2023-04-12T11:35:56.534275+0000 mgr.host8.jukgqm (mgr.4468627) 150 : cephadm [DBG] _reset_con close host1 2023-04-12T11:35:56.540135+0000 mgr.host8.jukgqm (mgr.4468627) 158 : cephadm [DBG] Host "host1" marked as offline. Skipping gather facts refresh 2023-04-12T11:35:56.540178+0000 mgr.host8.jukgqm (mgr.4468627) 159 : cephadm [DBG] Host "host1" marked as offline. Skipping network refresh 2023-04-12T11:35:56.540408+0000 mgr.host8.jukgqm (mgr.4468627) 160 : cephadm [DBG] Host "host1" marked as offline. Skipping device refresh 2023-04-12T11:35:56.540490+0000 mgr.host8.jukgqm (mgr.4468627) 161 : cephadm [DBG] Host "host1" marked as offline. Skipping osdspec preview refresh 2023-04-12T11:35:56.540527+0000 mgr.host8.jukgqm (mgr.4468627) 162 : cephadm [DBG] Host "host1" marked as offline. Skipping autotune 2023-04-12T11:35:56.540978+0000 mgr.host8.jukgqm (mgr.4468627) 163 : cephadm [DBG] Connection to host1 failed. Process exited with non-zero exit status 3 2023-04-12T11:35:56.796966+0000 mgr.host8.jukgqm (mgr.4468627) 728 : cephadm [ERR] Upgrade: Paused due to UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host host1 at addr (x.x.x.x) As I can see here, it turns out sudo is added to the code to be able to continue: https://github.com/ceph/ceph/blob/v17.2.5/src/pybind/mgr/cephadm/ssh.py#L143 I cannot privilege the cephadmin user to run sudo commands for some policy reasons, could this be the root cause of the issue? Best regards, Reza On Thu, 6 Apr 2023 at 14:59, Adam King &lt;adking(a)redhat.com&gt; wrote: > Does "ceph health detail" give any insight into what the unexpected > exception was? If not, I'm pretty confident some traceback would end up > being logged. Could maybe still grab it with "ceph log last 200 info > cephadm" if not a lot else has happened. Also, probably need to find out if > the check-host is failing due to the check on the host actually failing or > failing to connect to the host. Could try putting a copy of the cephadm > binary on one and running "cephadm check-host --expect-hostname <hostname>" > where the hostname is the name cephadm knows the host by. If that's not an > issue I'd expect it's a connection thing. Could maybe try going through > https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors > <https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors>. > Cephadm changed the backend ssh library from pacific to quincy due to the > one used in pacific no longer being supported so it's possible some general > ssh error has popped up in your env as a result. > > On Thu, Apr 6, 2023 at 8:38 AM Reza Bakhshayeshi &lt;reza.b2008(a)gmail.com&gt; > wrote: > >> Hi all, >> >> I have a problem regarding upgrading Ceph cluster from Pacific to Quincy >> version with cephadm. I have successfully upgraded the cluster to the >> latest Pacific (16.2.11). But when I run the following command to upgrade >> the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process >> stops >> with "Unexpected error". (everything is on a private network) >> >> ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 >> >> I also tried the 17.2.4 version. >> >> cephadm fails to check the hosts' status and marks them as offline: >> >> cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) >> 5782 >> : cephadm [DBG] host host4 (x.x.x.x) failed check >> cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) >> 5783 >> : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh >> cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) >> 5784 >> : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts >> refresh >> cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) >> 5785 >> : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh >> cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) >> 5786 >> : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh >> cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) >> 5787 >> : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview >> refresh >> cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) >> 5788 >> : cephadm [DBG] Host "host4" marked as offline. Skipping autotune >> cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : >> cluster >> [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: >> failed >> due to an unexpected exception >> cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : >> cluster >> [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check >> cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : >> cluster >> [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host >> host7. Process exited with non-zero exit status 3 >> cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : >> cluster >> [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host >> host2. Process exited with non-zero exit status 3 >> cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : >> cluster >> [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host >> host8. Process exited with non-zero exit status 3 >> cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : >> cluster >> [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host >> host4. Process exited with non-zero exit status 3 >> cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : >> cluster >> [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host >> host3. Process exited with non-zero exit status 3 >> >> and here are some outputs of the commands: >> >> [root@host8 ~]# ceph -s >> cluster: >> id: xxx >> health: HEALTH_ERR >> 9 hosts fail cephadm check >> Upgrade: failed due to an unexpected exception >> >> services: >> mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) >> mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, >> host1.warjsr, host2.qyavjj >> mds: 1/1 daemons up, 3 standby >> osd: 37 osds: 37 up (since 8h), 37 in (since 3w) >> >> data: >> >> >> io: >> client: >> >> progress: >> Upgrade to 17.2.5 (0s) >> [............................] >> >> [root@host8 ~]# ceph orch upgrade status >> { >> "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 >> :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", >> "in_progress": true, >> "which": "Upgrading all daemon types on all hosts", >> "services_complete": [], >> "progress": "3/59 daemons upgraded", >> "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an >> unexpected exception", >> "is_paused": true >> } >> [root@host8 ~]# ceph cephadm check-host host7 >> check-host failed: >> Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. >> [root@host8 ~]# ceph versions >> { >> "mon": { >> "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >> pacific (stable)": 5 >> }, >> "mgr": { >> "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >> pacific (stable)": 1, >> "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) >> quincy (stable)": 3 >> }, >> "osd": { >> "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >> pacific (stable)": 37 >> }, >> "mds": { >> "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >> pacific (stable)": 4 >> }, >> "overall": { >> "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >> pacific (stable)": 47, >> "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) >> quincy (stable)": 3 >> } >> } >> >> The strange thing is I can rollback the cluster status by failing to >> not-upgraded mgr like this: >> >> ceph mgr fail >> ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 >> >> Would you happen to have any idea about this? >> >> Best regards, >> Reza >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >>

Hi Adam, I'm still struggling with this issue. I also checked it one more time with newer versions, upgrading the cluster from 16.2.11 to 16.2.12 was successful but from 16.2.12 to 17.2.6 failed again with the same ssh errors (I checked https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors a couple of times and all keys/access are fine). [root@host1 ~]# ceph health detail HEALTH_ERR Upgrade: Failed to connect to host host2 at addr (x.x.x.x) [ERR] UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host host2 at addr (x.x.x.x) SSH connection failed to host2 at addr (x.x.x.x): Host(s) were marked offline: {'host2', 'host6', 'host9', 'host4', 'host3', 'host5', 'host1', 'host7', 'host8'} The interesting thing is that always (total number of mgrs) - 1 is upgraded, If I provision 5 MGRs then 4 of them, and for 3, 2 of them! As long as I'm in an internal environment, I also checked the process with Quincy cephadm binary file. FYI I'm using stretch mode on this cluster. I don't understand why Quincy MGRs cannot ssh into Pacific nodes, if you have any more hints I would be really glad to hear. Best regards, Reza On Wed, 12 Apr 2023 at 17:18, Adam King &lt;adking(a)redhat.com&gt; wrote: > Ah, okay. Someone else had opened an issue about the same thing after > the 17.2.5 release I believe. It's changed in 17.2.6 at least to only use > sudo for non-root users > https://github.com/ceph/ceph/blob/v17.2.6/src/pybind/mgr/cephadm/ssh.py#L14…. > But it looks like you're also using a non-root user anyway. We've required > passwordless sudo access for custom ssh users for a long time I think (e.g. > it's in pacific docs > https://docs.ceph.com/en/pacific/cephadm/install/#further-information-about…, > see the point on "--ssh-user"). Did this actually work for you before in > pacific with a non-root user that doesn't have sudo privileges? I had > assumed that had never worked. > > On Wed, Apr 12, 2023 at 10:38 AM Reza Bakhshayeshi &lt;reza.b2008(a)gmail.com&gt; > wrote: > >> Thank you Adam for your response, >> >> I tried all your comments and the troubleshooting link you sent. From >> the Quincy mgrs containers, they can ssh into all other Pacific nodes >> successfully by running the exact command in the log output and vice versa. >> >> Here are some debug logs from the cephadm while updating: >> >> 2023-04-12T11:35:56.260958+0000 mgr.host8.jukgqm (mgr.4468627) 103 : >> cephadm [DBG] Opening connection to cephadmin(a)x.x.x.x with ssh options >> '-F /tmp/cephadm-conf-2bbfubub -i /tmp/cephadm-identity-7x2m8gvr' >> 2023-04-12T11:35:56.525091+0000 mgr.host8.jukgqm (mgr.4468627) 144 : >> cephadm [DBG] _run_cephadm : command = ls >> 2023-04-12T11:35:56.525406+0000 mgr.host8.jukgqm (mgr.4468627) 145 : >> cephadm [DBG] _run_cephadm : args = [] >> 2023-04-12T11:35:56.525571+0000 mgr.host8.jukgqm (mgr.4468627) 146 : >> cephadm [DBG] mon container image my-private-repo/quay-io/ceph/ceph@sha256 >> :1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add >> 2023-04-12T11:35:56.525619+0000 mgr.host8.jukgqm (mgr.4468627) 147 : >> cephadm [DBG] args: --image my-private-repo/quay-io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add >> ls >> 2023-04-12T11:35:56.525738+0000 mgr.host8.jukgqm (mgr.4468627) 148 : >> cephadm [DBG] Running command: sudo which python3 >> 2023-04-12T11:35:56.534227+0000 mgr.host8.jukgqm (mgr.4468627) 149 : >> cephadm [DBG] Connection to host1 failed. Process exited with non-zero exit >> status 3 >> 2023-04-12T11:35:56.534275+0000 mgr.host8.jukgqm (mgr.4468627) 150 : >> cephadm [DBG] _reset_con close host1 >> 2023-04-12T11:35:56.540135+0000 mgr.host8.jukgqm (mgr.4468627) 158 : >> cephadm [DBG] Host "host1" marked as offline. Skipping gather facts refresh >> 2023-04-12T11:35:56.540178+0000 mgr.host8.jukgqm (mgr.4468627) 159 : >> cephadm [DBG] Host "host1" marked as offline. Skipping network refresh >> 2023-04-12T11:35:56.540408+0000 mgr.host8.jukgqm (mgr.4468627) 160 : >> cephadm [DBG] Host "host1" marked as offline. Skipping device refresh >> 2023-04-12T11:35:56.540490+0000 mgr.host8.jukgqm (mgr.4468627) 161 : >> cephadm [DBG] Host "host1" marked as offline. Skipping osdspec preview >> refresh >> 2023-04-12T11:35:56.540527+0000 mgr.host8.jukgqm (mgr.4468627) 162 : >> cephadm [DBG] Host "host1" marked as offline. Skipping autotune >> 2023-04-12T11:35:56.540978+0000 mgr.host8.jukgqm (mgr.4468627) 163 : >> cephadm [DBG] Connection to host1 failed. Process exited with non-zero exit >> status 3 >> 2023-04-12T11:35:56.796966+0000 mgr.host8.jukgqm (mgr.4468627) 728 : >> cephadm [ERR] Upgrade: Paused due to UPGRADE_OFFLINE_HOST: Upgrade: Failed >> to connect to host host1 at addr (x.x.x.x) >> >> As I can see here, it turns out sudo is added to the code to be able to >> continue: >> >> >> https://github.com/ceph/ceph/blob/v17.2.5/src/pybind/mgr/cephadm/ssh.py#L143 >> >> I cannot privilege the cephadmin user to run sudo commands for some >> policy reasons, could this be the root cause of the issue? >> >> Best regards, >> Reza >> >> On Thu, 6 Apr 2023 at 14:59, Adam King &lt;adking(a)redhat.com&gt; wrote: >> >>> Does "ceph health detail" give any insight into what the unexpected >>> exception was? If not, I'm pretty confident some traceback would end up >>> being logged. Could maybe still grab it with "ceph log last 200 info >>> cephadm" if not a lot else has happened. Also, probably need to find out if >>> the check-host is failing due to the check on the host actually failing or >>> failing to connect to the host. Could try putting a copy of the cephadm >>> binary on one and running "cephadm check-host --expect-hostname <hostname>" >>> where the hostname is the name cephadm knows the host by. If that's not an >>> issue I'd expect it's a connection thing. Could maybe try going through >>> https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors >>> <https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors>. >>> Cephadm changed the backend ssh library from pacific to quincy due to the >>> one used in pacific no longer being supported so it's possible some general >>> ssh error has popped up in your env as a result. >>> >>> On Thu, Apr 6, 2023 at 8:38 AM Reza Bakhshayeshi &lt;reza.b2008(a)gmail.com&gt; >>> wrote: >>> >>>> Hi all, >>>> >>>> I have a problem regarding upgrading Ceph cluster from Pacific to >>>> Quincy >>>> version with cephadm. I have successfully upgraded the cluster to the >>>> latest Pacific (16.2.11). But when I run the following command to >>>> upgrade >>>> the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process >>>> stops >>>> with "Unexpected error". (everything is on a private network) >>>> >>>> ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 >>>> >>>> I also tried the 17.2.4 version. >>>> >>>> cephadm fails to check the hosts' status and marks them as offline: >>>> >>>> cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) >>>> 5782 >>>> : cephadm [DBG] host host4 (x.x.x.x) failed check >>>> cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) >>>> 5783 >>>> : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh >>>> cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) >>>> 5784 >>>> : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts >>>> refresh >>>> cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) >>>> 5785 >>>> : cephadm [DBG] Host "host4" marked as offline. Skipping network >>>> refresh >>>> cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) >>>> 5786 >>>> : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh >>>> cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) >>>> 5787 >>>> : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec >>>> preview >>>> refresh >>>> cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) >>>> 5788 >>>> : cephadm [DBG] Host "host4" marked as offline. Skipping autotune >>>> cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : >>>> cluster >>>> [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: >>>> failed >>>> due to an unexpected exception >>>> cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : >>>> cluster >>>> [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check >>>> cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : >>>> cluster >>>> [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote >>>> host >>>> host7. Process exited with non-zero exit status 3 >>>> cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : >>>> cluster >>>> [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote >>>> host >>>> host2. Process exited with non-zero exit status 3 >>>> cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : >>>> cluster >>>> [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote >>>> host >>>> host8. Process exited with non-zero exit status 3 >>>> cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : >>>> cluster >>>> [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote >>>> host >>>> host4. Process exited with non-zero exit status 3 >>>> cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : >>>> cluster >>>> [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote >>>> host >>>> host3. Process exited with non-zero exit status 3 >>>> >>>> and here are some outputs of the commands: >>>> >>>> [root@host8 ~]# ceph -s >>>> cluster: >>>> id: xxx >>>> health: HEALTH_ERR >>>> 9 hosts fail cephadm check >>>> Upgrade: failed due to an unexpected exception >>>> >>>> services: >>>> mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) >>>> mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, >>>> host1.warjsr, host2.qyavjj >>>> mds: 1/1 daemons up, 3 standby >>>> osd: 37 osds: 37 up (since 8h), 37 in (since 3w) >>>> >>>> data: >>>> >>>> >>>> io: >>>> client: >>>> >>>> progress: >>>> Upgrade to 17.2.5 (0s) >>>> [............................] >>>> >>>> [root@host8 ~]# ceph orch upgrade status >>>> { >>>> "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 >>>> :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", >>>> "in_progress": true, >>>> "which": "Upgrading all daemon types on all hosts", >>>> "services_complete": [], >>>> "progress": "3/59 daemons upgraded", >>>> "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an >>>> unexpected exception", >>>> "is_paused": true >>>> } >>>> [root@host8 ~]# ceph cephadm check-host host7 >>>> check-host failed: >>>> Host 'host7' not found. Use 'ceph orch host ls' to see all managed >>>> hosts. >>>> [root@host8 ~]# ceph versions >>>> { >>>> "mon": { >>>> "ceph version 16.2.11 >>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>> pacific (stable)": 5 >>>> }, >>>> "mgr": { >>>> "ceph version 16.2.11 >>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>> pacific (stable)": 1, >>>> "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) >>>> quincy (stable)": 3 >>>> }, >>>> "osd": { >>>> "ceph version 16.2.11 >>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>> pacific (stable)": 37 >>>> }, >>>> "mds": { >>>> "ceph version 16.2.11 >>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>> pacific (stable)": 4 >>>> }, >>>> "overall": { >>>> "ceph version 16.2.11 >>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>> pacific (stable)": 47, >>>> "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) >>>> quincy (stable)": 3 >>>> } >>>> } >>>> >>>> The strange thing is I can rollback the cluster status by failing to >>>> not-upgraded mgr like this: >>>> >>>> ceph mgr fail >>>> ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 >>>> >>>> Would you happen to have any idea about this? >>>> >>>> Best regards, >>>> Reza >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>>

Thank you. I don't see any more errors rather than: 2023-05-04T15:07:38.003+0000 7ff96cbe0700 0 log_channel(cephadm) log [DBG] : Running command: sudo which python3 2023-05-04T15:07:38.025+0000 7ff96cbe0700 0 log_channel(cephadm) log [DBG] : Connection to host1 failed. Process exited with non-zero exit status 3 2023-05-04T15:07:38.025+0000 7ff96cbe0700 0 log_channel(cephadm) log [DBG] : _reset_con close host1 What is the best way to safely change the cephadm user to root for the existing cluster? It seems "ceph cephadm set-ssh-config" is not effective (BTW, my cephadmin user can run "sudo which python3" without prompting password on other hosts now, but nothing has been solved) Best regards, Reza On Tue, 2 May 2023 at 19:00, Adam King &lt;adking(a)redhat.com&gt; wrote: > The number of mgr daemons thing is expected. The way it works is it first > upgrades all the standby mgrs (which will be all but one) and then fails > over so the previously active mgr can be upgraded as well. After that > failover is when it's first actually running the newer cephadm code, which > is when you're hitting this issue. Are the logs still saying something > similar about how "sudo which python3" is failing? I'm thinking this > might just be a general issue with the user being used not having > passwordless sudo access, that sort of accidentally working in pacific, but > now not working any more in quincy. If the log lines confirm the same, we > might have to work on something in order to handle this case (making the > sudo optional somehow). As mentioned in the previous email, that setup > wasn't intended to be supported even in pacific, although if it did work, > we could bring something in to make it usable in quincy onward as well. > > On Tue, May 2, 2023 at 10:58 AM Reza Bakhshayeshi &lt;reza.b2008(a)gmail.com&gt; > wrote: > >> Hi Adam, >> >> I'm still struggling with this issue. I also checked it one more time >> with newer versions, upgrading the cluster from 16.2.11 to 16.2.12 was >> successful but from 16.2.12 to 17.2.6 failed again with the same ssh errors >> (I checked >> https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors a >> couple of times and all keys/access are fine). >> >> [root@host1 ~]# ceph health detail >> HEALTH_ERR Upgrade: Failed to connect to host host2 at addr (x.x.x.x) >> [ERR] UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host host2 at >> addr (x.x.x.x) >> SSH connection failed to host2 at addr (x.x.x.x): Host(s) were >> marked offline: {'host2', 'host6', 'host9', 'host4', 'host3', 'host5', >> 'host1', 'host7', 'host8'} >> >> The interesting thing is that always (total number of mgrs) - 1 is >> upgraded, If I provision 5 MGRs then 4 of them, and for 3, 2 of them! >> >> As long as I'm in an internal environment, I also checked the process >> with Quincy cephadm binary file. FYI I'm using stretch mode on this cluster. >> >> I don't understand why Quincy MGRs cannot ssh into Pacific nodes, if you >> have any more hints I would be really glad to hear. >> >> Best regards, >> Reza >> >> >> >> On Wed, 12 Apr 2023 at 17:18, Adam King &lt;adking(a)redhat.com&gt; wrote: >> >>> Ah, okay. Someone else had opened an issue about the same thing after >>> the 17.2.5 release I believe. It's changed in 17.2.6 at least to only use >>> sudo for non-root users >>> https://github.com/ceph/ceph/blob/v17.2.6/src/pybind/mgr/cephadm/ssh.py#L14…. >>> But it looks like you're also using a non-root user anyway. We've required >>> passwordless sudo access for custom ssh users for a long time I think (e.g. >>> it's in pacific docs >>> https://docs.ceph.com/en/pacific/cephadm/install/#further-information-about…, >>> see the point on "--ssh-user"). Did this actually work for you before in >>> pacific with a non-root user that doesn't have sudo privileges? I had >>> assumed that had never worked. >>> >>> On Wed, Apr 12, 2023 at 10:38 AM Reza Bakhshayeshi < >>> reza.b2008(a)gmail.com&gt; wrote: >>> >>>> Thank you Adam for your response, >>>> >>>> I tried all your comments and the troubleshooting link you sent. From >>>> the Quincy mgrs containers, they can ssh into all other Pacific nodes >>>> successfully by running the exact command in the log output and vice versa. >>>> >>>> Here are some debug logs from the cephadm while updating: >>>> >>>> 2023-04-12T11:35:56.260958+0000 mgr.host8.jukgqm (mgr.4468627) 103 : >>>> cephadm [DBG] Opening connection to cephadmin(a)x.x.x.x with ssh >>>> options '-F /tmp/cephadm-conf-2bbfubub -i /tmp/cephadm-identity-7x2m8gvr' >>>> 2023-04-12T11:35:56.525091+0000 mgr.host8.jukgqm (mgr.4468627) 144 : >>>> cephadm [DBG] _run_cephadm : command = ls >>>> 2023-04-12T11:35:56.525406+0000 mgr.host8.jukgqm (mgr.4468627) 145 : >>>> cephadm [DBG] _run_cephadm : args = [] >>>> 2023-04-12T11:35:56.525571+0000 mgr.host8.jukgqm (mgr.4468627) 146 : >>>> cephadm [DBG] mon container image my-private-repo/quay-io/ceph/ceph@sha256 >>>> :1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add >>>> 2023-04-12T11:35:56.525619+0000 mgr.host8.jukgqm (mgr.4468627) 147 : >>>> cephadm [DBG] args: --image my-private-repo/quay-io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add >>>> ls >>>> 2023-04-12T11:35:56.525738+0000 mgr.host8.jukgqm (mgr.4468627) 148 : >>>> cephadm [DBG] Running command: sudo which python3 >>>> 2023-04-12T11:35:56.534227+0000 mgr.host8.jukgqm (mgr.4468627) 149 : >>>> cephadm [DBG] Connection to host1 failed. Process exited with non-zero exit >>>> status 3 >>>> 2023-04-12T11:35:56.534275+0000 mgr.host8.jukgqm (mgr.4468627) 150 : >>>> cephadm [DBG] _reset_con close host1 >>>> 2023-04-12T11:35:56.540135+0000 mgr.host8.jukgqm (mgr.4468627) 158 : >>>> cephadm [DBG] Host "host1" marked as offline. Skipping gather facts refresh >>>> 2023-04-12T11:35:56.540178+0000 mgr.host8.jukgqm (mgr.4468627) 159 : >>>> cephadm [DBG] Host "host1" marked as offline. Skipping network refresh >>>> 2023-04-12T11:35:56.540408+0000 mgr.host8.jukgqm (mgr.4468627) 160 : >>>> cephadm [DBG] Host "host1" marked as offline. Skipping device refresh >>>> 2023-04-12T11:35:56.540490+0000 mgr.host8.jukgqm (mgr.4468627) 161 : >>>> cephadm [DBG] Host "host1" marked as offline. Skipping osdspec preview >>>> refresh >>>> 2023-04-12T11:35:56.540527+0000 mgr.host8.jukgqm (mgr.4468627) 162 : >>>> cephadm [DBG] Host "host1" marked as offline. Skipping autotune >>>> 2023-04-12T11:35:56.540978+0000 mgr.host8.jukgqm (mgr.4468627) 163 : >>>> cephadm [DBG] Connection to host1 failed. Process exited with non-zero exit >>>> status 3 >>>> 2023-04-12T11:35:56.796966+0000 mgr.host8.jukgqm (mgr.4468627) 728 : >>>> cephadm [ERR] Upgrade: Paused due to UPGRADE_OFFLINE_HOST: Upgrade: Failed >>>> to connect to host host1 at addr (x.x.x.x) >>>> >>>> As I can see here, it turns out sudo is added to the code to be able >>>> to continue: >>>> >>>> >>>> https://github.com/ceph/ceph/blob/v17.2.5/src/pybind/mgr/cephadm/ssh.py#L143 >>>> >>>> I cannot privilege the cephadmin user to run sudo commands for some >>>> policy reasons, could this be the root cause of the issue? >>>> >>>> Best regards, >>>> Reza >>>> >>>> On Thu, 6 Apr 2023 at 14:59, Adam King &lt;adking(a)redhat.com&gt; wrote: >>>> >>>>> Does "ceph health detail" give any insight into what the unexpected >>>>> exception was? If not, I'm pretty confident some traceback would end up >>>>> being logged. Could maybe still grab it with "ceph log last 200 info >>>>> cephadm" if not a lot else has happened. Also, probably need to find out if >>>>> the check-host is failing due to the check on the host actually failing or >>>>> failing to connect to the host. Could try putting a copy of the cephadm >>>>> binary on one and running "cephadm check-host --expect-hostname <hostname>" >>>>> where the hostname is the name cephadm knows the host by. If that's not an >>>>> issue I'd expect it's a connection thing. Could maybe try going through >>>>> https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors >>>>> <https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors>. >>>>> Cephadm changed the backend ssh library from pacific to quincy due to the >>>>> one used in pacific no longer being supported so it's possible some general >>>>> ssh error has popped up in your env as a result. >>>>> >>>>> On Thu, Apr 6, 2023 at 8:38 AM Reza Bakhshayeshi < >>>>> reza.b2008(a)gmail.com&gt; wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I have a problem regarding upgrading Ceph cluster from Pacific to >>>>>> Quincy >>>>>> version with cephadm. I have successfully upgraded the cluster to the >>>>>> latest Pacific (16.2.11). But when I run the following command to >>>>>> upgrade >>>>>> the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process >>>>>> stops >>>>>> with "Unexpected error". (everything is on a private network) >>>>>> >>>>>> ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 >>>>>> >>>>>> I also tried the 17.2.4 version. >>>>>> >>>>>> cephadm fails to check the hosts' status and marks them as offline: >>>>>> >>>>>> cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd >>>>>> (mgr.4516356) 5782 >>>>>> : cephadm [DBG] host host4 (x.x.x.x) failed check >>>>>> cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd >>>>>> (mgr.4516356) 5783 >>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping daemon >>>>>> refresh >>>>>> cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd >>>>>> (mgr.4516356) 5784 >>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts >>>>>> refresh >>>>>> cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd >>>>>> (mgr.4516356) 5785 >>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping network >>>>>> refresh >>>>>> cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd >>>>>> (mgr.4516356) 5786 >>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping device >>>>>> refresh >>>>>> cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd >>>>>> (mgr.4516356) 5787 >>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec >>>>>> preview >>>>>> refresh >>>>>> cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd >>>>>> (mgr.4516356) 5788 >>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping autotune >>>>>> cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : >>>>>> cluster >>>>>> [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: >>>>>> failed >>>>>> due to an unexpected exception >>>>>> cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : >>>>>> cluster >>>>>> [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check >>>>>> cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : >>>>>> cluster >>>>>> [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote >>>>>> host >>>>>> host7. Process exited with non-zero exit status 3 >>>>>> cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : >>>>>> cluster >>>>>> [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote >>>>>> host >>>>>> host2. Process exited with non-zero exit status 3 >>>>>> cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : >>>>>> cluster >>>>>> [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote >>>>>> host >>>>>> host8. Process exited with non-zero exit status 3 >>>>>> cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : >>>>>> cluster >>>>>> [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote >>>>>> host >>>>>> host4. Process exited with non-zero exit status 3 >>>>>> cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : >>>>>> cluster >>>>>> [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote >>>>>> host >>>>>> host3. Process exited with non-zero exit status 3 >>>>>> >>>>>> and here are some outputs of the commands: >>>>>> >>>>>> [root@host8 ~]# ceph -s >>>>>> cluster: >>>>>> id: xxx >>>>>> health: HEALTH_ERR >>>>>> 9 hosts fail cephadm check >>>>>> Upgrade: failed due to an unexpected exception >>>>>> >>>>>> services: >>>>>> mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) >>>>>> mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, >>>>>> host1.warjsr, host2.qyavjj >>>>>> mds: 1/1 daemons up, 3 standby >>>>>> osd: 37 osds: 37 up (since 8h), 37 in (since 3w) >>>>>> >>>>>> data: >>>>>> >>>>>> >>>>>> io: >>>>>> client: >>>>>> >>>>>> progress: >>>>>> Upgrade to 17.2.5 (0s) >>>>>> [............................] >>>>>> >>>>>> [root@host8 ~]# ceph orch upgrade status >>>>>> { >>>>>> "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 >>>>>> :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", >>>>>> "in_progress": true, >>>>>> "which": "Upgrading all daemon types on all hosts", >>>>>> "services_complete": [], >>>>>> "progress": "3/59 daemons upgraded", >>>>>> "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an >>>>>> unexpected exception", >>>>>> "is_paused": true >>>>>> } >>>>>> [root@host8 ~]# ceph cephadm check-host host7 >>>>>> check-host failed: >>>>>> Host 'host7' not found. Use 'ceph orch host ls' to see all managed >>>>>> hosts. >>>>>> [root@host8 ~]# ceph versions >>>>>> { >>>>>> "mon": { >>>>>> "ceph version 16.2.11 >>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>> pacific (stable)": 5 >>>>>> }, >>>>>> "mgr": { >>>>>> "ceph version 16.2.11 >>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>> pacific (stable)": 1, >>>>>> "ceph version 17.2.5 >>>>>> (98318ae89f1a893a6ded3a640405cdbb33e08757) >>>>>> quincy (stable)": 3 >>>>>> }, >>>>>> "osd": { >>>>>> "ceph version 16.2.11 >>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>> pacific (stable)": 37 >>>>>> }, >>>>>> "mds": { >>>>>> "ceph version 16.2.11 >>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>> pacific (stable)": 4 >>>>>> }, >>>>>> "overall": { >>>>>> "ceph version 16.2.11 >>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>> pacific (stable)": 47, >>>>>> "ceph version 17.2.5 >>>>>> (98318ae89f1a893a6ded3a640405cdbb33e08757) >>>>>> quincy (stable)": 3 >>>>>> } >>>>>> } >>>>>> >>>>>> The strange thing is I can rollback the cluster status by failing to >>>>>> not-upgraded mgr like this: >>>>>> >>>>>> ceph mgr fail >>>>>> ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 >>>>>> >>>>>> Would you happen to have any idea about this? >>>>>> >>>>>> Best regards, >>>>>> Reza >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>> >>>>>>