I'm trying to upgrade our 3-monitor cluster from Centos 7 and Nautilus to Rocky 9 and Quincy. This has been a very slow process of upgrading one thing, running the cluster for a while, then upgrading the next thing. I first upgraded to the last Centos 7 and upgraded to Octopus. That worked fine. Then I was going to upgrade the OS to Rocky 9 while staying on Octopus, but then found out that Octopus is not available for Rocky 9. So I broke my own rule and upgraded one of the monitor (and manager) nodes to Rocky 9 and Pacific, then rejoined it to the cluster. That seemed to work just fine. Feeling bold, I upgraded the second monitor and manager node to Rocky 9 and Pacific. That also seemed to work fine, with the cluster showing all the monitors and managers running. But now, if I shut down the last "Octopus" monitor, the cluster becomes unresponsive. This only happens when I shut down the Octopus monitor. If I shut down one of the Pacific monitors, the cluster keeps responding with the expected: "HEALTH_WARN 1/3 mons down" and then goes back to normal when the monitor process is started again. Is this expected? What am I missing? Thanks for any pointers!
On Thu, Oct 26, 2023, 6:16 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
from Centos 7 and Nautilus to Rocky 9 and Quincy.
I hate to be the bearer of bad news here, but: https://docs.ceph.com/en/latest/releases/quincy/#upgrading-from-pre-octopus-... "You *must* first upgrade to Octopus (15.2.z) or Pacific (16.2.z) before upgrading to Quincy."
Is this expected? What am I missing? Thanks for any pointers!
Personally, what I would do in this situation is treat the 2 Quincy mons as permanently failed, and recover from the Nautilus one that's still working. That is: take a copy of the monmap, then purge the two Quincy mons that were prematurely upgraded, and finally rebuild those same 2 mons with Centos 7/Nautilus. Preferably, this is done in some kind of staging environment first for validation as you do not want to mess with that last mon at this point. Cheers, Tyler
On Thu, Oct 26, 2023 at 6:36 PM Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Thu, Oct 26, 2023, 6:16 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
from Centos 7 and Nautilus to Rocky 9 and Quincy.
I hate to be the bearer of bad news here, but: https://docs.ceph.com/en/latest/releases/quincy/#upgrading-from-pre-octopus-...
Also note that you are supposed to upgrade all of the mons first (strictly), and *then and only then* the mgrs. You may also be observing a symptom of this, and the upgraded mons may be OK. Unsure. Maybe someone else can comment on a Nautilus to Quincy jump.
Cheers, Tyler
Hi Tyler, Maybe you didn't read the full message, but in the message you will notice that I'm doing exactly that, and the problem just occurred when I was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The original goal was to move from Nautilus to Quincy, but I have gone to Octopus (no problems) and now to Pacific (problems). On Thu, Oct 26, 2023 at 3:36 PM Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Thu, Oct 26, 2023, 6:16 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
from Centos 7 and Nautilus to Rocky 9 and Quincy.
I hate to be the bearer of bad news here, but:
https://docs.ceph.com/en/latest/releases/quincy/#upgrading-from-pre-octopus-...
"You *must* first upgrade to Octopus (15.2.z) or Pacific (16.2.z) before upgrading to Quincy."
Is this expected? What am I missing? Thanks for any pointers!
Personally, what I would do in this situation is treat the 2 Quincy mons as permanently failed, and recover from the Nautilus one that's still working. That is: take a copy of the monmap, then purge the two Quincy mons that were prematurely upgraded, and finally rebuild those same 2 mons with Centos 7/Nautilus.
Preferably, this is done in some kind of staging environment first for validation as you do not want to mess with that last mon at this point.
Cheers, Tyler
On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Hi Tyler,
Maybe you didn't read the full message, but in the message you will notice that I'm doing exactly that, and the problem just occurred when I was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The original goal was to move from Nautilus to Quincy, but I have gone to Octopus (no problems) and now to Pacific (problems).
I did not, apologies -- though do see my second message about ordering mon/mgr ordering... When you say "the cluster becomes unresponsive" -- does the client I/O lock up, or do you mean that `ceph -s` and such hangs? May help to look to Pacific mons via the asok and see if they respond in such a state (and their status) if I/O is not locked up and you can afford to leave it in that state for a couple minutes: $ ceph daemon mon.name mon_status Cheers, Tyler
Oh, I meant that "ceph -s" just hangs. I didn't even try to look at the I/O. Maybe I can do that, but the "ceph -s" hang just freaked me out. Also, I know that the recommended order is mon->mgr->osd->mds->rgw, but when you run mgr on the same hardware as the monitors, it's hard to not upgrade both at the same time. Particularly if you're upgrading the whole machine at once. Here's where upgrading to the new container method will help a lot! FWIW, the managers seem to be running fine. I guess, if nothing else works, I can try going: Centos7+Octopus to Rocky8+Octopus to Rocky8+Pacific to Rocky9+Pacific to Rocky9+Quincy I was just hoping that I could skip the Rocky8 installation altogether... On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Hi Tyler,
Maybe you didn't read the full message, but in the message you will
notice that I'm doing exactly that, and the problem just occurred when I was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The original goal was to move from Nautilus to Quincy, but I have gone to Octopus (no problems) and now to Pacific (problems).
I did not, apologies -- though do see my second message about ordering mon/mgr ordering...
When you say "the cluster becomes unresponsive" -- does the client I/O lock up, or do you mean that `ceph -s` and such hangs?
May help to look to Pacific mons via the asok and see if they respond in such a state (and their status) if I/O is not locked up and you can afford to leave it in that state for a couple minutes: $ ceph daemon mon.name mon_status
Cheers, Tyler
On Thu, Oct 26, 2023, 8:11 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Oh, I meant that "ceph -s" just hangs. I didn't even try to look at the I/O. Maybe I can do that, but the "ceph -s" hang just freaked me out.
Also, I know that the recommended order is mon->mgr->osd->mds->rgw, but when you run mgr on the same hardware as the monitors, it's hard to not upgrade both at the same time. Particularly if you're upgrading the whole machine at once. Here's where upgrading to the new container method will help a lot! FWIW, the managers seem to be running fine.
I recently did something like this, so I understand that it's difficult. Most of my testing and prep-work was centered around exactly this problem, which was avoided by first upgrading mons/mgrs to an interim OS while remaining on Octopus -- solely for the purposes of opening an avenue from Octopus to Quincy separate from tbe OS upgrade. In my pre-prod resting, trying to upgrade the mons/mgrs without that middle step that allowed mgrs to be upgraded separately did result in `ceph -s` locking up. Client I/O remained non-impacted in this state though. Maybe look at which mgr is active and/or try stopping all but the Octopus mgr when stopping the mon as well? Cheers, Tyler
On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Hi Tyler,
Maybe you didn't read the full message, but in the message you will
notice that I'm doing exactly that, and the problem just occurred when I was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The original goal was to move from Nautilus to Quincy, but I have gone to Octopus (no problems) and now to Pacific (problems).
I did not, apologies -- though do see my second message about ordering mon/mgr ordering...
When you say "the cluster becomes unresponsive" -- does the client I/O lock up, or do you mean that `ceph -s` and such hangs?
May help to look to Pacific mons via the asok and see if they respond in such a state (and their status) if I/O is not locked up and you can afford to leave it in that state for a couple minutes: $ ceph daemon mon.name mon_status
Cheers, Tyler
What do the two Pacific monitors log when you stop the Octopus mon? And when all of them are up, which one is the leader? ceph mon stat --format json | jq Zitat von Tyler Stachecki <stachecki.tyler@gmail.com>:
On Thu, Oct 26, 2023, 8:11 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Oh, I meant that "ceph -s" just hangs. I didn't even try to look at the I/O. Maybe I can do that, but the "ceph -s" hang just freaked me out.
Also, I know that the recommended order is mon->mgr->osd->mds->rgw, but when you run mgr on the same hardware as the monitors, it's hard to not upgrade both at the same time. Particularly if you're upgrading the whole machine at once. Here's where upgrading to the new container method will help a lot! FWIW, the managers seem to be running fine.
I recently did something like this, so I understand that it's difficult. Most of my testing and prep-work was centered around exactly this problem, which was avoided by first upgrading mons/mgrs to an interim OS while remaining on Octopus -- solely for the purposes of opening an avenue from Octopus to Quincy separate from tbe OS upgrade.
In my pre-prod resting, trying to upgrade the mons/mgrs without that middle step that allowed mgrs to be upgraded separately did result in `ceph -s` locking up. Client I/O remained non-impacted in this state though.
Maybe look at which mgr is active and/or try stopping all but the Octopus mgr when stopping the mon as well?
Cheers, Tyler
On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Hi Tyler,
Maybe you didn't read the full message, but in the message you will
notice that I'm doing exactly that, and the problem just occurred when I was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The original goal was to move from Nautilus to Quincy, but I have gone to Octopus (no problems) and now to Pacific (problems).
I did not, apologies -- though do see my second message about ordering mon/mgr ordering...
When you say "the cluster becomes unresponsive" -- does the client I/O lock up, or do you mean that `ceph -s` and such hangs?
May help to look to Pacific mons via the asok and see if they respond in such a state (and their status) if I/O is not locked up and you can afford to leave it in that state for a couple minutes: $ ceph daemon mon.name mon_status
Cheers, Tyler
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think I figured it out. The problem was that my ceph.conf file only listed the first machine in mon_initial_members and in mon_host. I'm not sure why. I added the other monitors, restarted the monitors and the managers, and everything is now working as expected. I have now upgraded all the monitors and all the managers to Pacific and Rocky 9. Now on to the OSDs. Well, maybe next week... On Thu, Oct 26, 2023 at 5:37 PM Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Thu, Oct 26, 2023, 8:11 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Oh, I meant that "ceph -s" just hangs. I didn't even try to look at the I/O. Maybe I can do that, but the "ceph -s" hang just freaked me out.
Also, I know that the recommended order is mon->mgr->osd->mds->rgw, but when you run mgr on the same hardware as the monitors, it's hard to not upgrade both at the same time. Particularly if you're upgrading the whole machine at once. Here's where upgrading to the new container method will help a lot! FWIW, the managers seem to be running fine.
I recently did something like this, so I understand that it's difficult. Most of my testing and prep-work was centered around exactly this problem, which was avoided by first upgrading mons/mgrs to an interim OS while remaining on Octopus -- solely for the purposes of opening an avenue from Octopus to Quincy separate from tbe OS upgrade.
In my pre-prod resting, trying to upgrade the mons/mgrs without that middle step that allowed mgrs to be upgraded separately did result in `ceph -s` locking up. Client I/O remained non-impacted in this state though.
Maybe look at which mgr is active and/or try stopping all but the Octopus mgr when stopping the mon as well?
Cheers, Tyler
On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki < stachecki.tyler@gmail.com> wrote:
On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Hi Tyler,
Maybe you didn't read the full message, but in the message you will
notice that I'm doing exactly that, and the problem just occurred when I was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The original goal was to move from Nautilus to Quincy, but I have gone to Octopus (no problems) and now to Pacific (problems).
I did not, apologies -- though do see my second message about ordering mon/mgr ordering...
When you say "the cluster becomes unresponsive" -- does the client I/O lock up, or do you mean that `ceph -s` and such hangs?
May help to look to Pacific mons via the asok and see if they respond in such a state (and their status) if I/O is not locked up and you can afford to leave it in that state for a couple minutes: $ ceph daemon mon.name mon_status
Cheers, Tyler
Ah yes, this is a real classic ;-) I assume that after bootstrapping the first node no update to the ceph.conf was done. Anyway, good luck with the rest of the upgrade! Zitat von Jorge Garcia <jgarcia@soe.ucsc.edu>:
I think I figured it out. The problem was that my ceph.conf file only listed the first machine in mon_initial_members and in mon_host. I'm not sure why. I added the other monitors, restarted the monitors and the managers, and everything is now working as expected. I have now upgraded all the monitors and all the managers to Pacific and Rocky 9. Now on to the OSDs. Well, maybe next week...
On Thu, Oct 26, 2023 at 5:37 PM Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Thu, Oct 26, 2023, 8:11 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Oh, I meant that "ceph -s" just hangs. I didn't even try to look at the I/O. Maybe I can do that, but the "ceph -s" hang just freaked me out.
Also, I know that the recommended order is mon->mgr->osd->mds->rgw, but when you run mgr on the same hardware as the monitors, it's hard to not upgrade both at the same time. Particularly if you're upgrading the whole machine at once. Here's where upgrading to the new container method will help a lot! FWIW, the managers seem to be running fine.
I recently did something like this, so I understand that it's difficult. Most of my testing and prep-work was centered around exactly this problem, which was avoided by first upgrading mons/mgrs to an interim OS while remaining on Octopus -- solely for the purposes of opening an avenue from Octopus to Quincy separate from tbe OS upgrade.
In my pre-prod resting, trying to upgrade the mons/mgrs without that middle step that allowed mgrs to be upgraded separately did result in `ceph -s` locking up. Client I/O remained non-impacted in this state though.
Maybe look at which mgr is active and/or try stopping all but the Octopus mgr when stopping the mon as well?
Cheers, Tyler
On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki < stachecki.tyler@gmail.com> wrote:
On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
Hi Tyler,
Maybe you didn't read the full message, but in the message you will
notice that I'm doing exactly that, and the problem just occurred when I was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The original goal was to move from Nautilus to Quincy, but I have gone to Octopus (no problems) and now to Pacific (problems).
I did not, apologies -- though do see my second message about ordering mon/mgr ordering...
When you say "the cluster becomes unresponsive" -- does the client I/O lock up, or do you mean that `ceph -s` and such hangs?
May help to look to Pacific mons via the asok and see if they respond in such a state (and their status) if I/O is not locked up and you can afford to leave it in that state for a couple minutes: $ ceph daemon mon.name mon_status
Cheers, Tyler
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Jorge Garcia
-
Tyler Stachecki