Brief build outage to switch in upgraded shaman
I plan to switch in the upgraded shaman this weekend, because past experience shows builds slow regularly on Friday/Saturday. (It is currently running on a very old version of Ubuntu with a very old version of Postgres and a very old version of Python; we've resolved all that.) It will involve a shutdown of the existing shaman servers to 1) snapshot and populate the new server's database, and 2) change the IP of the server to the existing 1.shaman.ceph.com IP I've run through this process a few times in isolation, and I expect it to take no more than 10-15 minutes. I'll announce when it's imminent, either later this evening or tomorrow.
shaman is updated, of course it took longer than 10-15 minutes, but I believe everything is back in order. I paused the Jenkins queue and waited for any shaman users to exit before shuffling things around, so I think no builds should have been affected (only possibly delayed, but it looks like those are all auto-scheduled jobs, not PRs). Let me know here or in slack if you see something that looks like a problem with Jenkins and shaman. On 4/18/2025 5:35 PM, Dan Mick wrote:
I plan to switch in the upgraded shaman this weekend, because past experience shows builds slow regularly on Friday/Saturday. (It is currently running on a very old version of Ubuntu with a very old version of Postgres and a very old version of Python; we've resolved all that.)
It will involve a shutdown of the existing shaman servers to
1) snapshot and populate the new server's database, and 2) change the IP of the server to the existing 1.shaman.ceph.com IP
I've run through this process a few times in isolation, and I expect it to take no more than 10-15 minutes. I'll announce when it's imminent, either later this evening or tomorrow.
Thanks, Dan! On Sat, Apr 19, 2025 at 1:12 AM Dan Mick <dmick@redhat.com> wrote:
shaman is updated, of course it took longer than 10-15 minutes, but I believe everything is back in order. I paused the Jenkins queue and waited for any shaman users to exit before shuffling things around, so I think no builds should have been affected (only possibly delayed, but it looks like those are all auto-scheduled jobs, not PRs).
Let me know here or in slack if you see something that looks like a problem with Jenkins and shaman.
On 4/18/2025 5:35 PM, Dan Mick wrote:
I plan to switch in the upgraded shaman this weekend, because past experience shows builds slow regularly on Friday/Saturday. (It is currently running on a very old version of Ubuntu with a very old version of Postgres and a very old version of Python; we've resolved all that.)
It will involve a shutdown of the existing shaman servers to
1) snapshot and populate the new server's database, and 2) change the IP of the server to the existing 1.shaman.ceph.com IP
I've run through this process a few times in isolation, and I expect it to take no more than 10-15 minutes. I'll announce when it's imminent, either later this evening or tomorrow.
Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
participants (2)
-
Brett Niver
-
Dan Mick