centos 8 stream eol on friday may 31
from https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and... all ceph builds for centos 8 will start failing then, because dnf won't be able to find the centos package repos any more the teuthology suites for main and squid don't rely on centos 8 packages any more, but cephadm suites still use the centos 8-based container. these suites will break until we switch to the centos 9-based images. this should happen very soon, if it hasn't already the quincy and reef qa suites still depend on centos 8 for packages and containers, though we do build centos 9 packages for both. we'll just need to backport a lot of the changes (like https://github.com/ceph/ceph/pull/53517 and https://github.com/ceph/ceph/pull/53901) that removed centos 8 testing on main we've been planning to switch to centos 9-based containers for main and squid, and most of the work there is done. but this will also necessitate a switch for the quincy and reef releases because ceph containers need security updates the centos 9 builds for quincy and reef are already trying to build containers, but that containerization step has been failing for quincy with what looks like a ganesha-related issue (see https://tracker.ceph.com/issues/66253) am i missing anything?
On Tue, 2024-05-28 at 11:29 -0400, Casey Bodley wrote: ...
the centos 9 builds for quincy and reef are already trying to build containers, but that containerization step has been failing for quincy with what looks like a ganesha-related issue
https://tracker.ceph.com/issues/66253 contains a reference to nfsganesha-4, which does not exist for CentOS 9. It needs to use nfsganesha-5 instead, see https://buildlogs.centos.org/centos/9/storage/x86_64/ To use the right repository file, the best way is to install the centos-release-nfs-ganesha5 package (just like epel-release). But that could be a larger change, depending on how the .repo file is obtained/created at the moment. HTH, Niels Unless otherwise stated above: IBM Nederland B.V. Gevestigd te Amsterdam Inschrijving Handelsregister Amsterdam Nr. 33054214
On Tue, May 28, 2024 at 11:30 AM Casey Bodley <cbodley@redhat.com> wrote:
from https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and...
all ceph builds for centos 8 will start failing then, because dnf won't be able to find the centos package repos any more
the teuthology suites for main and squid don't rely on centos 8 packages any more, but cephadm suites still use the centos 8-based container. these suites will break until we switch to the centos 9-based images. this should happen very soon, if it hasn't already
the quincy and reef qa suites still depend on centos 8 for packages and containers, though we do build centos 9 packages for both. we'll just need to backport a lot of the changes (like https://github.com/ceph/ceph/pull/53517 and https://github.com/ceph/ceph/pull/53901) that removed centos 8 testing on main
we've been planning to switch to centos 9-based containers for main and squid, and most of the work there is done. but this will also necessitate a switch for the quincy and reef releases because ceph containers need security updates
the centos 9 builds for quincy and reef are already trying to build containers, but that containerization step has been failing for quincy with what looks like a ganesha-related issue (see https://tracker.ceph.com/issues/66253)
https://github.com/ceph/ceph-build/pull/2235#issuecomment-2150782468 This is insane. They've actually turned off the mirrors and painted this as a "helpful" exercise in disaster recovery? Wow. Are we going to have to play this game every time Centos EOLs a release? If so, I propose we switch to another distribution that is not a guinea pig environment for a downstream distro. Either switch to actual RHEL or a centos fork? I'm not at all amused by this waste of developers' time. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Are we going to have to play this game every time Centos EOLs a release? If so, I propose we switch to another distribution that is not a guinea pig environment for a downstream distro. Either switch to actual RHEL or a centos fork?
RHEL would need a license. Dare I suggest Ubuntu?
I'm not at all amused by this waste of developers' time.
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
If we're open to ideas, I'd put my 2c in suggesting Debian over Ubuntu. It's extraordinarily stable, tends to adopt changes more pragmatically and with more community involvement, has long release cycles, has been around decades, and is completely free. Debian-slim is also a great starting point for containers; we've moved to it for all of our needs (only 27.8M for the latest stable base image on AMD64): https://hub.docker.com/layers/library/debian/stable-slim/images/sha256-d9b8e... Ubuntu tends to 'go its own way' often, which can be problematic (upstart, snap, etc). I'd certainly prefer it to RHEL/CentOS, but I think Debian is a better fit for this kind of application. On Wed, Jun 5, 2024, at 14:33, Anthony D'Atri wrote:
Are we going to have to play this game every time Centos EOLs a release? If so, I propose we switch to another distribution that is not a guinea pig environment for a downstream distro. Either switch to actual RHEL or a centos fork?
RHEL would need a license. Dare I suggest Ubuntu?
I'm not at all amused by this waste of developers' time.
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 05/06/2024 21:44, David Orman wrote:
If we're open to ideas, I'd put my 2c in suggesting Debian over Ubuntu.
This is a fine plan (I'm totally biased, because I'm already building Debian-based images at work, and one of my other hats is a Debian developer, but I do think it'd be a good move). Regards, Matthew
So the distribution that Ceph itself isn't tested against, instead of the one it is. https://docs.ceph.com/en/latest/start/os-recommendations/#platforms
On Jun 6, 2024, at 08:55, Matthew Vernon <mvernon@wikimedia.org> wrote:
On 05/06/2024 21:44, David Orman wrote:
If we're open to ideas, I'd put my 2c in suggesting Debian over Ubuntu.
This is a fine plan (I'm totally biased, because I'm already building Debian-based images at work, and one of my other hats is a Debian developer, but I do think it'd be a good move).
Regards,
Matthew _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Any change to the distribution used for images must involve ensuring testing was performed with that choice; there is no question about that. Modifying the builds to use a different distribution for a target image is going to require updating at various levels to support the choice regardless of what direction is taken. David On Thu, Jun 6, 2024, at 08:16, Anthony D'Atri wrote:
So the distribution that Ceph itself isn't tested against, instead of the one it is.
OS Recommendations — Ceph Documentation <https://docs.ceph.com/en/latest/start/os-recommendations/#platforms> docs.ceph.com <https://docs.ceph.com/en/latest/start/os-recommendations/#platforms> favicon.ico <https://docs.ceph.com/en/latest/start/os-recommendations/#platforms> <https://docs.ceph.com/en/latest/start/os-recommendations/#platforms>
On Jun 6, 2024, at 08:55, Matthew Vernon <mvernon@wikimedia.org> wrote:
On 05/06/2024 21:44, David Orman wrote:
If we're open to ideas, I'd put my 2c in suggesting Debian over Ubuntu.
This is a fine plan (I'm totally biased, because I'm already building Debian-based images at work, and one of my other hats is a Debian developer, but I do think it'd be a good move).
Regards,
Matthew _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Wed, Jun 5, 2024 at 3:30 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
This is insane. They've actually turned off the mirrors and painted this as a "helpful" exercise in disaster recovery? Wow.
Stream8 is still available from the CentOS Vault. See Fabian's recent email to CentOS's discuss, devel, and announce lists, where he writes: ... The latest compose from today was archived to Vault (https://vault.centos.org/8-stream/)... ... AFAIK you should be able to contrive to use vault in lieu of the mirrors. (They might not like the load that Ceph using it might place on it, but it could be argued that they have it coming. ;-)) You could also front your own mirror of vault. Just sayin' -- Kaleb
On Wed, Jun 5, 2024 at 12:29 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Tue, May 28, 2024 at 11:30 AM Casey Bodley <cbodley@redhat.com> wrote:
from https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and...
all ceph builds for centos 8 will start failing then, because dnf won't be able to find the centos package repos any more
the teuthology suites for main and squid don't rely on centos 8 packages any more, but cephadm suites still use the centos 8-based container. these suites will break until we switch to the centos 9-based images. this should happen very soon, if it hasn't already
the quincy and reef qa suites still depend on centos 8 for packages and containers, though we do build centos 9 packages for both. we'll just need to backport a lot of the changes (like https://github.com/ceph/ceph/pull/53517 and https://github.com/ceph/ceph/pull/53901) that removed centos 8 testing on main
we've been planning to switch to centos 9-based containers for main and squid, and most of the work there is done. but this will also necessitate a switch for the quincy and reef releases because ceph containers need security updates
the centos 9 builds for quincy and reef are already trying to build containers, but that containerization step has been failing for quincy with what looks like a ganesha-related issue (see https://tracker.ceph.com/issues/66253)
https://github.com/ceph/ceph-build/pull/2235#issuecomment-2150782468
This is insane. They've actually turned off the mirrors and painted this as a "helpful" exercise in disaster recovery? Wow.
Are we going to have to play this game every time Centos EOLs a release? If so, I propose we switch to another distribution that is not a guinea pig environment for a downstream distro. Either switch to actual RHEL or a centos fork?
I'm not at all amused by this waste of developers' time.
http://mirror.centos.org/centos/8-stream/readme Bonkers. Yeah a clone or one of the Debians, Let's do it! Stream dailies were always a questionable choice, considering we're storage software. And while we're at it, could we switch to a base image with a minimal SBOM? Whatever we do here, it's a reminder that we need to maintain our old builds, to bridge users across OS's and our Releases. There are plenty of users still running Nautilus on el7. We have rpms for 14.2.22 on el7 and el8. 16.2.15 is el8 only, then we have 17.2.7 builds for el8 and el9. regards, dan
On Wed, Jun 5, 2024 at 3:29 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Tue, May 28, 2024 at 11:30 AM Casey Bodley <cbodley@redhat.com> wrote:
from https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and...
all ceph builds for centos 8 will start failing then, because dnf won't be able to find the centos package repos any more
the teuthology suites for main and squid don't rely on centos 8 packages any more, but cephadm suites still use the centos 8-based container. these suites will break until we switch to the centos 9-based images. this should happen very soon, if it hasn't already
the quincy and reef qa suites still depend on centos 8 for packages and containers, though we do build centos 9 packages for both. we'll just need to backport a lot of the changes (like https://github.com/ceph/ceph/pull/53517 and https://github.com/ceph/ceph/pull/53901) that removed centos 8 testing on main
we've been planning to switch to centos 9-based containers for main and squid, and most of the work there is done. but this will also necessitate a switch for the quincy and reef releases because ceph containers need security updates
the centos 9 builds for quincy and reef are already trying to build containers, but that containerization step has been failing for quincy with what looks like a ganesha-related issue (see https://tracker.ceph.com/issues/66253)
https://github.com/ceph/ceph-build/pull/2235#issuecomment-2150782468
This is insane. They've actually turned off the mirrors and painted this as a "helpful" exercise in disaster recovery? Wow.
it's important that users know when their distro stops getting security updates. archiving the package repo seems like a reasonable way to alert users (like us) that aren't paying attention to eol dates. that noticeably breaks things, but has a workaround to reenable the archived packages. if absolutely necessary, we could probably revive c8 builds and test suites using vault.centos.org. but shouldn't we just stop using it?
Are we going to have to play this game every time Centos EOLs a release? If so, I propose we switch to another distribution that is not a guinea pig environment for a downstream distro. Either switch to actual RHEL or a centos fork?
I'm not at all amused by this waste of developers' time.
i understand the frustration, but pretty much any distro choice will require some level of maintenance on our part. i'm not against moving away from centos, but that's going to cost developer time too. and we'll still have to keep building and testing centos 9 packages for the releases that already support it
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Wed, Jun 5, 2024 at 4:41 PM Casey Bodley <cbodley@redhat.com> wrote:
On Wed, Jun 5, 2024 at 3:29 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Tue, May 28, 2024 at 11:30 AM Casey Bodley <cbodley@redhat.com> wrote:
from https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and...
all ceph builds for centos 8 will start failing then, because dnf won't be able to find the centos package repos any more
the teuthology suites for main and squid don't rely on centos 8 packages any more, but cephadm suites still use the centos 8-based container. these suites will break until we switch to the centos 9-based images. this should happen very soon, if it hasn't already
the quincy and reef qa suites still depend on centos 8 for packages and containers, though we do build centos 9 packages for both. we'll just need to backport a lot of the changes (like https://github.com/ceph/ceph/pull/53517 and https://github.com/ceph/ceph/pull/53901) that removed centos 8 testing on main
we've been planning to switch to centos 9-based containers for main and squid, and most of the work there is done. but this will also necessitate a switch for the quincy and reef releases because ceph containers need security updates
the centos 9 builds for quincy and reef are already trying to build containers, but that containerization step has been failing for quincy with what looks like a ganesha-related issue (see https://tracker.ceph.com/issues/66253)
https://github.com/ceph/ceph-build/pull/2235#issuecomment-2150782468
This is insane. They've actually turned off the mirrors and painted this as a "helpful" exercise in disaster recovery? Wow.
it's important that users know when their distro stops getting security updates. archiving the package repo seems like a reasonable way to alert users (like us) that aren't paying attention to eol dates. that noticeably breaks things, but has a workaround to reenable the archived packages. if absolutely necessary, we could probably revive c8 builds and test suites using vault.centos.org. but shouldn't we just stop using it?
It's fine for users to be strongly encouraged to migrate but we have automated tests for supported releases that depend on those distributions still functioning for build purposes. To pull the rug out from under every developer like this is simply unacceptable. I have zero interest in going back to every supported release to rearchitect support for the latest and greatest centos stream. Even worse, the workloads we run which are built against that release will probably need either upgraded too, dependencies found/updated, or any other number of "fun" things that need changed when we switch distributions. Beyond the main branch upgrade suite (especially fs:upgrade) being crippled by this change, I expect all of the upgrade suites for quincy/reef to be in a similar or worse state. We won't be able to install octopus/pacific on centos8 and then upgrade to quincy/reef.
Are we going to have to play this game every time Centos EOLs a release? If so, I propose we switch to another distribution that is not a guinea pig environment for a downstream distro. Either switch to actual RHEL or a centos fork?
I'm not at all amused by this waste of developers' time.
i understand the frustration, but pretty much any distro choice will require some level of maintenance on our part. i'm not against moving away from centos, but that's going to cost developer time too. and we'll still have to keep building and testing centos 9 packages for the releases that already support it
Well, let's officially start looking or developing plans for local mirrors so we can deploy those distributions even if the upstream mirrors are turned off. (To be clear, I am very appreciative of your efforts Casey in trying to prepare for this. I didn't understand how bad things would get until now.) -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
CentOS Stream you mean?
On Jun 10, 2024, at 12:58, Ken Dreyer <kdreyer@redhat.com> wrote:
On Wed, Jun 5, 2024 at 3:29 PM Patrick Donnelly <pdonnell@redhat.com <mailto:pdonnell@redhat.com>> wrote:
Are we going to have to play this game every time Centos EOLs a release?
No, because we're going to switch to CentOS 10 sooner. _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (10)
-
Anthony D'Atri
-
Anthony D'Atri
-
Casey Bodley
-
Dan van der Ster
-
David Orman
-
Kaleb Keithley
-
Ken Dreyer
-
Matthew Vernon
-
Niels de Vos
-
Patrick Donnelly