ceph reef on AlmaLinux9 (and the dependency problem on openssl-libs)
Dear all Some weeks ago we updated our ceph cluster (installed on AlmaLinux9 servers) to Ceph reef v. 18.2.6. The day after the email talking about the serious regression bug in that release was sent to this mailing list Frankly it is not fully clear to me if/how that bug is relevant for our deployment: I can say that the OSD start without problems and I can't see obvious problems 18.2.7 can't be installed because it needed openssl-libs v. 3.4 (as discussed in this mailing list) which is in CentOS9 stream but not yet in AlmaLinux9 I have no idea if/when that openssl-libs 3.4 stuff will be released in AlmaLinux9 (AlmaLinux 9.6, released a couple of days ago, doesn't provide an update for openssl-libs). I have now to install some new servers and I have no clear ideas how to proceed Should I keep version 18.2.6 on the existing servers and install 18.2.5 on the new ones ? Should I try installing version 18.2.6 also on the new servers ? Any suggestions? Thanks, Massimo
One good reason to use cephadm and the container based deployment! Michel Sent from my mobile Le 22 mai 2025 15:54:54 Massimo Sgaravatto <massimo.sgaravatto@gmail.com> a écrit :
Dear all
Some weeks ago we updated our ceph cluster (installed on AlmaLinux9 servers) to Ceph reef v. 18.2.6. The day after the email talking about the serious regression bug in that release was sent to this mailing list Frankly it is not fully clear to me if/how that bug is relevant for our deployment: I can say that the OSD start without problems and I can't see obvious problems
18.2.7 can't be installed because it needed openssl-libs v. 3.4 (as discussed in this mailing list) which is in CentOS9 stream but not yet in AlmaLinux9
I have no idea if/when that openssl-libs 3.4 stuff will be released in AlmaLinux9 (AlmaLinux 9.6, released a couple of days ago, doesn't provide an update for openssl-libs).
I have now to install some new servers and I have no clear ideas how to proceed
Should I keep version 18.2.6 on the existing servers and install 18.2.5 on the new ones ? Should I try installing version 18.2.6 also on the new servers ?
Any suggestions?
Thanks, Massimo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Massimo, Regarding bluestore bugs in 18.2.5 and 18.2.6, if you did not encounter them during the upgrade I think you should be relatively safe for now but can't tell for sure. You should upgrade to 18.2.7 as soon as you can. Regarding AlmaLinux, you could try to build openssl-libs v3.4 or download and install openssl-libs v3.5 on the new servers and see if you can install and run Ceph v18.2.7 on them. OR (better) Unless you have strong operational concerns about using containers (e.g. you operate tens of clusters with your own automation tools and don't want to change your operational methods) you could switch to containers with cephadm adopt [1] or cephadm-adopt.yml playbook [2] if you're using ceph-ansible. Containers have been around for a long time now and it's the only supported method for deploying and operating downstream Ceph since Red Hat Ceph Storage 5 (Pacific). It's mature, stable, eases and accelerates a lot of operations (e.g. upgrades, adding/removing OSDs, MONs, etc.) and offers new possibilities (e.g running multiple RGWs on the same host). Regards, Frédéric. [1] https://docs.ceph.com/en/reef/cephadm/adoption/ [2] https://github.com/ceph/ceph-ansible/blob/main/infrastructure-playbooks/ceph... ----- Le 22 Mai 25, à 18:54, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :
One good reason to use cephadm and the container based deployment!
Michel Sent from my mobile Le 22 mai 2025 15:54:54 Massimo Sgaravatto <massimo.sgaravatto@gmail.com> a écrit :
Dear all
Some weeks ago we updated our ceph cluster (installed on AlmaLinux9 servers) to Ceph reef v. 18.2.6. The day after the email talking about the serious regression bug in that release was sent to this mailing list Frankly it is not fully clear to me if/how that bug is relevant for our deployment: I can say that the OSD start without problems and I can't see obvious problems
18.2.7 can't be installed because it needed openssl-libs v. 3.4 (as discussed in this mailing list) which is in CentOS9 stream but not yet in AlmaLinux9
I have no idea if/when that openssl-libs 3.4 stuff will be released in AlmaLinux9 (AlmaLinux 9.6, released a couple of days ago, doesn't provide an update for openssl-libs).
I have now to install some new servers and I have no clear ideas how to proceed
Should I keep version 18.2.6 on the existing servers and install 18.2.5 on the new ones ? Should I try installing version 18.2.6 also on the new servers ?
Any suggestions?
Thanks, Massimo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den fre 23 maj 2025 kl 04:23 skrev Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:
One good reason to use cephadm and the container based deployment!
I would say that the Reef release has been an interesting ride, you hold off on the 18.2.0 and early releases to be safe from the worst burn-in bugs and by the time you get your first test cluster with reef, its around 18.2.2, so this is when the "fun" starts. First 18.2.3 leak out so 18.2.4 needs to get pushed out so people who accidentally got the not-good 18.2.3 can upgrade, then around this time rgw uploads start to fail due to how AWS-supporting clients want to have their checksumming done, so all the "we run latest aws-sdk" clients can't upload to S3 anymore, then come 18.2.5 with a solution for this so you upgrade into that, which in turn makes ipv6 configs fail/complain and dmcrypt for OSDs gets broken, so you quickly upgrade to the fresh 18.2.6, but that seems to make certain clusters OSDs to crash, so you hold your breath while waiting for 18.2.7 and if you run Alma Linux 9 (and presumably other rhel/centos derivates that have worked for the other 18.2.x releases) the upgrade now fails due to changed ssl deps and suddenly you need to flip your whole cluster deployment to containers. If anyone had an idea that upgrading minors Should Be Fine(tm) then ceph 18 will be a fresh break from that. No, I did not experience ALL these issues myself - only some of them - but its been quite the time to get the popcorn..
Some weeks ago we updated our ceph cluster (installed on AlmaLinux9 servers) to Ceph reef v. 18.2.6. The day after the email talking about the serious regression bug in that release was sent to this mailing list
18.2.7 can't be installed because it needed openssl-libs v. 3.4 (as discussed in this mailing list) which is in CentOS9 stream but not yet in AlmaLinux9
-- May the most significant bit of your life be positive.
Hi, Thanks for the perfect overview of the Reef release! I'll steal this as is for the slide, for another overview of why it's important to have an update strategy. Sometimes folks don't understand why our 75 clusters are using the Nautilus or Pacific release Thanks, k Sent from my iPhone
On 25 May 2025, at 10:18, Janne Johansson <icepic.dz@gmail.com> wrote:
No, I did not experience ALL these issues myself - only some of them - but its been quite the time to get the popcorn..
Absolutely. I always say to upgrade production when there’s a compelling reason. A new dot release — of any software — is not in isolation compelling. Bug fixes may be, as may new features. That said, we only learn of issues when new releases are stressed in diverse environments. Upstream performs exhaustive regression tests but can’t always model every permutation of use case and cluster history. So there’s a lot of value in smoke testing in dev and staging clusters.
On May 25, 2025, at 4:41 AM, Konstantin Shalygin <k0ste@k0ste.ru> wrote:
Hi,
Thanks for the perfect overview of the Reef release! I'll steal this as is for the slide, for another overview of why it's important to have an update strategy. Sometimes folks don't understand why our 75 clusters are using the Nautilus or Pacific release
Thanks, k Sent from my iPhone
On 25 May 2025, at 10:18, Janne Johansson <icepic.dz@gmail.com> wrote:
No, I did not experience ALL these issues myself - only some of them - but its been quite the time to get the popcorn..
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I cannot say if the overview is perfect. Among the compelling reasons, staying on a supported release may be one of them and Quincy is no longer one. BTW it suffers the same problem as 18.2.6 in its last minor and I understood an exception has been made to the EOL policy to get a patch out but it may not have been the case... My feeling about the 18.2.x difficulties is that at least the last one with 18.2.7 is not really a Ceph problem but a packaging one. And I'm afraid it will happen again as long as RH packages are built against CentOS. As CentOS is now upstream (instead of downstream) to RHEL, it means it delivers packages that are not in RHEL or any derivatives (that are downstream) and thus Ceph may not work on them. IMO, the only way to fix it would be to build Ceph packages on RHEL or one of its derivative (if there is a license issue with RHEL). For this particular point, I think the main solution is to rely on container-based Ceph, i.e. cephadm, as discussed in another thread. Best regards, Michel Le 25/05/2025 à 16:08, Anthony D'Atri a écrit :
Absolutely. I always say to upgrade production when there’s a compelling reason. A new dot release — of any software — is not in isolation compelling. Bug fixes may be, as may new features. That said, we only learn of issues when new releases are stressed in diverse environments. Upstream performs exhaustive regression tests but can’t always model every permutation of use case and cluster history. So there’s a lot of value in smoke testing in dev and staging clusters.
On May 25, 2025, at 4:41 AM, Konstantin Shalygin <k0ste@k0ste.ru> wrote:
Hi,
Thanks for the perfect overview of the Reef release! I'll steal this as is for the slide, for another overview of why it's important to have an update strategy. Sometimes folks don't understand why our 75 clusters are using the Nautilus or Pacific release
Thanks, k Sent from my iPhone
On 25 May 2025, at 10:18, Janne Johansson <icepic.dz@gmail.com> wrote:
No, I did not experience ALL these issues myself - only some of them - but its been quite the time to get the popcorn..
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Anthony D'Atri
-
Frédéric Nass
-
Janne Johansson
-
Konstantin Shalygin
-
Massimo Sgaravatto
-
Michel Jouvin