Ceph Tentacle release timeline — when?
Hi all, We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?) * Unfortunately, due to issues in the sepia lab and with testing infrastructure, we had big slips on Reef (August), and we tried to get Squid back on track but it ended up slipping farther (September). * The sepia lab needs to physically move this year out of the current Red Hat community hosting area in to a new lab (hosted by IBM). We're not sure how disruptive this will be, but it will definitely require changes as our current tests are somewhat dependent on the shape of the underlying hardware, which is changing. * There are not many major features available yet. Major upcoming features include the fast EC code, userspace and management support for cephfs encryption, and RBD mirroring groups. Different parts of these are coming throughout the calendar year but we don't have firm timetables yet. We have basically discarded attempting to do an early spring release, since there doesn't seem to be much point. The obvious options are 1) Do a fall release, targeting 12 months after Squid (so, late September, with a freeze in early July). 2) Wait until early spring 2026, with a freeze in January. (This doesn't preclude other options as we gather more input, but we are starting with these for discussion.) We are soliciting feedback on the impact these options will have on our various downstreams and direct upstream consumers. The CSC's *early and tentative* preference is for spring 2026, for the following reasons: 1) the sepia lab move is going to involve an unknown amount of development, and we are gunshy after issues the last couple years. 2) While there is feature work in development, we are unsure how much of interest will be both ready to freeze by mid-summer, and not suitable for backport to our stable releases. 3) Fall releases seem awkward with many organizations wanting to freeze infrastructure over the holidays. 4) While the data is conflicting, it seems like deployments are lagging more on major-version upgrades and so a longer release cycle may make future updates easier for everyone. If the Ceph release schedule has a large impact on your organization or planning, please let us know your thoughts and forward to your downstreams. :) Thanks! -The Ceph Steering Committee, by way of Greg
We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?)
Please do not assume everyone runs ceph in containers so the OS doesn't matter, we and others run ceph using the supplied rpms and debs. -- May the most significant bit of your life be positive.
On 05/02/2025 15:40, Janne Johansson wrote:
We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?) Please do not assume everyone runs ceph in containers so the OS doesn't matter, we and others run ceph using the supplied rpms and debs.
Absolutely agree with Janne about containers. We use the Centos 9 Stream RPMs and have no plans whatever to move to cephadm containers/orchestration. (Many reasons, but that's a different discussion). What matters to us is: * properly tested packaging for the supported distributions, especially the Category A ones (this is becoming more problematic, especially in the python area) * properly tested functionality * prompt fixing of serious bugs, especially in the latest production version (we haven't even been able to move to squid yet because of the balancer pg-iteration bug). I'd love to see all Ceph systems being able to move easily to the supported releases (n, n-1) so that so much time isn't wasted on problems with obsolete releases. Some of the new things in tentacle sound great, but we need to concentrate on the current basics first. Timescales for the new features are a much more minor consideration for us. Ceph is a fantastic product. Let's keep it that way. Chris
I would like to have it on record that I completely agree with the points Chris (and Janne) made. -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology ________________________________________ From: Chris Palmer <chris.palmer@idnet.com> Sent: Wednesday, February 5, 2025 11:17 AM To: Janne Johansson; dev@ceph.io Cc: ceph-users Subject: [ceph-users] Re: Ceph Tentacle release timeline — when? On 05/02/2025 15:40, Janne Johansson wrote:
We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?) Please do not assume everyone runs ceph in containers so the OS doesn't matter, we and others run ceph using the supplied rpms and debs.
Absolutely agree with Janne about containers. We use the Centos 9 Stream RPMs and have no plans whatever to move to cephadm containers/orchestration. (Many reasons, but that's a different discussion). What matters to us is: * properly tested packaging for the supported distributions, especially the Category A ones (this is becoming more problematic, especially in the python area) * properly tested functionality * prompt fixing of serious bugs, especially in the latest production version (we haven't even been able to move to squid yet because of the balancer pg-iteration bug). I'd love to see all Ceph systems being able to move easily to the supported releases (n, n-1) so that so much time isn't wasted on problems with obsolete releases. Some of the new things in tentacle sound great, but we need to concentrate on the current basics first. Timescales for the new features are a much more minor consideration for us. Ceph is a fantastic product. Let's keep it that way. Chris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Bloomberg is mainly agnostic to the time delay, obviously getting back in alignment with OS releases is ideal. We cannot overstate our agreement on RPM and bare metal support. We also have no desire or interest in being forced to containers. So we also agree to the other on that matter. Sent from Bloomberg Professional for iPhone ----- Original Message ----- From: Paul Mezzanini <pfmeec@rit.edu> To: chris.palmer@idnet.com, icepic.dz@gmail.com, dev@ceph.io CC: ceph-users@ceph.io At: 02/06/25 08:07:58 UTC-05:00 I would like to have it on record that I completely agree with the points Chris (and Janne) made. -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology ________________________________________ From: Chris Palmer <chris.palmer@idnet.com> Sent: Wednesday, February 5, 2025 11:17 AM To: Janne Johansson; dev@ceph.io Cc: ceph-users Subject: [ceph-users] Re: Ceph Tentacle release timeline — when? On 05/02/2025 15:40, Janne Johansson wrote:
We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?) Please do not assume everyone runs ceph in containers so the OS doesn't matter, we and others run ceph using the supplied rpms and debs.
Absolutely agree with Janne about containers. We use the Centos 9 Stream RPMs and have no plans whatever to move to cephadm containers/orchestration. (Many reasons, but that's a different discussion). What matters to us is: * properly tested packaging for the supported distributions, especially the Category A ones (this is becoming more problematic, especially in the python area) * properly tested functionality * prompt fixing of serious bugs, especially in the latest production version (we haven't even been able to move to squid yet because of the balancer pg-iteration bug). I'd love to see all Ceph systems being able to move easily to the supported releases (n, n-1) so that so much time isn't wasted on problems with obsolete releases. Some of the new things in tentacle sound great, but we need to concentrate on the current basics first. Timescales for the new features are a much more minor consideration for us. Ceph is a fantastic product. Let's keep it that way. Chris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I concur strongly with Matthew’s assessment. k Sent from my iPhone
On 6 Feb 2025, at 16:13, Matthew Leonard (BLOOMBERG/ 120 PARK) <mleonard33@bloomberg.net> wrote:
Bloomberg is mainly agnostic to the time delay, obviously getting back in alignment with OS releases is ideal.
We cannot overstate our agreement on RPM and bare metal support. We also have no desire or interest in being forced to containers. So we also agree to the other on that matter.
+1 to this from 45Drives. On 2025-02-06 09:12, Matthew Leonard (BLOOMBERG/ 120 PARK) wrote:
Bloomberg is mainly agnostic to the time delay, obviously getting back in alignment with OS releases is ideal.
We cannot overstate our agreement on RPM and bare metal support. We also have no desire or interest in being forced to containers. So we also agree to the other on that matter.
Sent from Bloomberg Professional for iPhone
----- Original Message ----- From: Paul Mezzanini <pfmeec@rit.edu> To: chris.palmer@idnet.com, icepic.dz@gmail.com, dev@ceph.io CC: ceph-users@ceph.io At: 02/06/25 08:07:58 UTC-05:00
I would like to have it on record that I completely agree with the points Chris (and Janne) made.
--
Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology
________________________________________ From: Chris Palmer <chris.palmer@idnet.com> Sent: Wednesday, February 5, 2025 11:17 AM To: Janne Johansson; dev@ceph.io Cc: ceph-users Subject: [ceph-users] Re: Ceph Tentacle release timeline — when?
On 05/02/2025 15:40, Janne Johansson wrote:
We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?) Please do not assume everyone runs ceph in containers so the OS doesn't matter, we and others run ceph using the supplied rpms and debs.
Absolutely agree with Janne about containers. We use the Centos 9 Stream RPMs and have no plans whatever to move to cephadm containers/orchestration. (Many reasons, but that's a different discussion).
What matters to us is:
* properly tested packaging for the supported distributions, especially the Category A ones (this is becoming more problematic, especially in the python area) * properly tested functionality * prompt fixing of serious bugs, especially in the latest production version (we haven't even been able to move to squid yet because of the balancer pg-iteration bug). I'd love to see all Ceph systems being able to move easily to the supported releases (n, n-1) so that so much time isn't wasted on problems with obsolete releases.
Some of the new things in tentacle sound great, but we need to concentrate on the current basics first. Timescales for the new features are a much more minor consideration for us.
Ceph is a fantastic product. Let's keep it that way.
Chris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2/6/25 14:12, Matthew Leonard (BLOOMBERG/ 120 PARK) wrote:
We cannot overstate our agreement on RPM and bare metal support. We also have no desire or interest in being forced to containers. So we also agree to the other on that matter.
yes - strongly seconded from us too, thanks. Regards, Daniel
To be sure for posterity, nobody’s suggesting abandoning package builds, the discussion is about the major release cadence given the delays experienced in getting Squid out. The discussion is re the importance of co-ordinating with Ubuntu et al package builds that are put into their OS distribution repositories.
On Feb 6, 2025, at 12:44 PM, Daniel Baumann <daniel.baumann@bfh.ch> wrote:
On 2/6/25 14:12, Matthew Leonard (BLOOMBERG/ 120 PARK) wrote:
We cannot overstate our agreement on RPM and bare metal support. We also have no desire or interest in being forced to containers. So we also agree to the other on that matter.
yes - strongly seconded from us too, thanks.
Regards, Daniel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, The respondents are not confused in the topic, only once again it is highlighted that releases once every 9 months are not really necessary, considering that the team presented the last bugfix release in July of last year (now it is February). Regarding containers, once again the community and administrators of large deployments around the world are expressing their opinion, and finding multiple support k Sent from my iPhone
On 6 Feb 2025, at 23:31, Anthony D'Atri <anthony.datri@gmail.com> wrote:
To be sure for posterity, nobody’s suggesting abandoning package builds, the discussion is about the major release cadence given the delays experienced in getting Squid out. The discussion is re the importance of co-ordinating with Ubuntu et al package builds that are put into their OS distribution repositories.
On Feb 6, 2025, at 12:44 PM, Daniel Baumann <daniel.baumann@bfh.ch> wrote:
On 2/6/25 14:12, Matthew Leonard (BLOOMBERG/ 120 PARK) wrote: We cannot overstate our agreement on RPM and bare metal support. We also have no desire or interest in being forced to containers. So we also agree to the other on that matter.
yes - strongly seconded from us too, thanks.
Regards, Daniel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?) Please do not assume everyone runs ceph in containers so the OS doesn't matter, we and others run ceph using the supplied rpms and debs.
Fully agree with Janne on the container assumption comment. The OS release alignment and upgrade avenue on bare machines is of utmost importance for us. Milan -- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing
On 5/2/25 23:04, Gregory Farnum wrote:
We are soliciting feedback on the impact these options will have on our various downstreams and direct upstream consumers. The CSC's *early and tentative* preference is for spring 2026
This looks like a good option from all of the context you provided. The standard dates for releases have value, and since the current one is slipping a little more each time this would be a chance to truly get it back on track. All of the same would be true about us getting our systems upgraded to current versions, as you alluded to.
On 05-02-2025 16:04, Gregory Farnum wrote:
Hi all, We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?)
Sage Weil explicitly mentioned this as the reason to align Ceph releases with OS distributions during Cephalocon Barcelona. The Mimic release did not get picked up because of this misalignment. I would still vote for having this aligned. While you can run ceph in containers, there is also a need (at least for us) to have ceph packages installed on the physical hardware as well (monitoring related). Without having OS packages you can't just use cephadm-ansible to prepare the physical hosts. And if you are not running in containers but using packages it will matter even more so. Gr. Stefan
Hi,
On 5. Feb 2025, at 16:04, Gregory Farnum <gfarnum@redhat.com> wrote:
Hi all, We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?)
We run Ceph with custom compiled packages on NixOS and aren’t particularly worried about distribution timing. However, this is due to us currently lagging A LOT due to having been bitten previously again and again with unstable majors. So, we’re currently on Nautilus, starting to upgrade on Pacific this spring and might be ready to move to Squid in Fall. I *LIKE* being flexible of which packages to include, independent of the underlying distribution. As a side note: NixOS tries to be very support-focused, so there is a part of the community that would like to not even include a package if it runs EOL during a distro’s release cycle. However, that’s only 6 months and Ceph generally supports longer cycles, so that should work out for the general population just fine.
* Unfortunately, due to issues in the sepia lab and with testing infrastructure, we had big slips on Reef (August), and we tried to get Squid back on track but it ended up slipping farther (September). * The sepia lab needs to physically move this year out of the current Red Hat community hosting area in to a new lab (hosted by IBM). We're not sure how disruptive this will be, but it will definitely require changes as our current tests are somewhat dependent on the shape of the underlying hardware, which is changing. * There are not many major features available yet. Major upcoming features include the fast EC code, userspace and management support for cephfs encryption, and RBD mirroring groups. Different parts of these are coming throughout the calendar year but we don't have firm timetables yet.
Yeah, looking at the subset that we use, we’d like to do better staying up to date but not particularly focused on new features, rather stability and operability. RFC 1925, Point 1. ;) (Online reshaping of EC pools, anyone? *cough*)
We have basically discarded attempting to do an early spring release, since there doesn't seem to be much point. The obvious options are 1) Do a fall release, targeting 12 months after Squid (so, late September, with a freeze in early July). 2) Wait until early spring 2026, with a freeze in January. (This doesn't preclude other options as we gather more input, but we are starting with these for discussion.)
I’d be happy with this as it gives us more time (and motivation) to catch up. To repeat on our goals: I’d rather have smaller, well-tested releases that provide a smooth upgrade path. Taking care of the testing infrastructure is a big part of that IMHO, so I’d applaud you to take the time to do it with sufficient attention to detail and not try to push out a release while juggling that. Christian -- Christian Theune · ct@flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
participants (13)
-
Anthony D'Atri
-
Bailey Allison
-
Chris Palmer
-
Christian Theune
-
Daniel Baumann
-
Gregory Farnum
-
Gregory Orange
-
Janne Johansson
-
Konstantin Shalygin
-
Matthew Leonard (BLOOMBERG/ 120 PARK)
-
Milan Kupcevic
-
Paul Mezzanini
-
Stefan Kooman