Changing the release cadence
Hi everyone, Since luminous, we have had the follow release cadence and policy: - release every 9 months - maintain backports for the last two releases - enable upgrades to move either 1 or 2 releases heads (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...) This has mostly worked out well, except that the mimic release received less attention that we wanted due to the fact that multiple downstream Ceph products (from Red Has and SUSE) decided to based their next release on nautilus. Even though upstream every release is an "LTS" release, as a practical matter mimic got less attention than luminous or nautilus. We've had several requests/proposals to shift to a 12 month cadence. This has several advantages: - Stable/conservative clusters only have to be upgraded every 2 years (instead of every 18 months) - Yearly releases are more likely to intersect with downstream distribution release (e.g., Debian). In the past there have been problems where the Ceph releases included in consecutive releases of a distro weren't easily upgradeable. - Vendors that make downstream Ceph distributions/products tend to release yearly. Aligning with those vendors means they are more likely to productize *every* Ceph release. This will help make every Ceph release an "LTS" release (not just in name but also in terms of maintenance attention). So far the balance of opinion seems to favor a shift to a 12 month cycle[1], especially among developers, so it seems pretty likely we'll make that shift. (If you do have strong concerns about such a move, now is the time to raise them.) That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time *every year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully! A few options: - November: If we release Octopus 9 months from the Nautilus release (planned for Feb, released in Mar) then we'd target this November. We could shift to a 12 months candence after that. - February: That's 12 months from the Nautilus target. - March: That's 12 months from when Nautilus was *actually* released. November is nice in the sense that we'd wrap things up before the holidays. It's less good in that users may not be inclined to install the new release when many developers will be less available in December. February kind of sucked in that the scramble to get the last few things done happened during the holidays. OTOH, we should be doing what we can to avoid such scrambles, so that might not be something we should factor in. March may be a bit more balanced, with a solid 3 months before when people are productive, and 3 months after before they disappear on holiday to address any post-release issues. People tend to be somewhat less available over the summer months due to holidays etc, so an early or late summer release might also be less than ideal. Thoughts? If we can narrow it down to a few options maybe we could do a poll to gauge user preferences. Thanks! sage [1] https://twitter.com/larsmb/status/1130010208971952129
On 6/5/19 5:57 PM, Sage Weil wrote:
So far the balance of opinion seems to favor a shift to a 12 month cycle [...] it seems pretty likely we'll make that shift.
thanks, much appreciated (from an cluster operating point of view).
Thoughts?
GNOME and a few others are doing April and October releases which seems balanced and to be good timing for most people; personally I prefer spring rather than autum for upgrades, hence.. would suggest April. Regards, Daniel
Hi,
- November: If we release Octopus 9 months from the Nautilus release (planned for Feb, released in Mar) then we'd target this November. We could shift to a 12 months candence after that.
For the 2 last debian releases, the freeze was around january-february, november seem to be a good time for ceph release. ----- Mail original ----- De: "Sage Weil" <sage@newdream.net> À: "ceph-users" <ceph-users@ceph.com>, "ceph-devel" <ceph-devel@vger.kernel.org>, dev@ceph.io Envoyé: Mercredi 5 Juin 2019 17:57:52 Objet: Changing the release cadence Hi everyone, Since luminous, we have had the follow release cadence and policy: - release every 9 months - maintain backports for the last two releases - enable upgrades to move either 1 or 2 releases heads (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...) This has mostly worked out well, except that the mimic release received less attention that we wanted due to the fact that multiple downstream Ceph products (from Red Has and SUSE) decided to based their next release on nautilus. Even though upstream every release is an "LTS" release, as a practical matter mimic got less attention than luminous or nautilus. We've had several requests/proposals to shift to a 12 month cadence. This has several advantages: - Stable/conservative clusters only have to be upgraded every 2 years (instead of every 18 months) - Yearly releases are more likely to intersect with downstream distribution release (e.g., Debian). In the past there have been problems where the Ceph releases included in consecutive releases of a distro weren't easily upgradeable. - Vendors that make downstream Ceph distributions/products tend to release yearly. Aligning with those vendors means they are more likely to productize *every* Ceph release. This will help make every Ceph release an "LTS" release (not just in name but also in terms of maintenance attention). So far the balance of opinion seems to favor a shift to a 12 month cycle[1], especially among developers, so it seems pretty likely we'll make that shift. (If you do have strong concerns about such a move, now is the time to raise them.) That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time *every year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully! A few options: - November: If we release Octopus 9 months from the Nautilus release (planned for Feb, released in Mar) then we'd target this November. We could shift to a 12 months candence after that. - February: That's 12 months from the Nautilus target. - March: That's 12 months from when Nautilus was *actually* released. November is nice in the sense that we'd wrap things up before the holidays. It's less good in that users may not be inclined to install the new release when many developers will be less available in December. February kind of sucked in that the scramble to get the last few things done happened during the holidays. OTOH, we should be doing what we can to avoid such scrambles, so that might not be something we should factor in. March may be a bit more balanced, with a solid 3 months before when people are productive, and 3 months after before they disappear on holiday to address any post-release issues. People tend to be somewhat less available over the summer months due to holidays etc, so an early or late summer release might also be less than ideal. Thoughts? If we can narrow it down to a few options maybe we could do a poll to gauge user preferences. Thanks! sage [1] https://twitter.com/larsmb/status/1130010208971952129
It seems like since the change to the 9 months cadence it has been bumpy for the Debian based installs. Changing to a 12 month cadence sounds like a good idea. Perhaps some Debian maintainers can suggest a good month for them to get the packages in time for their release cycle. On 2019-06-05 12:16 pm, Alexandre DERUMIER wrote:
Hi,
- November: If we release Octopus 9 months from the Nautilus release (planned for Feb, released in Mar) then we'd target this November. We could shift to a 12 months candence after that.
For the 2 last debian releases, the freeze was around january-february, november seem to be a good time for ceph release.
----- Mail original ----- De: "Sage Weil" <sage@newdream.net> À: "ceph-users" <ceph-users@ceph.com>, "ceph-devel" <ceph-devel@vger.kernel.org>, dev@ceph.io Envoyé: Mercredi 5 Juin 2019 17:57:52 Objet: Changing the release cadence
Hi everyone,
Since luminous, we have had the follow release cadence and policy: - release every 9 months - maintain backports for the last two releases - enable upgrades to move either 1 or 2 releases heads (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)
This has mostly worked out well, except that the mimic release received less attention that we wanted due to the fact that multiple downstream Ceph products (from Red Has and SUSE) decided to based their next release on nautilus. Even though upstream every release is an "LTS" release, as a practical matter mimic got less attention than luminous or nautilus.
We've had several requests/proposals to shift to a 12 month cadence. This has several advantages:
- Stable/conservative clusters only have to be upgraded every 2 years (instead of every 18 months) - Yearly releases are more likely to intersect with downstream distribution release (e.g., Debian). In the past there have been problems where the Ceph releases included in consecutive releases of a distro weren't easily upgradeable. - Vendors that make downstream Ceph distributions/products tend to release yearly. Aligning with those vendors means they are more likely to productize *every* Ceph release. This will help make every Ceph release an "LTS" release (not just in name but also in terms of maintenance attention).
So far the balance of opinion seems to favor a shift to a 12 month cycle[1], especially among developers, so it seems pretty likely we'll make that shift. (If you do have strong concerns about such a move, now is the time to raise them.)
That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time *every year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully!
A few options:
- November: If we release Octopus 9 months from the Nautilus release (planned for Feb, released in Mar) then we'd target this November. We could shift to a 12 months candence after that. - February: That's 12 months from the Nautilus target. - March: That's 12 months from when Nautilus was *actually* released.
November is nice in the sense that we'd wrap things up before the holidays. It's less good in that users may not be inclined to install the new release when many developers will be less available in December.
February kind of sucked in that the scramble to get the last few things done happened during the holidays. OTOH, we should be doing what we can to avoid such scrambles, so that might not be something we should factor in. March may be a bit more balanced, with a solid 3 months before when people are productive, and 3 months after before they disappear on holiday to address any post-release issues.
People tend to be somewhat less available over the summer months due to holidays etc, so an early or late summer release might also be less than ideal.
Thoughts? If we can narrow it down to a few options maybe we could do a poll to gauge user preferences.
Thanks! sage
[1] https://twitter.com/larsmb/status/1130010208971952129
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
I think 12 months cycle is much better from the cluster operations perspective. I also like March as a release month as well. ________________________________ From: ceph-users <ceph-users-bounces@lists.ceph.com> on behalf of Sage Weil <sage@newdream.net> Sent: Thursday, 6 June 2019 1:57 AM To: ceph-users@ceph.com; ceph-devel@vger.kernel.org; dev@ceph.io Subject: [ceph-users] Changing the release cadence Hi everyone, Since luminous, we have had the follow release cadence and policy: - release every 9 months - maintain backports for the last two releases - enable upgrades to move either 1 or 2 releases heads (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...) This has mostly worked out well, except that the mimic release received less attention that we wanted due to the fact that multiple downstream Ceph products (from Red Has and SUSE) decided to based their next release on nautilus. Even though upstream every release is an "LTS" release, as a practical matter mimic got less attention than luminous or nautilus. We've had several requests/proposals to shift to a 12 month cadence. This has several advantages: - Stable/conservative clusters only have to be upgraded every 2 years (instead of every 18 months) - Yearly releases are more likely to intersect with downstream distribution release (e.g., Debian). In the past there have been problems where the Ceph releases included in consecutive releases of a distro weren't easily upgradeable. - Vendors that make downstream Ceph distributions/products tend to release yearly. Aligning with those vendors means they are more likely to productize *every* Ceph release. This will help make every Ceph release an "LTS" release (not just in name but also in terms of maintenance attention). So far the balance of opinion seems to favor a shift to a 12 month cycle[1], especially among developers, so it seems pretty likely we'll make that shift. (If you do have strong concerns about such a move, now is the time to raise them.) That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time *every year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully! A few options: - November: If we release Octopus 9 months from the Nautilus release (planned for Feb, released in Mar) then we'd target this November. We could shift to a 12 months candence after that. - February: That's 12 months from the Nautilus target. - March: That's 12 months from when Nautilus was *actually* released. November is nice in the sense that we'd wrap things up before the holidays. It's less good in that users may not be inclined to install the new release when many developers will be less available in December. February kind of sucked in that the scramble to get the last few things done happened during the holidays. OTOH, we should be doing what we can to avoid such scrambles, so that might not be something we should factor in. March may be a bit more balanced, with a solid 3 months before when people are productive, and 3 months after before they disappear on holiday to address any post-release issues. People tend to be somewhat less available over the summer months due to holidays etc, so an early or late summer release might also be less than ideal. Thoughts? If we can narrow it down to a few options maybe we could do a poll to gauge user preferences. Thanks! sage [1] https://twitter.com/larsmb/status/1130010208971952129 _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
On Wed, 5 Jun 2019, Sage Weil wrote:
That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time *every year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully!
I've put up a twitter poll: https://twitter.com/liewegas/status/1140655233430970369 Thanks! sage
This was a little long to respond with on Twitter, so I thought I'd share my thoughts here. I love the idea of a 12 month cadence. I like October because admins aren't upgrading production within the first few months of a new release. It gives it plenty of time to be stable for the OS distros as well as giving admins something low-key to work on over the holidays with testing the new releases in stage/QA. On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage@newdream.net> wrote:
On Wed, 5 Jun 2019, Sage Weil wrote:
That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time *every year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks! sage _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Hi, I didn't bother to create a twitter account just to be able to participate in the poll.. so.. please count me in for October. Regards, Daniel
On Mon, Jun 17, 2019 at 4:09 PM David Turner <drakonstein@gmail.com> wrote:
This was a little long to respond with on Twitter, so I thought I'd share my thoughts here. I love the idea of a 12 month cadence. I like October because admins aren't upgrading production within the first few months of a new release. It gives it plenty of time to be stable for the OS distros as well as giving admins something low-key to work on over the holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release right on time as long as I can remember. Realistically, if we set October, we are probably going to get into November/December. For example, Nautilus was set to release in February and we got it out late in late March (Almost April) Would love to see more of a discussion around solving the problem of releasing when we say we are going to - so that we can then choose what the cadence is.
On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage@newdream.net> wrote:
On Wed, 5 Jun 2019, Sage Weil wrote:
That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time *every year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks! sage _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
I think March is a good idea. On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza <adeza@redhat.com> wrote:
On Mon, Jun 17, 2019 at 4:09 PM David Turner <drakonstein@gmail.com> wrote:
This was a little long to respond with on Twitter, so I thought I'd
share my thoughts here. I love the idea of a 12 month cadence. I like October because admins aren't upgrading production within the first few months of a new release. It gives it plenty of time to be stable for the OS distros as well as giving admins something low-key to work on over the holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release right on time as long as I can remember. Realistically, if we set October, we are probably going to get into November/December.
For example, Nautilus was set to release in February and we got it out late in late March (Almost April)
Would love to see more of a discussion around solving the problem of releasing when we say we are going to - so that we can then choose what the cadence is.
On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage@newdream.net> wrote:
On Wed, 5 Jun 2019, Sage Weil wrote:
That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time
*every
year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks! sage _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-- Alfonso Martínez Senior Software Engineer, Ceph Storage Red Hat <https://www.redhat.com> <https://red.ht/sig>
On Wed, 26 Jun 2019, Alfonso Martinez Hidalgo wrote:
I think March is a good idea.
Spring had a slight edge over fall in the twitter poll (for whatever that's worth). I see the appeal for fall when it comes to down time for retailers, but as a practical matter for Octopus specifically, a target of say October means freezing in August, which means we only have 2 more months of development time. I'm worried that will turn Octopus in another weak (aka lightly adopted) release. March would mean freezing in January again, which would give us July to Dec... 6 more months. sage
On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza <adeza@redhat.com> wrote:
On Mon, Jun 17, 2019 at 4:09 PM David Turner <drakonstein@gmail.com> wrote:
This was a little long to respond with on Twitter, so I thought I'd
share my thoughts here. I love the idea of a 12 month cadence. I like October because admins aren't upgrading production within the first few months of a new release. It gives it plenty of time to be stable for the OS distros as well as giving admins something low-key to work on over the holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release right on time as long as I can remember. Realistically, if we set October, we are probably going to get into November/December.
For example, Nautilus was set to release in February and we got it out late in late March (Almost April)
Would love to see more of a discussion around solving the problem of releasing when we say we are going to - so that we can then choose what the cadence is.
On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage@newdream.net> wrote:
On Wed, 5 Jun 2019, Sage Weil wrote:
That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time
*every
year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks! sage _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Alfonso Martínez
Senior Software Engineer, Ceph Storage
Red Hat <https://www.redhat.com> <https://red.ht/sig>
Hi everyone, We talked a bit about this during the CLT meeting this morning. How about the following proposal: - Target release date of Mar 1 each year. - Target freeze in Dec. That will allow us to use the holidays to do a lot of testing when the lab infrastructure tends to be somewhat idle. If we get an early build out at the point of the freeze (or even earlier), perhaps this capture some of the time that the retailers have during their lockdown to identify structural issues with release. It is probably better to do more of this testing at this point in the cycle so that we have time to properly fix any big issues (like performance or scaling regressions). It is of course a challenge to motivate testing on something that is too far from the final a release, but we can try. This avoids an abbreviated octopus cycle, and avoids placing August (which also often has people out for vacations) right in the middle of the lead-up to the freeze. Thoughts? sage On Wed, 26 Jun 2019, Sage Weil wrote:
On Wed, 26 Jun 2019, Alfonso Martinez Hidalgo wrote:
I think March is a good idea.
Spring had a slight edge over fall in the twitter poll (for whatever that's worth). I see the appeal for fall when it comes to down time for retailers, but as a practical matter for Octopus specifically, a target of say October means freezing in August, which means we only have 2 more months of development time. I'm worried that will turn Octopus in another weak (aka lightly adopted) release.
March would mean freezing in January again, which would give us July to Dec... 6 more months.
sage
On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza <adeza@redhat.com> wrote:
On Mon, Jun 17, 2019 at 4:09 PM David Turner <drakonstein@gmail.com> wrote:
This was a little long to respond with on Twitter, so I thought I'd
share my thoughts here. I love the idea of a 12 month cadence. I like October because admins aren't upgrading production within the first few months of a new release. It gives it plenty of time to be stable for the OS distros as well as giving admins something low-key to work on over the holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release right on time as long as I can remember. Realistically, if we set October, we are probably going to get into November/December.
For example, Nautilus was set to release in February and we got it out late in late March (Almost April)
Would love to see more of a discussion around solving the problem of releasing when we say we are going to - so that we can then choose what the cadence is.
On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage@newdream.net> wrote:
On Wed, 5 Jun 2019, Sage Weil wrote:
That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time
*every
year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks! sage _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Alfonso Martínez
Senior Software Engineer, Ceph Storage
Red Hat <https://www.redhat.com> <https://red.ht/sig>
March seems sensible to me for the reasons you stated. If a release gets delayed, I'd prefer it to be on the spring side of Christmas (again for the reasons already mentioned). That aside, I'm now very impatient to install Octopus on my 8-node cluster. : ) On Wed, 26 Jun 2019 at 15:46, Sage Weil <sweil@redhat.com> wrote:
Hi everyone,
We talked a bit about this during the CLT meeting this morning. How about the following proposal:
- Target release date of Mar 1 each year. - Target freeze in Dec. That will allow us to use the holidays to do a lot of testing when the lab infrastructure tends to be somewhat idle.
If we get an early build out at the point of the freeze (or even earlier), perhaps this capture some of the time that the retailers have during their lockdown to identify structural issues with release. It is probably better to do more of this testing at this point in the cycle so that we have time to properly fix any big issues (like performance or scaling regressions). It is of course a challenge to motivate testing on something that is too far from the final a release, but we can try.
This avoids an abbreviated octopus cycle, and avoids placing August (which also often has people out for vacations) right in the middle of the lead-up to the freeze.
Thoughts? sage
On Wed, 26 Jun 2019, Sage Weil wrote:
On Wed, 26 Jun 2019, Alfonso Martinez Hidalgo wrote:
I think March is a good idea.
Spring had a slight edge over fall in the twitter poll (for whatever that's worth). I see the appeal for fall when it comes to down time for retailers, but as a practical matter for Octopus specifically, a target of say October means freezing in August, which means we only have 2 more months of development time. I'm worried that will turn Octopus in another weak (aka lightly adopted) release.
March would mean freezing in January again, which would give us July to Dec... 6 more months.
sage
On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza <adeza@redhat.com> wrote:
On Mon, Jun 17, 2019 at 4:09 PM David Turner <drakonstein@gmail.com> wrote:
This was a little long to respond with on Twitter, so I thought I'd
share my thoughts here. I love the idea of a 12 month cadence. I like October because admins aren't upgrading production within the first
months of a new release. It gives it plenty of time to be stable for
distros as well as giving admins something low-key to work on over
few the OS the
holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release right on time as long as I can remember. Realistically, if we set October, we are probably going to get into November/December.
For example, Nautilus was set to release in February and we got it out late in late March (Almost April)
Would love to see more of a discussion around solving the problem of releasing when we say we are going to - so that we can then choose what the cadence is.
On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage@newdream.net>
wrote:
On Wed, 5 Jun 2019, Sage Weil wrote: > That brings us to an important decision: what time of year
should we
> release? Once we pick the timing, we'll be releasing at that time *every > year* for each release (barring another schedule shift, which we want to > avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks! sage _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Alfonso Martínez
Senior Software Engineer, Ceph Storage
Red Hat <https://www.redhat.com> <https://red.ht/sig> _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
On 2019-06-26T14:45:31, Sage Weil <sweil@redhat.com> wrote: Hi Sage, I think that makes sense. I'd have preferred the Oct/Nov target, but that'd have made Octopus quite short. Unsure whether freezing in December with a release in March is too long though. But given how much people scramble, setting that as a goal probably will help with stabilization. I'm also hoping that one day, we can move towards a more agile continuously integration model (like the Linux kernel does) instead of massive yearly forklifts. But hey, that's just me ;-) Regards, Lars -- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
On Tue, 25 Jun 2019, Alfredo Deza wrote:
On Mon, Jun 17, 2019 at 4:09 PM David Turner <drakonstein@gmail.com> wrote:
This was a little long to respond with on Twitter, so I thought I'd share my thoughts here. I love the idea of a 12 month cadence. I like October because admins aren't upgrading production within the first few months of a new release. It gives it plenty of time to be stable for the OS distros as well as giving admins something low-key to work on over the holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release right on time as long as I can remember. Realistically, if we set October, we are probably going to get into November/December.
For example, Nautilus was set to release in February and we got it out late in late March (Almost April)
Would love to see more of a discussion around solving the problem of releasing when we say we are going to - so that we can then choose what the cadence is.
I think the "on time" part is solveable. We should just use the amount of time between take the previous release's freeze date and the target release date and go with that. It is a bit fuzzy because I left it up to the leads how they handle the freeze, but I think mid-Januaray is about right (in reality we waiting longer than that for lots of RADOS stuff). v14.2.0 was Mar 18, so ~2 months. The cadence is really separate from that, though: even if every release were 2 full months late, if we start with the same target it's still a 1 year cycle. sage
On Wed, Jun 5, 2019 at 11:58 AM Sage Weil <sage@newdream.net> wrote:
...
This has mostly worked out well, except that the mimic release received less attention that we wanted due to the fact that multiple downstream Ceph products (from Red Has and SUSE) decided to based their next release on nautilus. Even though upstream every release is an "LTS" release, as a practical matter mimic got less attention than luminous or nautilus.
Speaking as (one of) the Ceph packager(s) in Fedora: 1) I have been told (don't recall by who now) that odd numbered releases are experimental or short term support or red haired step child releases. 2) even though there have been ceph-{10,11,12}.0.z tarballs on http://download.ceph.com/tarballs/ in the past, there still isn't a ceph-15.0.0.tar.gz there. And I was told that there wouldn't be one. Yes, I'm aware that X.0.z are alpha releases, X.1.z are beta releases, and X.2.z are GA releases. 3) ceph-13 was skipped over in Fedora because I could never get it to build in Fedora in the limited time I make to do upstream packaging of Ceph and other things (because it's not my $dayjob.) And there seemed to be no interest on the part of anyone in the Ceph devel community to fix it (because see #1?) If Octopus is really an LTS release like all the others, and you want bleeding edge users to test/use it and give early feedback, then Fedora is probably one of the better places to get that feedback. FWIW, In the absence of a ceph-14.0.z.tar.gz from http://download.ceph.com/tarballs/ branto came up with one (possibly from https://github.com/ceph/ceph/releases/...???) but I haven't had any luck building ceph-15 with the tarball from there. My 2¢.
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
On Wed, Jun 5, 2019 at 11:58 AM Sage Weil <sage@newdream.net> wrote:
...
This has mostly worked out well, except that the mimic release received less attention that we wanted due to the fact that multiple downstream Ceph products (from Red Has and SUSE) decided to based their next release on nautilus. Even though upstream every release is an "LTS" release, as a practical matter mimic got less attention than luminous or nautilus.
Speaking as (one of) the Ceph packager(s) in Fedora:
1) I have been told (don't recall by who now) that odd numbered releases are experimental or short term support or red haired step child releases.
In principle that hasn't been the case since 12.2.z (luminous). 13 has gotten somewhat less backport attention because the people doing backports work for Red Hat and SUSE and neither shipped a product based on 13.
2) even though there have been ceph-{10,11,12}.0.z tarballs on http://download.ceph.com/tarballs/ in the past, there still isn't a ceph-15.0.0.tar.gz there. And I was told that there wouldn't be one. Yes, I'm aware that X.0.z are alpha releases, X.1.z are beta releases, and X.2.z are GA releases.
In the past we haven't published tarballs until x.1.z (rc), since the dev checkpoints are not meant to be used by anyone other than developers. (They're really just a 'pause and make sure the qa suites are nice and clean', nothing more.)
3) ceph-13 was skipped over in Fedora because I could never get it to build in Fedora in the limited time I make to do upstream packaging of Ceph and other things (because it's not my $dayjob.) And there seemed to be no interest on the part of anyone in the Ceph devel community to fix it (because see #1?)
If Octopus is really an LTS release like all the others, and you want bleeding edge users to test/use it and give early feedback, then Fedora is probably one of the better places to get that feedback.
I think the first release worth testing outside of the dev community is the release candidate. I don't like the idea of having any distro carry an untested dev checkpoint or else someone will lose data... even the rc should be tested cautiously and, since it is only relevant for a week or two, I'm not sure that distros can play much of a role there? sage
FWIW, In the absence of a ceph-14.0.z.tar.gz from http://download.ceph.com/tarballs/ branto came up with one (possibly from https://github.com/ceph/ceph/releases/...???) but I haven't had any luck building ceph-15 with the tarball from there.
My 2¢.
On Mon, Jul 15, 2019 at 10:10 AM Sage Weil <sage@newdream.net> wrote:
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
If Octopus is really an LTS release like all the others, and you want bleeding edge users to test/use it and give early feedback, then Fedora
is
probably one of the better places to get that feedback.
I think the first release worth testing outside of the dev community is the release candidate. I don't like the idea of having any distro carry an untested dev checkpoint or else someone will lose data... even the rc should be tested cautiously and, since it is only relevant for a week or two, I'm not sure that distros can play much of a role there?
To be clear, when I talk about packaging a new version Ceph, it starts out by _only_ going into Fedora Rawhide. It would be extremely foolish, IMO, for anyone to run Rawhide for anything that's mission critical. Sometimes it takes a long time to work through build and packaging issues. The switch to gcc-8 and gcc-9 are good illustrations of the kinds of issues that we, as packagers, run into. Those kinds of things are why I — at least — like to get as early a start as I can. Even short-lived release candidates are useful stepping stones to the eventual GA. Personally I'd say that any fears anyone has of people losing data by using an early dev checkpoint of Ceph on Rawhide are probably a teeny bit misplaced. My 2.5¢ worth. -- Kaleb
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
On Mon, Jul 15, 2019 at 10:10 AM Sage Weil <sage@newdream.net> wrote:
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
If Octopus is really an LTS release like all the others, and you want bleeding edge users to test/use it and give early feedback, then Fedora
is
probably one of the better places to get that feedback.
I think the first release worth testing outside of the dev community is the release candidate. I don't like the idea of having any distro carry an untested dev checkpoint or else someone will lose data... even the rc should be tested cautiously and, since it is only relevant for a week or two, I'm not sure that distros can play much of a role there?
To be clear, when I talk about packaging a new version Ceph, it starts out by _only_ going into Fedora Rawhide. It would be extremely foolish, IMO, for anyone to run Rawhide for anything that's mission critical.
Sometimes it takes a long time to work through build and packaging issues. The switch to gcc-8 and gcc-9 are good illustrations of the kinds of issues that we, as packagers, run into. Those kinds of things are why I — at least — like to get as early a start as I can. Even short-lived release candidates are useful stepping stones to the eventual GA.
Personally I'd say that any fears anyone has of people losing data by using an early dev checkpoint of Ceph on Rawhide are probably a teeny bit misplaced.
Okay, it sounds like the rc x.1.z releases are a good fit, then. They should start about ~2 months before the first stable release. I'm not so sure about the dev checkpoints since they are quite arbitrary. I suspect less effort would be needed to just do a manual build a few times partway through the cycle (e.g., now), identify any issues, and open PRs with build fixes. sage
Hey, On 15.07.19 09:58, Kaleb Keithley wrote:
Speaking as (one of) the Ceph packager(s) in Fedora:
Arch Linux packager for Ceph here o/
If Octopus is really an LTS release like all the others, and you want bleeding edge users to test/use it and give early feedback, then Fedora is probably one of the better places to get that feedback.
FWIW, In the absence of a ceph-14.0.z.tar.gz from http://download.ceph.com/tarballs/ branto came up with one (possibly from https://github.com/ceph/ceph/releases/...???) but I haven't had any luck building ceph-15 with the tarball from there.
I've been watching the RSS feed of the GitHub releases and use the GitHub release tarballs for building it. Due to unforseen complications and lack of time (similar situation like you) the last packaged version of ceph in the Arch Linux repos is 13.2.1 right now. Over the last couple of weeks I have spent numerous hours and worked with people in oftc.#ceph-devel to work on the issues I was experiencing while building it. Right now I'm finalizing the 14.2.1 build and have some last kinks to work out, for some reason the zstd compression tests are failing for me, gonna look into that over the coming days. Seeing that Arch Linux and Fedora should have a few more things common then, let's say Ubuntu or Debian, I would like to offer my help however I can. I'm not that familiar with building packages on/for Fedora but I've had my fair share of debugging building ceph with bleeding edge components (gcc, cmake, boost, python) and could possibly help out there. Feel free to get in touch, maybe we can help each other out. Cheers, Thore PS: I was thinking about maybe getting Arch Linux (and Fedora) onboarded to the ceph-dev-build matrix on jenkins.ceph.com. Thinking about the possibility to spot arising issues and/or incompatibilities with upstream / bleeding edge compilers, helpers and dependencies that should have common intereset with Ceph upstream. Although I haven't talked to anyone upstream about this and am just throwing it out here. -- Thore Bödecker GPG ID: 0xD622431AF8DB80F3 GPG FP: 0F96 559D 3556 24FC 2226 A864 D622 431A F8DB 80F3
12 months sounds good to me, I like the idea of march as well since we plan on doing upgrades in June/July each year. Gives it time to be discussed and marinate before we decide to upgrade. -Brent -----Original Message----- From: ceph-users <ceph-users-bounces@lists.ceph.com> On Behalf Of Sage Weil Sent: Wednesday, June 5, 2019 11:58 AM To: ceph-users@ceph.com; ceph-devel@vger.kernel.org; dev@ceph.io Subject: [ceph-users] Changing the release cadence Hi everyone, Since luminous, we have had the follow release cadence and policy: - release every 9 months - maintain backports for the last two releases - enable upgrades to move either 1 or 2 releases heads (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...) This has mostly worked out well, except that the mimic release received less attention that we wanted due to the fact that multiple downstream Ceph products (from Red Has and SUSE) decided to based their next release on nautilus. Even though upstream every release is an "LTS" release, as a practical matter mimic got less attention than luminous or nautilus. We've had several requests/proposals to shift to a 12 month cadence. This has several advantages: - Stable/conservative clusters only have to be upgraded every 2 years (instead of every 18 months) - Yearly releases are more likely to intersect with downstream distribution release (e.g., Debian). In the past there have been problems where the Ceph releases included in consecutive releases of a distro weren't easily upgradeable. - Vendors that make downstream Ceph distributions/products tend to release yearly. Aligning with those vendors means they are more likely to productize *every* Ceph release. This will help make every Ceph release an "LTS" release (not just in name but also in terms of maintenance attention). So far the balance of opinion seems to favor a shift to a 12 month cycle[1], especially among developers, so it seems pretty likely we'll make that shift. (If you do have strong concerns about such a move, now is the time to raise them.) That brings us to an important decision: what time of year should we release? Once we pick the timing, we'll be releasing at that time *every year* for each release (barring another schedule shift, which we want to avoid), so let's choose carefully! A few options: - November: If we release Octopus 9 months from the Nautilus release (planned for Feb, released in Mar) then we'd target this November. We could shift to a 12 months candence after that. - February: That's 12 months from the Nautilus target. - March: That's 12 months from when Nautilus was *actually* released. November is nice in the sense that we'd wrap things up before the holidays. It's less good in that users may not be inclined to install the new release when many developers will be less available in December. February kind of sucked in that the scramble to get the last few things done happened during the holidays. OTOH, we should be doing what we can to avoid such scrambles, so that might not be something we should factor in. March may be a bit more balanced, with a solid 3 months before when people are productive, and 3 months after before they disappear on holiday to address any post-release issues. People tend to be somewhat less available over the summer months due to holidays etc, so an early or late summer release might also be less than ideal. Thoughts? If we can narrow it down to a few options maybe we could do a poll to gauge user preferences. Thanks! sage [1] https://twitter.com/larsmb/status/1130010208971952129 _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
participants (14)
-
Alexandre DERUMIER
-
Alfonso Martinez Hidalgo
-
Alfredo Deza
-
Bob Farrell
-
Brent Kennedy
-
Chris Taylor
-
Daniel Baumann
-
David Turner
-
Kaleb Keithley
-
Lars Marowsky-Bree
-
Linh Vu
-
Sage Weil
-
Sage Weil
-
Thore Bödecker