5 Jun
2019
5 Jun
'19
9:27 p.m.
Hi everyone,
Since luminous, we have had the follow release cadence and policy:
- release every 9 months
- maintain backports for the last two releases
- enable upgrades to move either 1 or 2 releases heads
(e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)
This has mostly worked out well, except that the mimic release received
less attention that we wanted due to the fact that multiple downstream
Ceph products (from Red Has and SUSE) decided to based their next release
on nautilus. Even though upstream every release is an "LTS" release, as a
practical matter mimic got less attention than luminous or nautilus.
We've had several requests/proposals to shift to a 12 month cadence. This
has several advantages:
- Stable/conservative clusters only have to be upgraded every 2 years
(instead of every 18 months)
- Yearly releases are more likely to intersect with downstream
distribution release (e.g., Debian). In the past there have been
problems where the Ceph releases included in consecutive releases of a
distro weren't easily upgradeable.
- Vendors that make downstream Ceph distributions/products tend to
release yearly. Aligning with those vendors means they are more likely
to productize *every* Ceph release. This will help make every Ceph
release an "LTS" release (not just in name but also in terms of
maintenance attention).
So far the balance of opinion seems to favor a shift to a 12 month
cycle[1], especially among developers, so it seems pretty likely we'll
make that shift. (If you do have strong concerns about such a move, now
is the time to raise them.)
That brings us to an important decision: what time of year should we
release? Once we pick the timing, we'll be releasing at that time *every
year* for each release (barring another schedule shift, which we want to
avoid), so let's choose carefully!
A few options:
- November: If we release Octopus 9 months from the Nautilus release
(planned for Feb, released in Mar) then we'd target this November. We
could shift to a 12 months candence after that.
- February: That's 12 months from the Nautilus target.
- March: That's 12 months from when Nautilus was *actually* released.
November is nice in the sense that we'd wrap things up before the
holidays. It's less good in that users may not be inclined to install the
new release when many developers will be less available in December.
February kind of sucked in that the scramble to get the last few things
done happened during the holidays. OTOH, we should be doing what we can
to avoid such scrambles, so that might not be something we should factor
in. March may be a bit more balanced, with a solid 3 months before when
people are productive, and 3 months after before they disappear on holiday
to address any post-release issues.
People tend to be somewhat less available over the summer months due to
holidays etc, so an early or late summer release might also be less than
ideal.
Thoughts? If we can narrow it down to a few options maybe we could do a
poll to gauge user preferences.
Thanks!
sage
[1] https://twitter.com/larsmb/status/1130010208971952129
5 Jun
5 Jun
11:07 p.m.
New subject: [ceph-users] Changing the release cadence
On 6/5/19 5:57 PM, Sage Weil wrote:
So far the balance of opinion seems to favor a shift
to a 12 month
cycle [...] it seems pretty likely we'll make that shift.
thanks, much appreciated (from an cluster operating point of view).
Thoughts?
GNOME and a few others are doing April and October releases which seems
balanced and to be good timing for most people; personally I prefer
spring rather than autum for upgrades, hence.. would suggest April.
Regards,
Daniel
6 Jun
6 Jun
12:46 a.m.
Hi,
>- November: If we release Octopus 9 months from the
Nautilus release
>(planned for Feb, released in Mar) then we'd target this November. We
>could shift to a 12 months candence after that.
For the 2 last debian releases, the freeze was around january-february,
november seem to be a good time for ceph release.
----- Mail original -----
De: "Sage Weil" <sage(a)newdream.net>
À: "ceph-users" <ceph-users(a)ceph.com>om>, "ceph-devel"
<ceph-devel(a)vger.kernel.org>rg>, dev(a)ceph.io
Envoyé: Mercredi 5 Juin 2019 17:57:52
Objet: Changing the release cadence
Hi everyone,
Since luminous, we have had the follow release cadence and policy:
- release every 9 months
- maintain backports for the last two releases
- enable upgrades to move either 1 or 2 releases heads
(e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)
This has mostly worked out well, except that the mimic release received
less attention that we wanted due to the fact that multiple downstream
Ceph products (from Red Has and SUSE) decided to based their next release
on nautilus. Even though upstream every release is an "LTS" release, as a
practical matter mimic got less attention than luminous or nautilus.
We've had several requests/proposals to shift to a 12 month cadence. This
has several advantages:
- Stable/conservative clusters only have to be upgraded every 2 years
(instead of every 18 months)
- Yearly releases are more likely to intersect with downstream
distribution release (e.g., Debian). In the past there have been
problems where the Ceph releases included in consecutive releases of a
distro weren't easily upgradeable.
- Vendors that make downstream Ceph distributions/products tend to
release yearly. Aligning with those vendors means they are more likely
to productize *every* Ceph release. This will help make every Ceph
release an "LTS" release (not just in name but also in terms of
maintenance attention).
So far the balance of opinion seems to favor a shift to a 12 month
cycle[1], especially among developers, so it seems pretty likely we'll
make that shift. (If you do have strong concerns about such a move, now
is the time to raise them.)
That brings us to an important decision: what time of year should we
release? Once we pick the timing, we'll be releasing at that time *every
year* for each release (barring another schedule shift, which we want to
avoid), so let's choose carefully!
A few options:
- November: If we release Octopus 9 months from the Nautilus release
(planned for Feb, released in Mar) then we'd target this November. We
could shift to a 12 months candence after that.
- February: That's 12 months from the Nautilus target.
- March: That's 12 months from when Nautilus was *actually* released.
November is nice in the sense that we'd wrap things up before the
holidays. It's less good in that users may not be inclined to install the
new release when many developers will be less available in December.
February kind of sucked in that the scramble to get the last few things
done happened during the holidays. OTOH, we should be doing what we can
to avoid such scrambles, so that might not be something we should factor
in. March may be a bit more balanced, with a solid 3 months before when
people are productive, and 3 months after before they disappear on holiday
to address any post-release issues.
People tend to be somewhat less available over the summer months due to
holidays etc, so an early or late summer release might also be less than
ideal.
Thoughts? If we can narrow it down to a few options maybe we could do a
poll to gauge user preferences.
Thanks!
sage
[1] https://twitter.com/larsmb/status/1130010208971952129
1:57 a.m.
New subject: [ceph-users] Changing the release cadence
It seems like since the change to the 9 months cadence it has been bumpy
for the Debian based installs. Changing to a 12 month cadence sounds
like a good idea. Perhaps some Debian maintainers can suggest a good
month for them to get the packages in time for their release cycle.
On 2019-06-05 12:16 pm, Alexandre DERUMIER wrote:
> Hi,
>
>
>>> - November: If we release Octopus 9 months from the Nautilus release
>>> (planned for Feb, released in Mar) then we'd target this November. We
>>> could shift to a 12 months candence after that.
>
> For the 2 last debian releases, the freeze was around january-february,
> november seem to be a good time for ceph release.
>
> ----- Mail original -----
> De: "Sage Weil" <sage(a)newdream.net>
> À: "ceph-users" <ceph-users(a)ceph.com>om>, "ceph-devel"
> <ceph-devel(a)vger.kernel.org>rg>, dev(a)ceph.io
> Envoyé: Mercredi 5 Juin 2019 17:57:52
> Objet: Changing the release cadence
>
> Hi everyone,
>
> Since luminous, we have had the follow release cadence and policy:
> - release every 9 months
> - maintain backports for the last two releases
> - enable upgrades to move either 1 or 2 releases heads
> (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus;
> ...)
>
> This has mostly worked out well, except that the mimic release received
> less attention that we wanted due to the fact that multiple downstream
> Ceph products (from Red Has and SUSE) decided to based their next
> release
> on nautilus. Even though upstream every release is an "LTS" release, as
> a
> practical matter mimic got less attention than luminous or nautilus.
>
> We've had several requests/proposals to shift to a 12 month cadence.
> This
> has several advantages:
>
> - Stable/conservative clusters only have to be upgraded every 2 years
> (instead of every 18 months)
> - Yearly releases are more likely to intersect with downstream
> distribution release (e.g., Debian). In the past there have been
> problems where the Ceph releases included in consecutive releases of a
> distro weren't easily upgradeable.
> - Vendors that make downstream Ceph distributions/products tend to
> release yearly. Aligning with those vendors means they are more likely
> to productize *every* Ceph release. This will help make every Ceph
> release an "LTS" release (not just in name but also in terms of
> maintenance attention).
>
> So far the balance of opinion seems to favor a shift to a 12 month
> cycle[1], especially among developers, so it seems pretty likely we'll
> make that shift. (If you do have strong concerns about such a move, now
> is the time to raise them.)
>
> That brings us to an important decision: what time of year should we
> release? Once we pick the timing, we'll be releasing at that time
> *every
> year* for each release (barring another schedule shift, which we want
> to
> avoid), so let's choose carefully!
>
> A few options:
>
> - November: If we release Octopus 9 months from the Nautilus release
> (planned for Feb, released in Mar) then we'd target this November. We
> could shift to a 12 months candence after that.
> - February: That's 12 months from the Nautilus target.
> - March: That's 12 months from when Nautilus was *actually* released.
>
> November is nice in the sense that we'd wrap things up before the
> holidays. It's less good in that users may not be inclined to install
> the
> new release when many developers will be less available in December.
>
> February kind of sucked in that the scramble to get the last few things
> done happened during the holidays. OTOH, we should be doing what we can
> to avoid such scrambles, so that might not be something we should
> factor
> in. March may be a bit more balanced, with a solid 3 months before when
> people are productive, and 3 months after before they disappear on
> holiday
> to address any post-release issues.
>
> People tend to be somewhat less available over the summer months due to
> holidays etc, so an early or late summer release might also be less
> than
> ideal.
>
> Thoughts? If we can narrow it down to a few options maybe we could do a
> poll to gauge user preferences.
>
> Thanks!
> sage
>
>
> [1] https://twitter.com/larsmb/status/1130010208971952129
>
> _______________________________________________
> ceph-users mailing list
> ceph-users(a)lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
6:01 a.m.
New subject: [ceph-users] Changing the release cadence
I think 12 months cycle is much better from the cluster operations perspective. I also
like March as a release month as well.
________________________________
From: ceph-users <ceph-users-bounces(a)lists.ceph.com> on behalf of Sage Weil
<sage(a)newdream.net>
Sent: Thursday, 6 June 2019 1:57 AM
To: ceph-users(a)ceph.com; ceph-devel(a)vger.kernel.org; dev(a)ceph.io
Subject: [ceph-users] Changing the release cadence
Hi everyone,
Since luminous, we have had the follow release cadence and policy:
- release every 9 months
- maintain backports for the last two releases
- enable upgrades to move either 1 or 2 releases heads
(e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)
This has mostly worked out well, except that the mimic release received
less attention that we wanted due to the fact that multiple downstream
Ceph products (from Red Has and SUSE) decided to based their next release
on nautilus. Even though upstream every release is an "LTS" release, as a
practical matter mimic got less attention than luminous or nautilus.
We've had several requests/proposals to shift to a 12 month cadence. This
has several advantages:
- Stable/conservative clusters only have to be upgraded every 2 years
(instead of every 18 months)
- Yearly releases are more likely to intersect with downstream
distribution release (e.g., Debian). In the past there have been
problems where the Ceph releases included in consecutive releases of a
distro weren't easily upgradeable.
- Vendors that make downstream Ceph distributions/products tend to
release yearly. Aligning with those vendors means they are more likely
to productize *every* Ceph release. This will help make every Ceph
release an "LTS" release (not just in name but also in terms of
maintenance attention).
So far the balance of opinion seems to favor a shift to a 12 month
cycle[1], especially among developers, so it seems pretty likely we'll
make that shift. (If you do have strong concerns about such a move, now
is the time to raise them.)
That brings us to an important decision: what time of year should we
release? Once we pick the timing, we'll be releasing at that time *every
year* for each release (barring another schedule shift, which we want to
avoid), so let's choose carefully!
A few options:
- November: If we release Octopus 9 months from the Nautilus release
(planned for Feb, released in Mar) then we'd target this November. We
could shift to a 12 months candence after that.
- February: That's 12 months from the Nautilus target.
- March: That's 12 months from when Nautilus was *actually* released.
November is nice in the sense that we'd wrap things up before the
holidays. It's less good in that users may not be inclined to install the
new release when many developers will be less available in December.
February kind of sucked in that the scramble to get the last few things
done happened during the holidays. OTOH, we should be doing what we can
to avoid such scrambles, so that might not be something we should factor
in. March may be a bit more balanced, with a solid 3 months before when
people are productive, and 3 months after before they disappear on holiday
to address any post-release issues.
People tend to be somewhat less available over the summer months due to
holidays etc, so an early or late summer release might also be less than
ideal.
Thoughts? If we can narrow it down to a few options maybe we could do a
poll to gauge user preferences.
Thanks!
sage
[1] https://twitter.com/larsmb/status/1130010208971952129
_______________________________________________
ceph-users mailing list
ceph-users(a)lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
17 Jun
17 Jun
9:51 p.m.
On Wed, 5 Jun 2019, Sage Weil wrote:
That brings us to an important decision: what time of
year should we
release? Once we pick the timing, we'll be releasing at that time *every
year* for each release (barring another schedule shift, which we want to
avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks!
sage
18 Jun
18 Jun
1:38 a.m.
New subject: [ceph-users] Changing the release cadence
This was a little long to respond with on Twitter, so I thought I'd share
my thoughts here. I love the idea of a 12 month cadence. I like October
because admins aren't upgrading production within the first few months of a
new release. It gives it plenty of time to be stable for the OS distros as
well as giving admins something low-key to work on over the holidays with
testing the new releases in stage/QA.
On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage(a)newdream.net> wrote:
On Wed, 5 Jun 2019, Sage Weil wrote:
That brings us to an important decision: what
time of year should we
release? Once we pick the timing, we'll be releasing at that time
*every
year* for each release (barring another schedule
shift, which we want to
avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks!
sage
_______________________________________________
ceph-users mailing list
ceph-users(a)lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
11:09 a.m.
New subject: [ceph-users] Changing the release cadence
Hi,
I didn't bother to create a twitter account just to be able to
participate in the poll.. so.. please count me in for October.
Regards,
Daniel
25 Jun
25 Jun
8:01 p.m.
New subject: [ceph-users] Changing the release cadence
On Mon, Jun 17, 2019 at 4:09 PM David Turner <drakonstein(a)gmail.com> wrote:
This was a little long to respond with on Twitter, so I thought I'd share my thoughts
here. I love the idea of a 12 month cadence. I like October because admins aren't
upgrading production within the first few months of a new release. It gives it plenty of
time to be stable for the OS distros as well as giving admins something low-key to work on
over the holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release
right on time as long as I can remember. Realistically, if we set
October, we are probably going to get into November/December.
For example, Nautilus was set to release in February and we got it out
late in late March (Almost April)
Would love to see more of a discussion around solving the problem of
releasing when we say we are going to - so that we can then choose
what the cadence is.
>
> On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage(a)newdream.net> wrote:
>>
>> On Wed, 5 Jun 2019, Sage Weil wrote:
>> > That brings us to an important decision: what time of year should we
>> > release? Once we pick the timing, we'll be releasing at that time
*every
>> > year* for each release (barring another schedule shift, which we want to
>> > avoid), so let's choose carefully!
>>
>> I've put up a twitter poll:
>>
>> https://twitter.com/liewegas/status/1140655233430970369
>>
>> Thanks!
>> sage
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users(a)lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users(a)lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
26 Jun
26 Jun
12:03 p.m.
New subject: [ceph-users] Changing the release cadence
I think March is a good idea.
On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza <adeza(a)redhat.com> wrote:
On Mon, Jun 17, 2019 at 4:09 PM David Turner
<drakonstein(a)gmail.com>
wrote:
_______________________________________________
ceph-users mailing list
ceph-users(a)lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Alfonso Martínez
Senior Software Engineer, Ceph Storage
Red Hat <https://www.redhat.com>
<https://red.ht/sig>
This was a little long to respond with on Twitter, so I thought I'd
share my
thoughts here. I love the idea of a 12 month cadence. I like
October because admins aren't upgrading production within the first few
months of a new release. It gives it plenty of time to be stable for the OS
distros as well as giving admins something low-key to work on over the
holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release
right on time as long as I can remember. Realistically, if we set
October, we are probably going to get into November/December.
For example, Nautilus was set to release in February and we got it out
late in late March (Almost April)
Would love to see more of a discussion around solving the problem of
releasing when we say we are going to - so that we can then choose
what the cadence is.
On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage(a)newdream.net> wrote:
>
> On Wed, 5 Jun 2019, Sage Weil wrote:
> > That brings us to an important decision: what time of year should we
> > release? Once we pick the timing, we'll be releasing at that time
*every
> > year* for each release (barring another
schedule shift, which we want
to
avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks!
sage
_______________________________________________
ceph-users mailing list
ceph-users(a)lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
6:27 p.m.
New subject: [ceph-users] Changing the release cadence
On Wed, 26 Jun 2019, Alfonso Martinez Hidalgo wrote:
I think March is a good idea.
Spring had a slight edge over fall in the twitter poll (for whatever
that's worth). I see the appeal for fall when it comes to down time for
retailers, but as a practical matter for Octopus specifically, a target of
say October means freezing in August, which means we only have 2
more months of development time. I'm worried that will turn Octopus
in another weak (aka lightly adopted) release.
March would mean freezing in January again, which would give us July to
Dec... 6 more months.
sage
On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza <adeza(a)redhat.com> wrote:
On Mon, Jun 17, 2019 at 4:09 PM David Turner
<drakonstein(a)gmail.com>
wrote:
--
Alfonso Martínez
Senior Software Engineer, Ceph Storage
Red Hat <https://www.redhat.com>
<https://red.ht/sig>
This was a little long to respond with on Twitter, so I thought I'd
share my
thoughts here. I love the idea of a 12 month cadence. I like
October because admins aren't upgrading production within the first few
months of a new release. It gives it plenty of time to be stable for the OS
distros as well as giving admins something low-key to work on over the
holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release
right on time as long as I can remember. Realistically, if we set
October, we are probably going to get into November/December.
For example, Nautilus was set to release in February and we got it out
late in late March (Almost April)
Would love to see more of a discussion around solving the problem of
releasing when we say we are going to - so that we can then choose
what the cadence is.
On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage(a)newdream.net> wrote:
>
> On Wed, 5 Jun 2019, Sage Weil wrote:
> > That brings us to an important decision: what time of year should we
> > release? Once we pick the timing, we'll be releasing at that time
*every
> > year* for each release (barring another
schedule shift, which we want
to
>
avoid), so let's choose carefully!
I've put up a twitter poll:
https://twitter.com/liewegas/status/1140655233430970369
Thanks!
sage
_______________________________________________
ceph-users mailing list
ceph-users(a)lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users(a)lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
8:15 p.m.
New subject: [ceph-users] Changing the release cadence
Hi everyone,
We talked a bit about this during the CLT meeting this morning. How about
the following proposal:
- Target release date of Mar 1 each year.
- Target freeze in Dec. That will allow us to use the holidays to do a
lot of testing when the lab infrastructure tends to be somewhat idle.
If we get an early build out at the point of the freeze (or even earlier),
perhaps this capture some of the time that the retailers have during their
lockdown to identify structural issues with release. It is probably
better to do more of this testing at this point in the cycle so that we
have time to properly fix any big issues (like performance or scaling
regressions). It is of course a challenge to motivate testing on
something that is too far from the final a release, but we can try.
This avoids an abbreviated octopus cycle, and avoids placing August (which
also often has people out for vacations) right in the middle of the
lead-up to the freeze.
Thoughts?
sage
On Wed, 26 Jun 2019, Sage Weil wrote:
On Wed, 26 Jun 2019, Alfonso Martinez Hidalgo wrote:
I think March is a good idea.
Spring had a slight edge over fall in the twitter poll (for whatever
that's worth). I see the appeal for fall when it comes to down time for
retailers, but as a practical matter for Octopus specifically, a target of
say October means freezing in August, which means we only have 2
more months of development time. I'm worried that will turn Octopus
in another weak (aka lightly adopted) release.
March would mean freezing in January again, which would give us July to
Dec... 6 more months.
sage
>
> On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza <adeza(a)redhat.com> wrote:
>
> > On Mon, Jun 17, 2019 at 4:09 PM David Turner <drakonstein(a)gmail.com>
> > wrote:
> > >
> > > This was a little long to respond with on Twitter, so I thought I'd
> > share my thoughts here. I love the idea of a 12 month cadence. I like
> > October because admins aren't upgrading production within the first few
> > months of a new release. It gives it plenty of time to be stable for the OS
> > distros as well as giving admins something low-key to work on over the
> > holidays with testing the new releases in stage/QA.
> >
> > October sounds ideal, but in reality, we haven't been able to release
> > right on time as long as I can remember. Realistically, if we set
> > October, we are probably going to get into November/December.
> >
> > For example, Nautilus was set to release in February and we got it out
> > late in late March (Almost April)
> >
> > Would love to see more of a discussion around solving the problem of
> > releasing when we say we are going to - so that we can then choose
> > what the cadence is.
> >
> > >
> > > On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage(a)newdream.net>
wrote:
> > >>
> > >> On Wed, 5 Jun 2019, Sage Weil wrote:
> > >> > That brings us to an important decision: what time of year should
we
> > >> > release? Once we pick the timing, we'll be releasing at that
time
> > *every
> > >> > year* for each release (barring another schedule shift, which we
want
> > to
> > >> > avoid), so let's choose carefully!
> > >>
> > >> I've put up a twitter poll:
> > >>
> > >> https://twitter.com/liewegas/status/1140655233430970369
> > >>
> > >> Thanks!
> > >> sage
> > >> _______________________________________________
> > >> ceph-users mailing list
> > >> ceph-users(a)lists.ceph.com
> > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users(a)lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> --
>
> Alfonso Martínez
>
> Senior Software Engineer, Ceph Storage
>
> Red Hat <https://www.redhat.com>
> <https://red.ht/sig>
> 
8:26 p.m.
New subject: [ceph-users] Changing the release cadence
March seems sensible to me for the reasons you stated. If a release gets
delayed, I'd prefer it to be on the spring side of Christmas (again for the
reasons already mentioned).
That aside, I'm now very impatient to install Octopus on my 8-node cluster.
: )
On Wed, 26 Jun 2019 at 15:46, Sage Weil <sweil(a)redhat.com> wrote:
Hi everyone,
We talked a bit about this during the CLT meeting this morning. How about
the following proposal:
- Target release date of Mar 1 each year.
- Target freeze in Dec. That will allow us to use the holidays to do a
lot of testing when the lab infrastructure tends to be somewhat idle.
If we get an early build out at the point of the freeze (or even earlier),
perhaps this capture some of the time that the retailers have during their
lockdown to identify structural issues with release. It is probably
better to do more of this testing at this point in the cycle so that we
have time to properly fix any big issues (like performance or scaling
regressions). It is of course a challenge to motivate testing on
something that is too far from the final a release, but we can try.
This avoids an abbreviated octopus cycle, and avoids placing August (which
also often has people out for vacations) right in the middle of the
lead-up to the freeze.
Thoughts?
sage
On Wed, 26 Jun 2019, Sage Weil wrote:
On Wed, 26 Jun 2019, Alfonso Martinez Hidalgo
wrote:
for
I think March is a good idea.
Spring had a slight edge over fall in the twitter poll (for whatever
that's worth). I see the appeal for fall when it comes to down time retailers, but as a practical matter for Octopus
specifically, a target
of
say October means freezing in August, which means
we only have 2
more months of development time. I'm worried that will turn Octopus
in another weak (aka lightly adopted) release.
March would mean freezing in January again, which would give us July to
Dec... 6 more months.
sage
>
> On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza <adeza(a)redhat.com> wrote:
>
> > On Mon, Jun 17, 2019 at 4:09 PM David Turner <drakonstein(a)gmail.com>
> > wrote:
> > >
> > > This was a little long to respond with on Twitter, so I thought I'd
> > share my thoughts here. I love the idea of a 12 month cadence. I like
> > October because admins aren't upgrading production within the first
few
> > months of a new release. It gives it
plenty of time to be stable for
the OS
> > distros as well as giving admins
something low-key to work on over
the
> > holidays with testing the new releases
in stage/QA.
> >
> > October sounds ideal, but in reality, we haven't been able to release
> > right on time as long as I can remember. Realistically, if we set
> > October, we are probably going to get into November/December.
> >
> > For example, Nautilus was set to release in February and we got it
out
> > late in late March (Almost April)
> >
> > Would love to see more of a discussion around solving the problem of
> > releasing when we say we are going to - so that we can then choose
> > what the cadence is.
> >
> > >
> > > On Mon, Jun 17, 2019 at 12:22 PM Sage Weil <sage(a)newdream.net>
wrote:
> > >>
> > >> On Wed, 5 Jun 2019, Sage Weil wrote:
> > >> > That brings us to an important decision: what time of year
should we
> > >> > release? Once we pick
the timing, we'll be releasing at that
time
> > *every
> > >> > year* for each release (barring another schedule shift, which
we want
> > to
> > >> > avoid), so let's choose carefully!
> > >>
> > >> I've put up a twitter poll:
> > >>
> > >> https://twitter.com/liewegas/status/1140655233430970369
> > >>
> > >> Thanks!
> > >> sage
> > >> _______________________________________________
> > >> ceph-users mailing list
> > >> ceph-users(a)lists.ceph.com
> > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users(a)lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> --
>
> Alfonso Martínez
>
> Senior Software Engineer, Ceph Storage
>
> Red Hat <https://www.redhat.com>
> <https://red.ht/sig>
> _______________________________________________
ceph-users mailing list
ceph-users(a)lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
8:51 p.m.
New subject: [ceph-users] Changing the release cadence
On 2019-06-26T14:45:31, Sage Weil <sweil(a)redhat.com> wrote:
Hi Sage,
I think that makes sense. I'd have preferred the Oct/Nov target, but
that'd have made Octopus quite short.
Unsure whether freezing in December with a release in March is too long
though. But given how much people scramble, setting that as a goal
probably will help with stabilization.
I'm also hoping that one day, we can move towards a more agile
continuously integration model (like the Linux kernel does) instead of
massive yearly forklifts. But hey, that's just me ;-)
Regards,
Lars
--
SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
"Architects should open possibilities and not determine everything." (Ueli
Zbinden)
6:25 p.m.
New subject: [ceph-users] Changing the release cadence
On Tue, 25 Jun 2019, Alfredo Deza wrote:
On Mon, Jun 17, 2019 at 4:09 PM David Turner
<drakonstein(a)gmail.com> wrote:
I think the "on time" part is solveable. We should just use the amount
of time between take the previous release's freeze date and the
target release date and go with that. It is a bit fuzzy because I left it
up to the leads how they handle the freeze, but I think mid-Januaray is
about right (in reality we waiting longer than that for lots of RADOS
stuff). v14.2.0 was Mar 18, so ~2 months.
The cadence is really separate from that, though: even if every release
were 2 full months late, if we start with the same target it's still a 1
year cycle.
sage
This was a little long to respond with on Twitter, so I thought I'd share my thoughts
here. I love the idea of a 12 month cadence. I like October because admins aren't
upgrading production within the first few months of a new release. It gives it plenty of
time to be stable for the OS distros as well as giving admins something low-key to work on
over the holidays with testing the new releases in stage/QA.
October sounds ideal, but in reality, we haven't been able to release
right on time as long as I can remember. Realistically, if we set
October, we are probably going to get into November/December.
For example, Nautilus was set to release in February and we got it out
late in late March (Almost April)
Would love to see more of a discussion around solving the problem of
releasing when we say we are going to - so that we can then choose
what the cadence is. 
15 Jul
15 Jul
7:28 p.m.
On Wed, Jun 5, 2019 at 11:58 AM Sage Weil <sage(a)newdream.net> wrote:
...
This has mostly worked out well, except that the mimic release received
less attention that we wanted due to the fact that multiple downstream
Ceph products (from Red Has and SUSE) decided to based their next release
on nautilus. Even though upstream every release is an "LTS" release, as a
practical matter mimic got less attention than luminous or nautilus.
Speaking as (one of) the Ceph packager(s) in Fedora:
1) I have been told (don't recall by who now) that odd numbered releases
are experimental or short term support or red haired step child releases.
2) even though there have been ceph-{10,11,12}.0.z tarballs on
http://download.ceph.com/tarballs/ in the past, there still isn't a
ceph-15.0.0.tar.gz there. And I was told that there wouldn't be one. Yes,
I'm aware that X.0.z are alpha releases, X.1.z are beta releases, and X.2.z
are GA releases.
3) ceph-13 was skipped over in Fedora because I could never get it to build
in Fedora in the limited time I make to do upstream packaging of Ceph and
other things (because it's not my $dayjob.) And there seemed to be no
interest on the part of anyone in the Ceph devel community to fix it
(because see #1?)
If Octopus is really an LTS release like all the others, and you want
bleeding edge users to test/use it and give early feedback, then Fedora is
probably one of the better places to get that feedback.
FWIW, In the absence of a ceph-14.0.z.tar.gz from
http://download.ceph.com/tarballs/ branto came up with one (possibly from
https://github.com/ceph/ceph/releases/...???) but I haven't had any luck
building ceph-15 with the tarball from there.
My 2¢.
7:40 p.m.
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
On Wed, Jun 5, 2019 at 11:58 AM Sage Weil
<sage(a)newdream.net> wrote:
In principle that hasn't been the case since 12.2.z (luminous). 13 has
gotten somewhat less backport attention because the people doing backports
work for Red Hat and SUSE and neither shipped a product based on 13.
...
This has mostly worked out well, except that the mimic release received
less attention that we wanted due to the fact that multiple downstream
Ceph products (from Red Has and SUSE) decided to based their next release
on nautilus. Even though upstream every release is an "LTS" release, as a
practical matter mimic got less attention than luminous or nautilus.
Speaking as (one of) the Ceph packager(s) in Fedora:
1) I have been told (don't recall by who now) that odd numbered releases
are experimental or short term support or red haired step child releases. 2) even though there have been ceph-{10,11,12}.0.z
tarballs on
http://download.ceph.com/tarballs/ in the past, there still isn't a
ceph-15.0.0.tar.gz there. And I was told that there wouldn't be one. Yes,
I'm aware that X.0.z are alpha releases, X.1.z are beta releases, and X.2.z
are GA releases.
In the past we haven't published tarballs until x.1.z (rc), since the dev
checkpoints are not meant to be used by anyone other than developers.
(They're really just a 'pause and make sure the qa suites are nice and
clean', nothing more.)
3) ceph-13 was skipped over in Fedora because I could
never get it to build
in Fedora in the limited time I make to do upstream packaging of Ceph and
other things (because it's not my $dayjob.) And there seemed to be no
interest on the part of anyone in the Ceph devel community to fix it
(because see #1?)
If Octopus is really an LTS release like all the others, and you want
bleeding edge users to test/use it and give early feedback, then Fedora is
probably one of the better places to get that feedback.
I think the first release worth testing outside of the dev community is
the release candidate. I don't like the idea of having any distro carry
an untested dev checkpoint or else someone will lose data... even the rc
should be tested cautiously and, since it is only relevant for a week or
two, I'm not sure that distros can play much of a role there?
sage
FWIW, In the absence of a ceph-14.0.z.tar.gz from
http://download.ceph.com/tarballs/ branto came up with one (possibly from
https://github.com/ceph/ceph/releases/...???) but I haven't had any luck
building ceph-15 with the tarball from there.
My 2¢.
9:11 p.m.
On Mon, Jul 15, 2019 at 10:10 AM Sage Weil <sage(a)newdream.net> wrote:
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
To be clear, when I talk about packaging a new version Ceph, it starts out
by _only_ going into Fedora Rawhide. It would be extremely foolish, IMO,
for anyone to run Rawhide for anything that's mission critical.
Sometimes it takes a long time to work through build and packaging issues.
The switch to gcc-8 and gcc-9 are good illustrations of the kinds of issues
that we, as packagers, run into. Those kinds of things are why I — at least
— like to get as early a start as I can. Even short-lived release
candidates are useful stepping stones to the eventual GA.
Personally I'd say that any fears anyone has of people losing data by using
an early dev checkpoint of Ceph on Rawhide are probably a teeny bit
misplaced.
My 2.5¢ worth.
--
Kaleb
If Octopus is really an LTS release like all the others, and you want
bleeding edge users to test/use it and give early feedback, then Fedora
is
probably one of the better places to get that
feedback.
I think the first release worth testing outside of the dev community is
the release candidate. I don't like the idea of having any distro carry
an untested dev checkpoint or else someone will lose data... even the rc
should be tested cautiously and, since it is only relevant for a week or
two, I'm not sure that distros can play much of a role there?
9:20 p.m.
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
On Mon, Jul 15, 2019 at 10:10 AM Sage Weil
<sage(a)newdream.net> wrote:
Okay, it sounds like the rc x.1.z releases are a good fit, then. They
should start about ~2 months before the first stable release. I'm not so
sure about the dev checkpoints since they are quite arbitrary. I suspect
less effort would be needed to just do a manual build a few times
partway through the cycle (e.g., now), identify any issues, and open PRs
with build fixes.
sage
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
To be clear, when I talk about packaging a new version Ceph, it starts out
by _only_ going into Fedora Rawhide. It would be extremely foolish, IMO,
for anyone to run Rawhide for anything that's mission critical.
Sometimes it takes a long time to work through build and packaging issues.
The switch to gcc-8 and gcc-9 are good illustrations of the kinds of issues
that we, as packagers, run into. Those kinds of things are why I — at least
— like to get as early a start as I can. Even short-lived release
candidates are useful stepping stones to the eventual GA.
Personally I'd say that any fears anyone has of people losing data by using
an early dev checkpoint of Ceph on Rawhide are probably a teeny bit
misplaced.
If Octopus is really an LTS release like all the others, and you want
bleeding edge users to test/use it and give early feedback, then Fedora
is
probably one of the better places to get that
feedback.
I think the first release worth testing outside of the dev community is
the release candidate. I don't like the idea of having any distro carry
an untested dev checkpoint or else someone will lose data... even the rc
should be tested cautiously and, since it is only relevant for a week or
two, I'm not sure that distros can play much of a role there?
7:46 p.m.
Hey,
On 15.07.19 09:58, Kaleb Keithley wrote:
Speaking as (one of) the Ceph packager(s) in Fedora:
Arch Linux packager for Ceph here o/
If Octopus is really an LTS release like all the
others, and you want
bleeding edge users to test/use it and give early feedback, then Fedora is
probably one of the better places to get that feedback.
FWIW, In the absence of a ceph-14.0.z.tar.gz from
http://download.ceph.com/tarballs/ branto came up with one (possibly from
https://github.com/ceph/ceph/releases/...???) but I haven't had any luck
building ceph-15 with the tarball from there.
I've been watching the RSS feed of the GitHub releases and use the
GitHub release tarballs for building it.
Due to unforseen complications and lack of time (similar situation
like you) the last packaged version of ceph in the Arch Linux repos is
13.2.1 right now.
Over the last couple of weeks I have spent numerous hours and worked
with people in oftc.#ceph-devel to work on the issues I was
experiencing while building it.
Right now I'm finalizing the 14.2.1 build and have some last kinks to
work out, for some reason the zstd compression tests are failing for
me, gonna look into that over the coming days.
Seeing that Arch Linux and Fedora should have a few more things common
then, let's say Ubuntu or Debian, I would like to offer my help
however I can. I'm not that familiar with building packages on/for
Fedora but I've had my fair share of debugging building ceph with
bleeding edge components (gcc, cmake, boost, python) and could
possibly help out there.
Feel free to get in touch, maybe we can help each other out.
Cheers,
Thore
PS: I was thinking about maybe getting Arch Linux (and Fedora)
onboarded to the ceph-dev-build matrix on jenkins.ceph.com.
Thinking about the possibility to spot arising issues and/or
incompatibilities with upstream / bleeding edge compilers, helpers and
dependencies that should have common intereset with Ceph upstream.
Although I haven't talked to anyone upstream about this and am just
throwing it out here.
--
Thore Bödecker
GPG ID: 0xD622431AF8DB80F3
GPG FP: 0F96 559D 3556 24FC 2226 A864 D622 431A F8DB 80F3
22 Jul
22 Jul
8:33 a.m.
New subject: [ceph-users] Changing the release cadence
12 months sounds good to me, I like the idea of march as well since we plan
on doing upgrades in June/July each year. Gives it time to be discussed and
marinate before we decide to upgrade.
-Brent
-----Original Message-----
From: ceph-users <ceph-users-bounces(a)lists.ceph.com> On Behalf Of Sage Weil
Sent: Wednesday, June 5, 2019 11:58 AM
To: ceph-users(a)ceph.com; ceph-devel(a)vger.kernel.org; dev(a)ceph.io
Subject: [ceph-users] Changing the release cadence
Hi everyone,
Since luminous, we have had the follow release cadence and policy:
- release every 9 months
- maintain backports for the last two releases
- enable upgrades to move either 1 or 2 releases heads
(e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)
This has mostly worked out well, except that the mimic release received less
attention that we wanted due to the fact that multiple downstream Ceph
products (from Red Has and SUSE) decided to based their next release on
nautilus. Even though upstream every release is an "LTS" release, as a
practical matter mimic got less attention than luminous or nautilus.
We've had several requests/proposals to shift to a 12 month cadence. This
has several advantages:
- Stable/conservative clusters only have to be upgraded every 2 years
(instead of every 18 months)
- Yearly releases are more likely to intersect with downstream
distribution release (e.g., Debian). In the past there have been
problems where the Ceph releases included in consecutive releases of a
distro weren't easily upgradeable.
- Vendors that make downstream Ceph distributions/products tend to
release yearly. Aligning with those vendors means they are more likely
to productize *every* Ceph release. This will help make every Ceph
release an "LTS" release (not just in name but also in terms of
maintenance attention).
So far the balance of opinion seems to favor a shift to a 12 month cycle[1],
especially among developers, so it seems pretty likely we'll make that
shift. (If you do have strong concerns about such a move, now is the time
to raise them.)
That brings us to an important decision: what time of year should we
release? Once we pick the timing, we'll be releasing at that time *every
year* for each release (barring another schedule shift, which we want to
avoid), so let's choose carefully!
A few options:
- November: If we release Octopus 9 months from the Nautilus release
(planned for Feb, released in Mar) then we'd target this November. We
could shift to a 12 months candence after that.
- February: That's 12 months from the Nautilus target.
- March: That's 12 months from when Nautilus was *actually* released.
November is nice in the sense that we'd wrap things up before the holidays.
It's less good in that users may not be inclined to install the new release
when many developers will be less available in December.
February kind of sucked in that the scramble to get the last few things done
happened during the holidays. OTOH, we should be doing what we can to avoid
such scrambles, so that might not be something we should factor in. March
may be a bit more balanced, with a solid 3 months before when people are
productive, and 3 months after before they disappear on holiday to address
any post-release issues.
People tend to be somewhat less available over the summer months due to
holidays etc, so an early or late summer release might also be less than
ideal.
Thoughts? If we can narrow it down to a few options maybe we could do a
poll to gauge user preferences.
Thanks!
sage
[1] https://twitter.com/larsmb/status/1130010208971952129
_______________________________________________
ceph-users mailing list
ceph-users(a)lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
1738
days inactive
1785
days old
20 comments
14 participants
participants (14)
-
Alexandre DERUMIER -
Alfonso Martinez Hidalgo -
Alfredo Deza -
Bob Farrell -
Brent Kennedy -
Chris Taylor -
Daniel Baumann -
David Turner -
Kaleb Keithley -
Lars Marowsky-Bree -
Linh Vu -
Sage Weil -
Sage Weil -
Thore Bödecker