CEPH Version choice
Hi all, I've been reading through this email list for a while now, but one thing that I'm curious about is why a lot of installations out there aren't upgraded to the latest version of CEPH (Quincy). What are the main reasons for not upgrading to the latest and greatest? Thanks. Tino This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427.
Hi,
On 15 May 2023, at 11:37, Tino Todino <tinot@marlan-tech.co.uk> wrote:
What are the main reasons for not upgrading to the latest and greatest?
One of the main reasons - "just can't", because your Ceph-based products will get worse at real (not benchmark) performance, see [1] [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2E67NW6BEAVI... k
Hi, as I understand, Pacific+ is having a performance issue that does not exist in older releases? So that why Ceph's new release (Reef) is delayed in this year?
The release of Reef has been delayed in part due to issues that sidelined the testing / validation infrastructure.
On May 15, 2023, at 05:40, huy nguyen <viplanghe6@gmail.com> wrote:
Hi, as I understand, Pacific+ is having a performance issue that does not exist in older releases? So that why Ceph's new release (Reef) is delayed in this year? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
What are the main reasons for not upgrading to the latest and greatest?
Because more often than not it isn't. I guess when you write "latest and greatest" you talk about features. When we admins talk about "latest and greatest" we talk about stability. The times that one could jump with a production system onto a "stable" release with the ending .2 are long gone. Anyone who becomes an early adapter is more and more likely to experience serious issues. Which leads to more admins waiting with upgrades. Which in turn leads to more bugs discovered only at late releases. Which again makes more admins postpone an upgrade. A vicious cycle. A long time ago there was a discussion about exactly this problem and the admins were pretty much in favor of increasing the release cadence to at least 4 years if not longer. Its simply too many releases with too many serious bugs not fixed, lately not even during their official life time. Octopus still has serious bugs but is EOL. I'm not surprised that admins give up on upgrading entirely and stay on a version until their system dies. To give you one from my own experience, upgrading from mimic latest to octopus latest. This experience almost certainly applies to every upgrade that involves an OSD format change (the infamous "quick fix" that could take several days per OSD and crush entire clusters). There is an OSD conversion involved in this upgrade and we found out that out of 2 possible upgrade paths, one leads to a heavily performance degraded cluster with no possibility to recover other than redeploying all OSDs step by step. Funnily enough, the problematic procedure is the one described in the documentation - it hasn't been updated until today despite users still getting caught in this trap. To give you an idea of what amount of work is now involved in an attempt to avoid such pitfalls, here our path: We set up a test cluster with a script producing realistic workload and started testing an upgrade under load. This took about a month (meaning repeating the upgrade with a cluster on mimic deployed and populated from scratch every time) to confirm that we managed to get onto a robust path avoiding a number of pitfalls along the way - mainly the serious performance degradation due to OSD conversion, but also an issue with stray entries plus noise. A month! Once we were convinced that it would work - meaning we did run it a couple of times without any further issues being discovered, we started upgrading our production cluster. Went smooth until we started the OSD conversion of our FS meta data OSDs. They had a special performance optimized deployment resulting in a large number of 100G OSDs with about 30-40% utilization. These OSDs started crashing with some weird corruption. Turns out - thanks Igor! - that while spill-over from fast to slow drive was handled, the other direction was not. Our OSDs crashed because Octopus apparently required substantially more space on the slow device and couldn't use the plenty of fast space that was actually available. The whole thing ended in 3 days of complete downtime and me working 12 hour days on the weekend. We managed to recover from this only because we had a larger delivery of hardware already on-site and I could scavenge parts from there. So, the story was that after 1 month of testing we still run into 3 days of downtime, because there was another unannounced change that broke a config that was working fine for years on mimic. To say the same thing with different words: major version upgrades have become very disruptive and require a lot of effort to get halfway right. And I'm not talking about the deployment system here. Add to this list the still open cases discussed on the list about MDS dentry corruption, snapshots disappearing/corrupting together with a lack of good built-in tools for detection and repair, performance degradation etc. all not even addressed in pacific. In this state the devs are pushing for pacific becoming EOL while at the same time the admins become ever more reluctant to upgrade. In my specific case, I planned to upgrade at least to pacific this year, but my time budget simply doesn't allow for the verification of the procedure and checking that all for us relevant bugs have been addressed. I gave up. Maybe next year. Maybe then its even a bit closer to rock solid. So to get back to my starting point, we admins actually value rock solid over features. I know that this is boring for devs, but nothing is worse than nobody using your latest and greatest - which probably was the motivation for your question. If the upgrade paths were more solid and things like the question "why does an OSD conversion not lead to an OSD that is identical to one deployed freshly" or "where does the performance go" would actually attempted to track down, we would be much less reluctant to upgrade. And then, but only then, would the latest and greatest features be of interest. I will bring it up here again: with the complexity that the code base reached now, the 2 year release cadence is way too fast, it doesn't provide sufficient maturity for upgrading fast as well. More and more admins will be several cycles behind and we are reaching the point where major bugs in so-called EOL versions will only be discovered before large clusters even reached this version. Which might become a fundamental blocker to upgrades entirely. An alternative to increasing the release cadence would be to keep more cycles in the life-time loop instead of only the last 2 major releases. 4 years really is nothing when it comes to storage. Hope this is helpful and puts some light on the mystery why admins don't want to move. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Monday, May 15, 2023 10:43 AM To: Tino Todino Cc: ceph-users@ceph.io Subject: [ceph-users] Re: CEPH Version choice Hi,
On 15 May 2023, at 11:37, Tino Todino <tinot@marlan-tech.co.uk> wrote:
What are the main reasons for not upgrading to the latest and greatest?
One of the main reasons - "just can't", because your Ceph-based products will get worse at real (not benchmark) performance, see [1] [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2E67NW6BEAVI... k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We set up a test cluster with a script producing realistic workload and started testing an upgrade under load. This took about a month (meaning repeating the upgrade with a cluster on mimic deployed and populated
Hi Frank, do you have such scripts online? On github or so? I was thinking of compiling el9 rpms for Nautilus and run tests for a few days on a test cluster with mixed el7 and el9 hosts.
So to get back to my starting point, we admins actually value rock solid over features. I know that this is boring for devs, but nothing is worse than nobody using your latest and greatest - which probably was the motivation for your question. If the upgrade paths were more solid and things like the question "why does an OSD conversion not lead to an OSD that is identical to one deployed freshly" or "where does the performance go" would actually attempted to track down, we would be much less reluctant to upgrade.
I will bring it up here again: with the complexity that the code base reached now, the 2 year release cadence is way too fast, it doesn't provide sufficient maturity for upgrading fast as well. More and more admins will be several cycles behind and we are reaching the point where major bugs in so-called EOL versions will only be discovered before large clusters even reached this version. Which might become a fundamental blocker to upgrades entirely.
Indeed.
An alternative to increasing the release cadence would be to keep more cycles in the life-time loop instead of only the last 2 major releases. 4 years really is nothing when it comes to storage.
I would like to see this change also.
Hi Marc, I planned to put it on-line. The hold-back is that the main test is un-taring a nasty archive and this archive might contain personal information, so I can't just upload it as is. I can try to put together a similar archive from public sources. Please give me a bit of time. I'm also a bit under stress right now with our users being hit by an FS meta data corruption. That's also why I'm a bit trigger happy. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc <Marc@f1-outsourcing.eu> Sent: Monday, May 15, 2023 1:03 PM To: Frank Schilder; Tino Todino Cc: ceph-users@ceph.io; dev@ceph.io Subject: RE: [ceph-users] Re: CEPH Version choice
We set up a test cluster with a script producing realistic workload and started testing an upgrade under load. This took about a month (meaning repeating the upgrade with a cluster on mimic deployed and populated
Hi Frank, do you have such scripts online? On github or so? I was thinking of compiling el9 rpms for Nautilus and run tests for a few days on a test cluster with mixed el7 and el9 hosts.
So to get back to my starting point, we admins actually value rock solid over features. I know that this is boring for devs, but nothing is worse than nobody using your latest and greatest - which probably was the motivation for your question. If the upgrade paths were more solid and things like the question "why does an OSD conversion not lead to an OSD that is identical to one deployed freshly" or "where does the performance go" would actually attempted to track down, we would be much less reluctant to upgrade.
I will bring it up here again: with the complexity that the code base reached now, the 2 year release cadence is way too fast, it doesn't provide sufficient maturity for upgrading fast as well. More and more admins will be several cycles behind and we are reaching the point where major bugs in so-called EOL versions will only be discovered before large clusters even reached this version. Which might become a fundamental blocker to upgrades entirely.
Indeed.
An alternative to increasing the release cadence would be to keep more cycles in the life-time loop instead of only the last 2 major releases. 4 years really is nothing when it comes to storage.
I would like to see this change also.
Hi, I know the problems that Frank has raised. However, it should also be mentioned that many critical bugs have been fixed in the major versions. We are working on the fixes ourselves. We and others have written a lot of tools for ourselves in the last 10 years to improve migration/update and upgrade paths/strategy. From version to version, we also test for up to 6 months before putting them into production. However, our goal is always to use Ceph versions that still get backports and on the other hand, only use the features we really need. Our developers also always aim to bring bug fixes upstream and into the supported versions. By the way, regarding performance I recommend the Cephalocon presentations by Adam and Mark. There you can learn what efforts are made to improve ceph performance for current and future versions. Regards, Joachim ___________________________________ ceph ambassador DACH ceph consultant since 2012 Clyso GmbH - Premier Ceph Foundation Member https://www.clyso.com/ Am 15.05.23 um 12:11 schrieb Frank Schilder:
What are the main reasons for not upgrading to the latest and greatest? Because more often than not it isn't.
I guess when you write "latest and greatest" you talk about features. When we admins talk about "latest and greatest" we talk about stability. The times that one could jump with a production system onto a "stable" release with the ending .2 are long gone. Anyone who becomes an early adapter is more and more likely to experience serious issues. Which leads to more admins waiting with upgrades. Which in turn leads to more bugs discovered only at late releases. Which again makes more admins postpone an upgrade. A vicious cycle.
A long time ago there was a discussion about exactly this problem and the admins were pretty much in favor of increasing the release cadence to at least 4 years if not longer. Its simply too many releases with too many serious bugs not fixed, lately not even during their official life time. Octopus still has serious bugs but is EOL.
I'm not surprised that admins give up on upgrading entirely and stay on a version until their system dies.
To give you one from my own experience, upgrading from mimic latest to octopus latest. This experience almost certainly applies to every upgrade that involves an OSD format change (the infamous "quick fix" that could take several days per OSD and crush entire clusters).
There is an OSD conversion involved in this upgrade and we found out that out of 2 possible upgrade paths, one leads to a heavily performance degraded cluster with no possibility to recover other than redeploying all OSDs step by step. Funnily enough, the problematic procedure is the one described in the documentation - it hasn't been updated until today despite users still getting caught in this trap.
To give you an idea of what amount of work is now involved in an attempt to avoid such pitfalls, here our path:
We set up a test cluster with a script producing realistic workload and started testing an upgrade under load. This took about a month (meaning repeating the upgrade with a cluster on mimic deployed and populated from scratch every time) to confirm that we managed to get onto a robust path avoiding a number of pitfalls along the way - mainly the serious performance degradation due to OSD conversion, but also an issue with stray entries plus noise. A month! Once we were convinced that it would work - meaning we did run it a couple of times without any further issues being discovered, we started upgrading our production cluster.
Went smooth until we started the OSD conversion of our FS meta data OSDs. They had a special performance optimized deployment resulting in a large number of 100G OSDs with about 30-40% utilization. These OSDs started crashing with some weird corruption. Turns out - thanks Igor! - that while spill-over from fast to slow drive was handled, the other direction was not. Our OSDs crashed because Octopus apparently required substantially more space on the slow device and couldn't use the plenty of fast space that was actually available.
The whole thing ended in 3 days of complete downtime and me working 12 hour days on the weekend. We managed to recover from this only because we had a larger delivery of hardware already on-site and I could scavenge parts from there.
So, the story was that after 1 month of testing we still run into 3 days of downtime, because there was another unannounced change that broke a config that was working fine for years on mimic.
To say the same thing with different words: major version upgrades have become very disruptive and require a lot of effort to get halfway right. And I'm not talking about the deployment system here.
Add to this list the still open cases discussed on the list about MDS dentry corruption, snapshots disappearing/corrupting together with a lack of good built-in tools for detection and repair, performance degradation etc. all not even addressed in pacific. In this state the devs are pushing for pacific becoming EOL while at the same time the admins become ever more reluctant to upgrade.
In my specific case, I planned to upgrade at least to pacific this year, but my time budget simply doesn't allow for the verification of the procedure and checking that all for us relevant bugs have been addressed. I gave up. Maybe next year. Maybe then its even a bit closer to rock solid.
So to get back to my starting point, we admins actually value rock solid over features. I know that this is boring for devs, but nothing is worse than nobody using your latest and greatest - which probably was the motivation for your question. If the upgrade paths were more solid and things like the question "why does an OSD conversion not lead to an OSD that is identical to one deployed freshly" or "where does the performance go" would actually attempted to track down, we would be much less reluctant to upgrade.
And then, but only then, would the latest and greatest features be of interest.
I will bring it up here again: with the complexity that the code base reached now, the 2 year release cadence is way too fast, it doesn't provide sufficient maturity for upgrading fast as well. More and more admins will be several cycles behind and we are reaching the point where major bugs in so-called EOL versions will only be discovered before large clusters even reached this version. Which might become a fundamental blocker to upgrades entirely.
An alternative to increasing the release cadence would be to keep more cycles in the life-time loop instead of only the last 2 major releases. 4 years really is nothing when it comes to storage.
Hope this is helpful and puts some light on the mystery why admins don't want to move.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Monday, May 15, 2023 10:43 AM To: Tino Todino Cc: ceph-users@ceph.io Subject: [ceph-users] Re: CEPH Version choice
Hi,
On 15 May 2023, at 11:37, Tino Todino <tinot@marlan-tech.co.uk> wrote:
What are the main reasons for not upgrading to the latest and greatest? One of the main reasons - "just can't", because your Ceph-based products will get worse at real (not benchmark) performance, see [1]
[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2E67NW6BEAVI...
k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
https://www.youtube.com/playlist?list=PLrBUGiINAakPd9nuoorqeOuS9P9MTWos3 -----Original Message----- From: Marc <Marc@f1-outsourcing.eu> Sent: Monday, May 15, 2023 4:42 PM To: Joachim Kraftmayer - ceph ambassador <joachim.kraftmayer@clyso.com>; Frank Schilder <frans@dtu.dk>; Tino Todino <tinot@marlan-tech.co.uk> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: CEPH Version choice
By the way, regarding performance I recommend the Cephalocon presentations by Adam and Mark. There you can learn what efforts are made to improve ceph performance for current and future versions.
Link? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Adam & Mark topics: bluestore and bluestore v2 https://youtu.be/FVUoGw6kY5k https://youtu.be/7D5Bgd5TuYw ___________________________________ ceph ambassador DACH ceph consultant since 2012 Clyso GmbH - Premier Ceph Foundation Member https://www.clyso.com/ Am 15.05.23 um 16:47 schrieb Jens Galsgaard:
https://www.youtube.com/playlist?list=PLrBUGiINAakPd9nuoorqeOuS9P9MTWos3
-----Original Message----- From: Marc <Marc@f1-outsourcing.eu> Sent: Monday, May 15, 2023 4:42 PM To: Joachim Kraftmayer - ceph ambassador <joachim.kraftmayer@clyso.com>; Frank Schilder <frans@dtu.dk>; Tino Todino <tinot@marlan-tech.co.uk> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: CEPH Version choice
By the way, regarding performance I recommend the Cephalocon presentations by Adam and Mark. There you can learn what efforts are made to improve ceph performance for current and future versions.
Link? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, to avoid a potentially wrong impression I would like to add some words. Slightly out of order:
By the way, regarding performance I recommend the Cephalocon presentations by Adam and Mark. There you can learn what efforts are made to improve ceph performance for current and future versions.
For me personally the ceph performance degradation due to removing the WAL re-use is not a problem as it is predictable and the reasons for removing it are solid. A bit more worrying is the degradation over time and I know that there is work spent on it and it is expensive to debug, because collecting data takes so long. I mentioned performance mainly because there was at least one other user who explicitly called this a show-stopper. I appreciate the effort put into this and am not complaining about a lack of effort. What I am complaining about is that this effort is under unnecessary pressure due to the short release cadence. 2 years are not much to mature for a system like ceph and starting to count with the .2 release seems a bit premature given recent experience.
I know the problems that Frank has raised. However, it should also be mentioned that many critical bugs have been fixed in the major versions. We are working on the fixes ourselves.
Again, I know all this and I very much appreciate it. I still consider the voluntary ceph support as better than support we got for enterprise systems. I got a lot of invaluable help from Igor during my upgrade experience and I got some important stuff fixed by Xiubo recently. Just to repeat it though, I am convinced one could reach much higher maturity with less time pressure - and maybe less often forget this one critical PR that causes so much trouble for some users.
However, our goal is always to use Ceph versions that still get backports and on the other hand, only use the features we really need. Our developers also always aim to bring bug fixes upstream and into the supported versions.
I would love to, but the speed of counting versions up is too fast and the problems trying to keep up are a bit too much for a one-man army. After last time I have a hard time now convincing users to take another possible hit. If there was a bit more time to take a breath after the last upgrade, I would probably be able to do it. However, with my current experience I look at not being able to catch up for the time being. We might even fall further behind. Which is a shame because we operate a large installation and tend to discover relevant bugs that don't show up in smaller systems with less load. My wish and hypothesis simply are that if we would reduce the speed of major release cycles, a lot more operators would be able to follow and the releases and upgrade procedures would be significantly more stable just for the fact that more clusters are continuously closer to latest. For example, don't start counting the life time with the release of the .2 release of a major version, start when 50% of the top-10 (25/50) sized clusters in telemetry are on or above that version and declare a version EOL if 90% of these clusters are at a newer major release. This would give a much better indication of the perceived maturity of major releases by operators of significant installations. It would also give an incentive to submit data to telemetry as well as helping the last few % over the upgrade hurdle to be able to declare a version as EOL. Kind of a community where the ones who fall behind get an extra helping hand so everyone can move on. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Joachim Kraftmayer - ceph ambassador <joachim.kraftmayer@clyso.com> Sent: Monday, May 15, 2023 4:34 PM To: Frank Schilder; Tino Todino Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: CEPH Version choice Hi, I know the problems that Frank has raised. However, it should also be mentioned that many critical bugs have been fixed in the major versions. We are working on the fixes ourselves. We and others have written a lot of tools for ourselves in the last 10 years to improve migration/update and upgrade paths/strategy. From version to version, we also test for up to 6 months before putting them into production. However, our goal is always to use Ceph versions that still get backports and on the other hand, only use the features we really need. Our developers also always aim to bring bug fixes upstream and into the supported versions. By the way, regarding performance I recommend the Cephalocon presentations by Adam and Mark. There you can learn what efforts are made to improve ceph performance for current and future versions. Regards, Joachim ___________________________________ ceph ambassador DACH ceph consultant since 2012 Clyso GmbH - Premier Ceph Foundation Member https://www.clyso.com/ Am 15.05.23 um 12:11 schrieb Frank Schilder:
What are the main reasons for not upgrading to the latest and greatest? Because more often than not it isn't.
I guess when you write "latest and greatest" you talk about features. When we admins talk about "latest and greatest" we talk about stability. The times that one could jump with a production system onto a "stable" release with the ending .2 are long gone. Anyone who becomes an early adapter is more and more likely to experience serious issues. Which leads to more admins waiting with upgrades. Which in turn leads to more bugs discovered only at late releases. Which again makes more admins postpone an upgrade. A vicious cycle.
A long time ago there was a discussion about exactly this problem and the admins were pretty much in favor of increasing the release cadence to at least 4 years if not longer. Its simply too many releases with too many serious bugs not fixed, lately not even during their official life time. Octopus still has serious bugs but is EOL.
I'm not surprised that admins give up on upgrading entirely and stay on a version until their system dies.
To give you one from my own experience, upgrading from mimic latest to octopus latest. This experience almost certainly applies to every upgrade that involves an OSD format change (the infamous "quick fix" that could take several days per OSD and crush entire clusters).
There is an OSD conversion involved in this upgrade and we found out that out of 2 possible upgrade paths, one leads to a heavily performance degraded cluster with no possibility to recover other than redeploying all OSDs step by step. Funnily enough, the problematic procedure is the one described in the documentation - it hasn't been updated until today despite users still getting caught in this trap.
To give you an idea of what amount of work is now involved in an attempt to avoid such pitfalls, here our path:
We set up a test cluster with a script producing realistic workload and started testing an upgrade under load. This took about a month (meaning repeating the upgrade with a cluster on mimic deployed and populated from scratch every time) to confirm that we managed to get onto a robust path avoiding a number of pitfalls along the way - mainly the serious performance degradation due to OSD conversion, but also an issue with stray entries plus noise. A month! Once we were convinced that it would work - meaning we did run it a couple of times without any further issues being discovered, we started upgrading our production cluster.
Went smooth until we started the OSD conversion of our FS meta data OSDs. They had a special performance optimized deployment resulting in a large number of 100G OSDs with about 30-40% utilization. These OSDs started crashing with some weird corruption. Turns out - thanks Igor! - that while spill-over from fast to slow drive was handled, the other direction was not. Our OSDs crashed because Octopus apparently required substantially more space on the slow device and couldn't use the plenty of fast space that was actually available.
The whole thing ended in 3 days of complete downtime and me working 12 hour days on the weekend. We managed to recover from this only because we had a larger delivery of hardware already on-site and I could scavenge parts from there.
So, the story was that after 1 month of testing we still run into 3 days of downtime, because there was another unannounced change that broke a config that was working fine for years on mimic.
To say the same thing with different words: major version upgrades have become very disruptive and require a lot of effort to get halfway right. And I'm not talking about the deployment system here.
Add to this list the still open cases discussed on the list about MDS dentry corruption, snapshots disappearing/corrupting together with a lack of good built-in tools for detection and repair, performance degradation etc. all not even addressed in pacific. In this state the devs are pushing for pacific becoming EOL while at the same time the admins become ever more reluctant to upgrade.
In my specific case, I planned to upgrade at least to pacific this year, but my time budget simply doesn't allow for the verification of the procedure and checking that all for us relevant bugs have been addressed. I gave up. Maybe next year. Maybe then its even a bit closer to rock solid.
So to get back to my starting point, we admins actually value rock solid over features. I know that this is boring for devs, but nothing is worse than nobody using your latest and greatest - which probably was the motivation for your question. If the upgrade paths were more solid and things like the question "why does an OSD conversion not lead to an OSD that is identical to one deployed freshly" or "where does the performance go" would actually attempted to track down, we would be much less reluctant to upgrade.
And then, but only then, would the latest and greatest features be of interest.
I will bring it up here again: with the complexity that the code base reached now, the 2 year release cadence is way too fast, it doesn't provide sufficient maturity for upgrading fast as well. More and more admins will be several cycles behind and we are reaching the point where major bugs in so-called EOL versions will only be discovered before large clusters even reached this version. Which might become a fundamental blocker to upgrades entirely.
An alternative to increasing the release cadence would be to keep more cycles in the life-time loop instead of only the last 2 major releases. 4 years really is nothing when it comes to storage.
Hope this is helpful and puts some light on the mystery why admins don't want to move.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Monday, May 15, 2023 10:43 AM To: Tino Todino Cc: ceph-users@ceph.io Subject: [ceph-users] Re: CEPH Version choice
Hi,
On 15 May 2023, at 11:37, Tino Todino <tinot@marlan-tech.co.uk> wrote:
What are the main reasons for not upgrading to the latest and greatest? One of the main reasons - "just can't", because your Ceph-based products will get worse at real (not benchmark) performance, see [1]
[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2E67NW6BEAVI...
k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 5/15/23 12:11, Frank Schilder wrote:
Because more often than not it isn't.
Sadly, I have to agree. We basically gave up after luminous, where every update (on our test-ceph cluster) was a major pain. Until then, we always updated after one week of a new release. To add one more point.. The current Ceph version (17.x) will not be included in the upcoming Debian 12 release to be released later this summer. This hasn't been a problem in the past, because we just built our own backports and everything was fine. Nowadays, Ceph 17 doesn't even build on Debian unstable/testing because some libraries (mostly fmtlib and others) are too new (sic!), so.. we'll be staying with Ceph 16 on Debian 12 until we'll trash the hardware. ..or in other words: it would be nice if you could more efforts into upgrade-tests/QA as well as on releasing stuff that actually compiles on non-RHEL/current Linux distributions. Regards, Daniel
On 5/15/23 13:03, Daniel Baumann wrote:
On 5/15/23 12:11, Frank Schilder wrote:
Because more often than not it isn't. Sadly, I have to agree. We basically gave up after luminous, where every update (on our test-ceph cluster) was a major pain. Until then, we always updated after one week of a new release.
Any chance you could highlight the major pain points? I've heard a couple of stories (especially related to swap and buffered IO), but it would be good to know what the others have been.
To add one more point..
The current Ceph version (17.x) will not be included in the upcoming Debian 12 release to be released later this summer. This hasn't been a problem in the past, because we just built our own backports and everything was fine.
Nowadays, Ceph 17 doesn't even build on Debian unstable/testing because some libraries (mostly fmtlib and others) are too new (sic!), so.. we'll be staying with Ceph 16 on Debian 12 until we'll trash the hardware.
This is getting off-topic, but I wanted to make sure you got a quick reply because it's an important topic. Sadly this one isn't our fault, isn't limited to Debian, and doesn't only affect Ceph. There are actually two separate problems affecting bookworm (and other very updated distros): LibFMT: Any OS using FMT 9.0+ is going to have issues due to a breaking change in the library: https://www.spinics.net/lists/fedora-devel/msg303183.html https://github.com/fmtlib/fmt/releases/tag/9.0.0 If you just need to compile ceph itself you can get around this by passing the flag to do_cmake.sh: *./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_FMT_HEADER_ONLY:BOOL=ON* ** *Snappy: This is an even more irritating problem imho. Snappy 1.1.9 breaks RTTI runtime support and it's also breaking multiple applications including Ceph. * ** *https://bugzilla.redhat.com/show_bug.cgi?id=1980614* *https://github.com/google/snappy/pull/129* ** *That second link includes a fix by a user and the response was:* ** *"The project's CMake configuration reflects the way it's used in Google Chrome. This is the only configuration we can maintain ourselves. To be clear, this doesn't mean your changes are not valid -- I think you'll be able to use Snappy with the build tweaks you posted here just fine. It's just that we can't accept this change in the official repository."* ** *I'm kind of in a ripping-out mood right now, but my inclination is to remove snappy support if google can't test it outside of how it's used in Chrome.* ** *Mark * **
..or in other words: it would be nice if you could more efforts into upgrade-tests/QA as well as on releasing stuff that actually compiles on non-RHEL/current Linux distributions.
Regards, Daniel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
I've been reading through this email list for a while now, but one thing that I'm curious about is why a lot of installations out there aren't upgraded to the latest version of CEPH (Quincy).
What are the main reasons for not upgrading to the latest and greatest?
If you are starting with a fresh install. Just go for the newest. Once you have it, don't go and install any update/upgrade blindly. Depending on your team size, it pays of to wait a bit and see what others are reporting. cephadm is a container implementation also 'new'. If you are not proficient with containers or have already a different container environment, maybe stick to the basic packages. This still seems to be the easiest to maintain.
participants (10)
-
Anthony D'Atri
-
Daniel Baumann
-
Frank Schilder
-
huy nguyen
-
Jens Galsgaard
-
Joachim Kraftmayer - ceph ambassador
-
Konstantin Shalygin
-
Marc
-
Mark Nelson
-
Tino Todino