v18.2.4 Reef released
We're happy to announce the 4th release in the Reef series. An early build of this release was accidentally exposed and packaged as 18.2.3 by the Debian project in April. That 18.2.3 release should not be used. The official release was re-tagged as v18.2.4 to avoid further confusion. v18.2.4 container images, now based on CentOS 9, may be incompatible on older kernels (e.g., Ubuntu 18.04) due to differences in thread creation methods. Users upgrading to v18.2.4 container images with older OS versions may encounter crashes during `pthread_create`. For workarounds, refer to the related tracker. However, we recommend upgrading your OS to avoid this unsupported combination. Related tracker: https://tracker.ceph.com/issues/66989 We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2024/v18-2-4-reef-released/ Notable Changes --------------- * RADOS: This release fixes a bug (https://tracker.ceph.com/issues/61948) where pre-reef clients were allowed to connect to the `pg-upmap-primary` (https://docs.ceph.com/en/reef/rados/operations/read-balancer/) interface despite users having set `require-min-compat-client=reef`, leading to an assert in the osds and mons. You are susceptible to this bug in reef versions prior to 18.2.3 if 1) you are using an osdmap generated via the offline osdmaptool with the `--read` option or 2) you have explicitly generated pg-upmap-primary mappings with the CLI command. Please note that the fix is minimal and does not address corner cases such as adding a mapping in the middle of an upgrade or in a partially upgraded cluster (related trackers linked in https://tracker.ceph.com/issues/61948). As such, we recommend removing any existing pg-upmap-primary mappings until remaining issues are addressed in future point releases. See https://tracker.ceph.com/issues/61948#note-32 for instructions on how to remove existing pg-upmap-primary mappings. * RBD: When diffing against the beginning of time (`fromsnapname == NULL`) in fast-diff mode (`whole_object == true` with `fast-diff` image feature enabled and valid), diff-iterate is now guaranteed to execute locally if exclusive lock is available. This brings a dramatic performance improvement for QEMU live disk synchronization and backup use cases. * RADOS: `get_pool_is_selfmanaged_snaps_mode` C++ API has been deprecated due to being prone to false negative results. It's safer replacement is `pool_is_in_selfmanaged_snaps_mode`. * RBD: The option ``--image-id`` has been added to `rbd children` CLI command, so it can be run for images in the trash. Related tracker: https://tracker.ceph.com/issues/65393 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.4.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: e7ad5345525c7aa95470c26863873b581076945d
Hi Yuri et al, The email announcement includes the fix for 61948, but the linked blog page omits it entirely. Suggest adding that note to the blog page. The language used also differs slightly between the two announcements. email:
We're happy to announce the 4th release in the Reef series. blog: | This is the third backport release in the Reef series. We recommend | that all users update to this release.
On Wed, Jul 24, 2024 at 02:12:25PM -0700, Yuri Weinstein wrote:
We're happy to announce the 4th release in the Reef series. ... Notable Changes --------------- * RADOS: This release fixes a bug (https://tracker.ceph.com/issues/61948) where pre-reef clients were allowed to connect to the `pg-upmap-primary` (https://docs.ceph.com/en/reef/rados/operations/read-balancer/) interface despite users having set `require-min-compat-client=reef`, leading to an assert in the osds and mons. You are susceptible to this bug in reef versions prior to 18.2.3 if 1) you are using an osdmap generated via the offline osdmaptool with the `--read` option or 2) you have explicitly generated pg-upmap-primary mappings with the CLI command. Please note that the fix is minimal and does not address corner cases such as adding a mapping in the middle of an upgrade or in a partially upgraded cluster (related trackers linked in https://tracker.ceph.com/issues/61948). As such, we recommend removing any existing pg-upmap-primary mappings until remaining issues are addressed in future point releases. See https://tracker.ceph.com/issues/61948#note-32 for instructions on how to remove existing pg-upmap-primary mappings. This is the missing item on the webpage.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Dear devel-team, As posted on the ceph-devel Slack channel: Just read the 18.2.4 release notes and the warning pointing tohttps://tracker.ceph.com/issues/66989 <https://tracker.ceph.com/issues/66989>.Unpleasantly surprised that this type of change has been made in a minor release update. RHEL 8 support is planned for up to 2029, why not make this change in the Squid release (allowing one to choose a time to upgrade the base OS)?! With kind regards, Jeroen -- _________________________________________________ Jeroen Roodhart University of Amsterdam Strategic IT Consultant Science faculty j.r.roodhart@uva.nl Researcher's IT support Tel. +31 20 525 7203 feiog-tech-science@uva.nl -- See http://www.uva.nl/profile/j.r.roodhart for openPGP public key
hi Jeroen, On Thu, Jul 25, 2024 at 8:49 AM Jeroen Roodhart <j.r.roodhart@uva.nl> wrote:
Dear devel-team,
As posted on the ceph-devel Slack channel:
Just read the 18.2.4 release notes and the warning pointing tohttps://tracker.ceph.com/issues/66989 <https://tracker.ceph.com/issues/66989>.Unpleasantly surprised that this type of change has been made in a minor release update. RHEL 8 support is planned for up to 2029, why not make this change in the Squid release (allowing one to choose a time to upgrade the base OS)?!
while i understand that this change was disruptive, Ceph has never listed RHEL as a supported distribution: https://docs.ceph.com/en/latest/start/os-recommendations/#platforms we use centos streams to build and test the rpms for our upstream releases. centos stream 8 reached its end of life on May 31 so we've stopped building and testing it on all releases that eol caught us by surprise. with better foresight and planning, we might have dropped stream 8 support for the initial Reef release last August to avoid changing supported distros in a minor release if you aren't ready for an OS upgrade, using Ceph in containers may help in the meantime
With kind regards,
Jeroen
-- _________________________________________________ Jeroen Roodhart University of Amsterdam Strategic IT Consultant Science faculty j.r.roodhart@uva.nl Researcher's IT support Tel. +31 20 525 7203 feiog-tech-science@uva.nl -- See http://www.uva.nl/profile/j.r.roodhart for openPGP public key
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Casey, Thank you for your answer.
if you aren't ready for an OS upgrade, using Ceph in containers may help in the meantime
Then maybe you can put my mind to ease since we are running our cluster containerised since the Pacific release on the advised platform at the time, RHEL 8 (well OK, Alma 8). We have since, as per recommendation, upgraded to Reef. However, as I understand the issue as outlined in the release notes and the call, the problem is that the new containers are performing a syscall that is not properly supported by the kernel of our host systems. If that would be the case and had I not read the release notes, I can only hope that our testing procedure would have caught this issue before going into production. Would the issue be of the kind that produces subtle misbehaviour, there would therefore be a decent chance that this would affect our production environment. Again, this is a minor release. Is it strange to expect minor upgrades not to brake compatibility with the kernel/cgroup versions that were current when going into production? (If yes, I would like to call "foul" and ask for a VAR-decision (apologies for the football reference) :-) ). With kind regards, Jeroen -- _________________________________________________ Jeroen Roodhart University of Amsterdam Strategic IT Consultant Science faculty j.r.roodhart@uva.nl Researcher's IT support Tel. +31 20 525 7203 feiog-tech-science@uva.nl -- See http://www.uva.nl/profile/j.r.roodhart for openPGP public key
This doesn't affect RHEL8 - the linked bug https://bugzilla.redhat.com/show_bug.cgi?id=1985499 shows it was fixed in RHEL in 2021. The issue was only seen on Ubuntu 18.04, which has a much older kernel (RHEL backports many kernel fixes, the version number doesn't tell you much). Later Ubuntu releases are also unaffected. On Thu, Jul 25, 2024 at 10:54 AM Jeroen Roodhart <j.r.roodhart@uva.nl> wrote:
Hi Casey,
Thank you for your answer.
if you aren't ready for an OS upgrade, using Ceph in containers may help in the meantime
Then maybe you can put my mind to ease since we are running our cluster containerised since the Pacific release on the advised platform at the time, RHEL 8 (well OK, Alma 8).
We have since, as per recommendation, upgraded to Reef.
However, as I understand the issue as outlined in the release notes and the call, the problem is that the new containers are performing a syscall that is not properly supported by the kernel of our host systems.
If that would be the case and had I not read the release notes, I can only hope that our testing procedure would have caught this issue before going into production. Would the issue be of the kind that produces subtle misbehaviour, there would therefore be a decent chance that this would affect our production environment.
Again, this is a minor release. Is it strange to expect minor upgrades not to brake compatibility with the kernel/cgroup versions that were current when going into production?
(If yes, I would like to call "foul" and ask for a VAR-decision (apologies for the football reference) :-) ).
With kind regards,
Jeroen
-- _________________________________________________ Jeroen Roodhart University of Amsterdam Strategic IT Consultant Science faculty j.r.roodhart@uva.nl Researcher's IT support Tel. +31 20 525 7203 feiog-tech-science@uva.nl -- See http://www.uva.nl/profile/j.r.roodhart for openPGP public key
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Josh, On 25/07/2024 20:15, Josh Durgin wrote:
The issue was only seen on Ubuntu 18.04, which has a much older kernel (RHEL backports many kernel fixes, the version number doesn't tell you much). Later Ubuntu releases are also unaffected.
That is good to hear, makes it more easy to trust the outcome of our testing. There will be some people on Ubuntu 18.04 though, since it was an supported option under Pacific. Probably best to have a stock answer ready, since I guess there will be questions asked here when people run into this proper ;-) I guess we are lucky for now to be in the RHEL-universe. Thank you for your insights and with kind regards, Jeroen -- _________________________________________________ Jeroen Roodhart University of Amsterdam Strategic IT Consultant Science faculty j.r.roodhart@uva.nl Researcher's IT support Tel. +31 20 525 7203 feiog-tech-science@uva.nl -- See http://www.uva.nl/profile/j.r.roodhart for openPGP public key
Robin, thx We will fix the typo On Wed, Jul 24, 2024 at 5:18 PM Robin H. Johnson <robbat2@gentoo.org> wrote:
Hi Yuri et al,
The email announcement includes the fix for 61948, but the linked blog page omits it entirely.
Suggest adding that note to the blog page.
The language used also differs slightly between the two announcements.
email:
We're happy to announce the 4th release in the Reef series. blog: | This is the third backport release in the Reef series. We recommend | that all users update to this release.
On Wed, Jul 24, 2024 at 02:12:25PM -0700, Yuri Weinstein wrote:
We're happy to announce the 4th release in the Reef series. ... Notable Changes --------------- * RADOS: This release fixes a bug (https://tracker.ceph.com/issues/61948) where pre-reef clients were allowed to connect to the `pg-upmap-primary` (https://docs.ceph.com/en/reef/rados/operations/read-balancer/) interface despite users having set `require-min-compat-client=reef`, leading to an assert in the osds and mons. You are susceptible to this bug in reef versions prior to 18.2.3 if 1) you are using an osdmap generated via the offline osdmaptool with the `--read` option or 2) you have explicitly generated pg-upmap-primary mappings with the CLI command. Please note that the fix is minimal and does not address corner cases such as adding a mapping in the middle of an upgrade or in a partially upgraded cluster (related trackers linked in https://tracker.ceph.com/issues/61948). As such, we recommend removing any existing pg-upmap-primary mappings until remaining issues are addressed in future point releases. See https://tracker.ceph.com/issues/61948#note-32 for instructions on how to remove existing pg-upmap-primary mappings. This is the missing item on the webpage.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi,
On 25 Jul 2024, at 00:12, Yuri Weinstein <yweinste@redhat.com> wrote:
We're happy to announce the 4th release in the Reef series.
The repo of reef now returns 404
GET /rpm-reef/el8/SRPMS/ HTTP/1.1 Host: download.ceph.com
< HTTP/1.1 404 Not Found If change the repo to previous version, the answer 200 as expected
GET /rpm-18.2.2/el8/SRPMS/ HTTP/1.1 Host: download.ceph.com
< HTTP/1.1 200 OK Seems something wrong with packages k
Rook users are seeing OSDs fail on arm64 with v18.2.4. I would think it also affects non-rook users. Tracker opened: https://tracker.ceph.com/issues/67213 Thanks, Travis On Wed, Jul 24, 2024 at 3:13 PM Yuri Weinstein <yweinste@redhat.com> wrote:
We're happy to announce the 4th release in the Reef series.
An early build of this release was accidentally exposed and packaged as 18.2.3 by the Debian project in April. That 18.2.3 release should not be used. The official release was re-tagged as v18.2.4 to avoid further confusion.
v18.2.4 container images, now based on CentOS 9, may be incompatible on older kernels (e.g., Ubuntu 18.04) due to differences in thread creation methods. Users upgrading to v18.2.4 container images with older OS versions may encounter crashes during `pthread_create`. For workarounds, refer to the related tracker. However, we recommend upgrading your OS to avoid this unsupported combination. Related tracker: https://tracker.ceph.com/issues/66989
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2024/v18-2-4-reef-released/
Notable Changes --------------- * RADOS: This release fixes a bug (https://tracker.ceph.com/issues/61948) where pre-reef clients were allowed to connect to the `pg-upmap-primary` (https://docs.ceph.com/en/reef/rados/operations/read-balancer/) interface despite users having set `require-min-compat-client=reef`, leading to an assert in the osds and mons. You are susceptible to this bug in reef versions prior to 18.2.3 if 1) you are using an osdmap generated via the offline osdmaptool with the `--read` option or 2) you have explicitly generated pg-upmap-primary mappings with the CLI command. Please note that the fix is minimal and does not address corner cases such as adding a mapping in the middle of an upgrade or in a partially upgraded cluster (related trackers linked in https://tracker.ceph.com/issues/61948). As such, we recommend removing any existing pg-upmap-primary mappings until remaining issues are addressed in future point releases. See https://tracker.ceph.com/issues/61948#note-32 for instructions on how to remove existing pg-upmap-primary mappings. * RBD: When diffing against the beginning of time (`fromsnapname == NULL`) in fast-diff mode (`whole_object == true` with `fast-diff` image feature enabled and valid), diff-iterate is now guaranteed to execute locally if exclusive lock is available. This brings a dramatic performance improvement for QEMU live disk synchronization and backup use cases. * RADOS: `get_pool_is_selfmanaged_snaps_mode` C++ API has been deprecated due to being prone to false negative results. It's safer replacement is `pool_is_in_selfmanaged_snaps_mode`. * RBD: The option ``--image-id`` has been added to `rbd children` CLI command, so it can be run for images in the trash.
Related tracker: https://tracker.ceph.com/issues/65393
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.4.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: e7ad5345525c7aa95470c26863873b581076945d _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
participants (7)
-
Casey Bodley
-
Jeroen Roodhart
-
Josh Durgin
-
Konstantin Shalygin
-
Robin H. Johnson
-
Travis Nielsen
-
Yuri Weinstein