v18.2.5 Reef released
We're happy to announce the 5th point release in the Reef series. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/ Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface. * RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired. Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1
Hi all, started an update on our staging cluster from v18.2.4 --> v18.2.5 ~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time. I was able to bring osds up again by redewploying with the old container image v18.2.4 ~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4 Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ... ~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23)) Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
Ok the two issues I see with reef release v18.2.5 - Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters. - common/pick_address: check if address in subnet all public address ( pr#57590 <https://github.com/ceph/ceph/pull/57590>, Nitzan Mordechai) - osd: Report health error if OSD public address is not within subnet ( pr#55697 <https://github.com/ceph/ceph/pull/55697>, Prashant D) - cryptsetup version check isn't working at least in the container image of v18.2.5 ( https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr...) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '" Happy to help with logs etc. BR Stephan Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn < stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
Hi, On 4/11/25 18:33, Stephan Hohn wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
I'm just updating to 18.2.5 and encountered the same problem in a IPv4-only cluster. One mon and one mgr are running on a machine in a different network, and it sets public_network locally in ceph.conf to its own ip range. All other hosts have a different values in ceph.conf. There was no value stored in the mon config database. The error popped up after the mgr debian package was updated (which for some probably undesired reason also restarts the service...). It refers to the ip range of the single extra machine. Setting a global value in ceph.conf did not solve the problem, so I disabled first the mon and then the mgr on the host. With the later step the error was gone. So I assume the either the mgr is assuming that all services are in its configured public_network, or the other mons took some time after disabling the mon to reconcile the problem. No containers involved, everything running on Debian 12, only public network for all services. I'll update the remaining services and afterwards run some tests on the single machine for further investigations. Best regards, Burkhard Linke
Hi Burkhard, Please provide the following additional information to help clarify the uncertainty regarding the IPv4 address and the impact of the warning message: Could you please provide: 1. The exact output of the "dump of configuration" you mentioned, specifically the section showing the public_network setting on the monitors and OSDs. This will help us understand the current configuration. 2. Confirmation of whether you are experiencing any real problems beyond the warning message you are seeing regarding the public address and the subnet. Specifically, are you seeing any issues with: - OSD availability? - Network connectivity? - Data replication or recovery? - Client access to the Ceph cluster? Understanding the actual impact, beyond just the warning, will help us prioritize and address the issue effectively. Please open a tracker if you believe this is a bug. Regards, Prashant On Mon, Apr 14, 2025 at 4:08 AM Burkhard Linke < Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
Hi,
On 4/11/25 18:33, Stephan Hohn wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
I'm just updating to 18.2.5 and encountered the same problem in a IPv4-only cluster.
One mon and one mgr are running on a machine in a different network, and it sets public_network locally in ceph.conf to its own ip range. All other hosts have a different values in ceph.conf. There was no value stored in the mon config database.
The error popped up after the mgr debian package was updated (which for some probably undesired reason also restarts the service...). It refers to the ip range of the single extra machine. Setting a global value in ceph.conf did not solve the problem, so I disabled first the mon and then the mgr on the host. With the later step the error was gone. So I assume the either the mgr is assuming that all services are in its configured public_network, or the other mons took some time after disabling the mon to reconcile the problem.
No containers involved, everything running on Debian 12, only public network for all services.
I'll update the remaining services and afterwards run some tests on the single machine for further investigations.
Best regards,
Burkhard Linke
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 4/11/25 18:33, Stephan Hohn wrote:
- cryptsetup version check isn't working at least in the container image of v18.2.5 ( https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr...) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
The release seems to be broken. In 18.2.4 the support for cryptsetup in newer RHEL releases was missing. This also affected Debian 12. A patch is available in a bug tracker entry (not sure which one). This patch seems to be applied to 18.2.5, but someone seems to mixed up the functions in the python re packages.... The output of 'cryptsetup --version' (at least for Debian 12) is: (Pdb) p out ['cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI '] The regex in https://github.com/ceph/ceph/blob/v18.2.5/src/ceph-volume/ceph_volume/util/e... is applied using 'match', which matches at the _beginning_ of the string. The original patch uses a slightly different regex and the _search_ function. Changing this fixes the bug: (Pdb) n
/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py(50)set_dmcrypt_no_workqueue() -> cryptsetup_version = re.search(pattern, out[0]) (Pdb) n
/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py(52)set_dmcrypt_no_workqueue() -> if cryptsetup_version is None: (Pdb) p out[0] 'cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI ' (Pdb) p pattern '\\b\\d+(\\.\\d+)*\\b' (Pdb) p cryptsetup_version <re.Match object; span=(11, 16), match='2.6.1'> Time for a hotfix... Best regards, Burkhard Linkle
Hi, On 4/11/25 18:33, Stephan Hohn wrote:
Ok the two issues I see with reef release v18.2.5
and additionally, it fails to compile again with gcc-14 (debian testing/unstable, default options). ---snip--- /build/ceph-18.2.5/src/common/options.h:132:18: error: ‘static Option::level_t Option::str_to_level(std::string_view)’ cannot be overloaded with ‘s tatic Option::level_t Option::str_to_level(std::string_view)’ 132 | static level_t str_to_level(std::string_view s) { | ^~~~~~~~~~~~ /build/ceph-18.2.5/src/common/options.h:120:18: note: previous declaration ‘static Option::level_t Option::str_to_level(std::string_view)’ 120 | static level_t str_to_level(std::string_view s) { | ^~~~~~~~~~~~ ---snap--- this was introduced in commit 0d94eebb0dd. Regards, Daniel
Hi Stephan, Yes, the limitation to IPv4-only subnet checks is understood and should be addressed by the reef backport PR: https://github.com/ceph/ceph/pull/62814. This PR includes the necessary changes to handle IPv6 subnets correctly. Regards, Prashant On Fri, Apr 11, 2025 at 12:36 PM Stephan Hohn <stephan@gridscale.io> wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
-
common/pick_address: check if address in subnet all public address ( pr#57590 <https://github.com/ceph/ceph/pull/57590>, Nitzan Mordechai) -
osd: Report health error if OSD public address is not within subnet ( pr#55697 <https://github.com/ceph/ceph/pull/55697>, Prashant D)
- cryptsetup version check isn't working at least in the container image of v18.2.5 (
https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr... ) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
Happy to help with logs etc.
BR
Stephan
Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn < stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Parshant thanks for the update. @Yuri Weinstein <yweinste@redhat.com> Is someone already looking into the cryptsetup issue I think it got introduced with this change here "ceph-volume: fix set_dmcrypt_no_workqueue() (pr#58997 <https://github.com/ceph/ceph/pull/58997>, Guillaume Abrioux)" Do you know if we have to wait for the next patch release or will there be a hotfix release soon? BR Stephan Am Di., 15. Apr. 2025 um 23:05 Uhr schrieb Prashant Dhange < pdhange@redhat.com>:
Hi Stephan,
Yes, the limitation to IPv4-only subnet checks is understood and should be addressed by the reef backport PR: https://github.com/ceph/ceph/pull/62814. This PR includes the necessary changes to handle IPv6 subnets correctly.
Regards, Prashant
On Fri, Apr 11, 2025 at 12:36 PM Stephan Hohn <stephan@gridscale.io> wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
-
common/pick_address: check if address in subnet all public address ( pr#57590 <https://github.com/ceph/ceph/pull/57590>, Nitzan Mordechai) -
osd: Report health error if OSD public address is not within subnet ( pr#55697 <https://github.com/ceph/ceph/pull/55697>, Prashant D)
- cryptsetup version check isn't working at least in the container image of v18.2.5 (
https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr... ) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
Happy to help with logs etc.
BR
Stephan
Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn < stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL=
io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... , *
~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We are working on 18.2.6 hotfix On Wed, Apr 16, 2025 at 5:32 AM Stephan Hohn <stephan@gridscale.io> wrote:
Hi Parshant
thanks for the update.
@Yuri Weinstein Is someone already looking into the cryptsetup issue
I think it got introduced with this change here "ceph-volume: fix set_dmcrypt_no_workqueue() (pr#58997, Guillaume Abrioux)"
Do you know if we have to wait for the next patch release or will there be a hotfix release soon?
BR
Stephan
Am Di., 15. Apr. 2025 um 23:05 Uhr schrieb Prashant Dhange <pdhange@redhat.com>:
Hi Stephan,
Yes, the limitation to IPv4-only subnet checks is understood and should be addressed by the reef backport PR: https://github.com/ceph/ceph/pull/62814. This PR includes the necessary changes to handle IPv6 subnets correctly.
Regards, Prashant
On Fri, Apr 11, 2025 at 12:36 PM Stephan Hohn <stephan@gridscale.io> wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
-
common/pick_address: check if address in subnet all public address ( pr#57590 <https://github.com/ceph/ceph/pull/57590>, Nitzan Mordechai) -
osd: Report health error if OSD public address is not within subnet ( pr#55697 <https://github.com/ceph/ceph/pull/55697>, Prashant D)
- cryptsetup version check isn't working at least in the container image of v18.2.5 ( https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr...) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
Happy to help with logs etc.
BR
Stephan
Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn < stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Burkhard Linke
-
Daniel Baumann
-
Prashant Dhange
-
Stephan Hohn
-
Yuri Weinstein