We're happy to announce the 5th point release in the Reef series. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/ Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface. * RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired. Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1
Hi all, started an update on our staging cluster from v18.2.4 --> v18.2.5 ~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time. I was able to bring osds up again by redewploying with the old container image v18.2.4 ~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4 Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ... ~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23)) Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
Ok the two issues I see with reef release v18.2.5 - Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters. - common/pick_address: check if address in subnet all public address ( pr#57590 <https://github.com/ceph/ceph/pull/57590>, Nitzan Mordechai) - osd: Report health error if OSD public address is not within subnet ( pr#55697 <https://github.com/ceph/ceph/pull/55697>, Prashant D) - cryptsetup version check isn't working at least in the container image of v18.2.5 ( https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr...) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '" Happy to help with logs etc. BR Stephan Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn < stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
Ah, I wasn't certain if this was my environment or not. I was encountering the cryptsetup thing. It looks like this commit would fix it: https://github.com/ceph/ceph/commit/607eb34b2c278566c386efcbf3018629cf08ccfd Or, as a dirty yuck, I've worked around this via: sed -i "s/pattern:.*/pattern: str = r'(\\\d+\\\.?)+'/g" /usr/lib/python3.9/site-packages/ceph_volume/util/encryption.py sed -i "s/cryptsetup_version = re.*/cryptsetup_version = re.search(pattern, out[0])/g" /usr/lib/python3.9/site-packages/ceph_volume/util/encryption.py in the containers that I issue a `ceph-volume lvm activate ...` for. On 2025-04-11 13:33, Stephan Hohn wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
*
common/pick_address: check if address in subnet all public address (pr#57590 [6], Nitzan Mordechai) *
osd: Report health error if OSD public address is not within subnet (pr#55697 [7], Prashant D)
- cryptsetup version check isn't working at least in the container image of v18.2.5 (https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr...) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
Happy to help with logs etc.
BR
Stephan
Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn <stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 [2] Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 [2] Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name [3]=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE=quay.io/centos/centos:stream9 [4], org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name [3]=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE=quay.io/centos/centos:stream9 [4], org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name [3]=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_REF=reef,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE=quay.io/centos/centos:stream9 [4], OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
org.label-schema.vendor=CentOS, org.label-schema.name [3]=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9 [4], io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 [2] Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name [3]=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE=quay.io/centos/centos:stream9 [4]) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name [3]=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9 [4], org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9 [4], org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name [3]=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name [3]=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9 [4]) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4 [5]
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well.
BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein <yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git [1] * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
Links: ------ [1] http://github.com/ceph/ceph.git [2] http://quay.io/ceph/ceph:v18.2.5 [3] http://org.label-schema.name [4] http://quay.io/centos/centos:stream9 [5] http://quay.io/ceph/ceph:v18.2.4 [6] https://github.com/ceph/ceph/pull/57590 [7] https://github.com/ceph/ceph/pull/55697 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
For completeness, I've cherry picked the referenced commit and proposed it here: https://github.com/ceph/ceph/pull/62791 On 2025-04-11 13:52, Matt Vandermeulen wrote:
Ah, I wasn't certain if this was my environment or not. I was encountering the cryptsetup thing. It looks like this commit would fix it: https://github.com/ceph/ceph/commit/607eb34b2c278566c386efcbf3018629cf08ccfd
Or, as a dirty yuck, I've worked around this via:
sed -i "s/pattern:.*/pattern: str = r'(\\\d+\\\.?)+'/g" /usr/lib/python3.9/site-packages/ceph_volume/util/encryption.py sed -i "s/cryptsetup_version = re.*/cryptsetup_version = re.search(pattern, out[0])/g" /usr/lib/python3.9/site-packages/ceph_volume/util/encryption.py
in the containers that I issue a `ceph-volume lvm activate ...` for.
On 2025-04-11 13:33, Stephan Hohn wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
*
common/pick_address: check if address in subnet all public address (pr#57590 [6], Nitzan Mordechai) *
osd: Report health error if OSD public address is not within subnet (pr#55697 [7], Prashant D)
- cryptsetup version check isn't working at least in the container image of v18.2.5 (https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr...) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
Happy to help with logs etc.
BR
Stephan
Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn <stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 [2] Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 [2] Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name [3]=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE=quay.io/centos/centos:stream9 [4], org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name [3]=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE=quay.io/centos/centos:stream9 [4], org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name [3]=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_REF=reef,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE=quay.io/centos/centos:stream9 [4], OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
org.label-schema.vendor=CentOS, org.label-schema.name [3]=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9 [4], io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 [2] Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name [3]=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE=quay.io/centos/centos:stream9 [4]) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name [3]=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9 [4], org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9 [4], org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name [3]=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image=quay.io/ceph/ceph:v18.2.5 [2], name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs...,
io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name [3]=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9 [4]) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4 [5]
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well.
BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein <yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git [1] * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
Links: ------ [1] http://github.com/ceph/ceph.git [2] http://quay.io/ceph/ceph:v18.2.5 [3] http://org.label-schema.name [4] http://quay.io/centos/centos:stream9 [5] http://quay.io/ceph/ceph:v18.2.4 [6] https://github.com/ceph/ceph/pull/57590 [7] https://github.com/ceph/ceph/pull/55697 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi, On 4/11/25 18:33, Stephan Hohn wrote:
- cryptsetup version check isn't working at least in the container image of v18.2.5 ( https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr...) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
The release seems to be broken. In 18.2.4 the support for cryptsetup in newer RHEL releases was missing. This also affected Debian 12. A patch is available in a bug tracker entry (not sure which one). This patch seems to be applied to 18.2.5, but someone seems to mixed up the functions in the python re packages.... The output of 'cryptsetup --version' (at least for Debian 12) is: (Pdb) p out ['cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI '] The regex in https://github.com/ceph/ceph/blob/v18.2.5/src/ceph-volume/ceph_volume/util/e... is applied using 'match', which matches at the _beginning_ of the string. The original patch uses a slightly different regex and the _search_ function. Changing this fixes the bug: (Pdb) n
/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py(50)set_dmcrypt_no_workqueue() -> cryptsetup_version = re.search(pattern, out[0]) (Pdb) n
/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py(52)set_dmcrypt_no_workqueue() -> if cryptsetup_version is None: (Pdb) p out[0] 'cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI ' (Pdb) p pattern '\\b\\d+(\\.\\d+)*\\b' (Pdb) p cryptsetup_version <re.Match object; span=(11, 16), match='2.6.1'> Time for a hotfix... Best regards, Burkhard Linkle
Hi Stephan, Yes, the limitation to IPv4-only subnet checks is understood and should be addressed by the reef backport PR: https://github.com/ceph/ceph/pull/62814. This PR includes the necessary changes to handle IPv6 subnets correctly. Regards, Prashant On Fri, Apr 11, 2025 at 12:36 PM Stephan Hohn <stephan@gridscale.io> wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
-
common/pick_address: check if address in subnet all public address ( pr#57590 <https://github.com/ceph/ceph/pull/57590>, Nitzan Mordechai) -
osd: Report health error if OSD public address is not within subnet ( pr#55697 <https://github.com/ceph/ceph/pull/55697>, Prashant D)
- cryptsetup version check isn't working at least in the container image of v18.2.5 (
https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr... ) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
Happy to help with logs etc.
BR
Stephan
Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn < stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Parshant thanks for the update. @Yuri Weinstein <yweinste@redhat.com> Is someone already looking into the cryptsetup issue I think it got introduced with this change here "ceph-volume: fix set_dmcrypt_no_workqueue() (pr#58997 <https://github.com/ceph/ceph/pull/58997>, Guillaume Abrioux)" Do you know if we have to wait for the next patch release or will there be a hotfix release soon? BR Stephan Am Di., 15. Apr. 2025 um 23:05 Uhr schrieb Prashant Dhange < pdhange@redhat.com>:
Hi Stephan,
Yes, the limitation to IPv4-only subnet checks is understood and should be addressed by the reef backport PR: https://github.com/ceph/ceph/pull/62814. This PR includes the necessary changes to handle IPv6 subnets correctly.
Regards, Prashant
On Fri, Apr 11, 2025 at 12:36 PM Stephan Hohn <stephan@gridscale.io> wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
-
common/pick_address: check if address in subnet all public address ( pr#57590 <https://github.com/ceph/ceph/pull/57590>, Nitzan Mordechai) -
osd: Report health error if OSD public address is not within subnet ( pr#55697 <https://github.com/ceph/ceph/pull/55697>, Prashant D)
- cryptsetup version check isn't working at least in the container image of v18.2.5 (
https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr... ) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
Happy to help with logs etc.
BR
Stephan
Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn < stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL=
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... ,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL=
io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64
https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs... , *
~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We are working on 18.2.6 hotfix On Wed, Apr 16, 2025 at 5:32 AM Stephan Hohn <stephan@gridscale.io> wrote:
Hi Parshant
thanks for the update.
@Yuri Weinstein Is someone already looking into the cryptsetup issue
I think it got introduced with this change here "ceph-volume: fix set_dmcrypt_no_workqueue() (pr#58997, Guillaume Abrioux)"
Do you know if we have to wait for the next patch release or will there be a hotfix release soon?
BR
Stephan
Am Di., 15. Apr. 2025 um 23:05 Uhr schrieb Prashant Dhange <pdhange@redhat.com>:
Hi Stephan,
Yes, the limitation to IPv4-only subnet checks is understood and should be addressed by the reef backport PR: https://github.com/ceph/ceph/pull/62814. This PR includes the necessary changes to handle IPv6 subnets correctly.
Regards, Prashant
On Fri, Apr 11, 2025 at 12:36 PM Stephan Hohn <stephan@gridscale.io> wrote:
Ok the two issues I see with reef release v18.2.5
- Subnet check seems to be ipv4 only which leads to e.g "public address is not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters.
-
common/pick_address: check if address in subnet all public address ( pr#57590 <https://github.com/ceph/ceph/pull/57590>, Nitzan Mordechai) -
osd: Report health error if OSD public address is not within subnet ( pr#55697 <https://github.com/ceph/ceph/pull/55697>, Prashant D)
- cryptsetup version check isn't working at least in the container image of v18.2.5 ( https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encr...) which leads to encrypted osds not starting due to "'Error while checking cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '"
Happy to help with logs etc.
BR
Stephan
Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn < stephan@gridscale.io>:
Hi all,
started an update on our staging cluster from v18.2.4 --> v18.2.5
~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and Mgr went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.778833855 +0000 UTC m=+0.138709869 container init 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325) Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.790749299 +0000 UTC m=+0.150625308 container start 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) Apr 11 06:59:56 0cc47a6df14e bash[263290]: 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for 03977a23-f00f-4bb0-b9a7-de57f40ba853. Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.90105365 +0000 UTC m=+0.076310419 container died 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 06:59:56.948423169 +0000 UTC m=+0.123679914 container remove 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:56 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process exited, code=exited, status=1/FAILURE Apr 11 06:59:57 0cc47a6df14e podman[263966]: Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.495704469 +0000 UTC m=+0.105177519 container create d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.39.3, org.opencontainers.image.documentation= https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, OSD_FLAVOR=default) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.447692243 +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.532516757 +0000 UTC m=+0.141989804 container init d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, CEPH_REF=reef, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO= https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation= https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, org.label-schema.build-date=20250325, FROM_IMAGE= quay.io/centos/centos:stream9) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542542891 +0000 UTC m=+0.152015938 container start d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE= quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO= https://github.com/ceph/ceph.git) Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 06:59:57.542633919 +0000 UTC m=+0.152106981 container attach d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, org.label-schema.name=CentOS Stream 9 Base Image) Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. Apr 11 06:59:58 0cc47a6df14e ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero exit status: 4 Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.401862967 +0000 UTC m=+1.011336030 container died d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 06:59:58.456490831 +0000 UTC m=+1.065963881 container remove d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= quay.io/ceph/ceph:v18.2.5, name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, org.opencontainers.image.authors=Ceph Release Team < ceph-maintainers@ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL= https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, org.label-schema.license=GPLv2, CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with result 'exit-code'. Apr 11 06:59:58 0cc47a6df14e systemd[1]: ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed 2.252s CPU time.
I was able to bring osds up again by redewploying with the old container image v18.2.4
~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4
Everything is up again but seeing the this error now. ~# ceph health detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are not reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet ...
~# ceph config dump | grep netw global advanced public_network fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 * ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 [::]:* users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23))
Looks like some see the same problem with latest squid release v19.2.2 as well. BR Stephan
Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < yweinste@redhat.com>:
We're happy to announce the 5th point release in the Reef series.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/
Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 _______________________________________________ Ceph-announce mailing list -- ceph-announce@ceph.io To unsubscribe send an email to ceph-announce-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Burkhard Linke
-
Matt Vandermeulen
-
Prashant Dhange
-
Stephan Hohn
-
Yuri Weinstein