- ceph-users - lists.ceph.io

Ceph crash service deployed with cephadm: unable to find a keyring
by Daniel Vogelbacher 16 Apr '25

16 Apr '25

Hello, recently I checked my server logs for error events and got some hits for the ceph crash service, deployed with cephadm. When restarted, crash service logs to journald: auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory AuthRegistry(0x7f21a0068da0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory AuthRegistry(0x7f21a861bff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 13] RADOS permission denied (error connecting to the cluster) The image is mounted with: -v /var/lib/ceph/7644057a-00f6-11f0-9a0c-eac00fed9338/crash.virt-master3/keyring:/etc/ceph/ceph.client.crash.virt-master3.keyring so I assume there should be a key available, but crash daemon searches for a admin keyring instead of "client.crash.virt-master3". Any advice how to fix this error? I see these error logs on all machines where crash service is deployed. I've tried a redeployment without any effect. # ceph -v ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable) # systemctl restart ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service # journalctl -u ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service -ef Output: Apr 12 18:36:24 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1402272]: *** Interrupted with signal 15 *** Apr 12 18:36:24 virt-master3 podman[1408122]: 2025-04-12 18:36:24.992684396 +0200 CEST m=+0.043782773 container died 6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_REF=squid, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, io.buildah.version=1.33.7, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20250124, OSD_FLAVOR=default) Apr 12 18:36:25 virt-master3 podman[1408122]: 2025-04-12 18:36:25.016797163 +0200 CEST m=+0.067895534 container remove 6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20250124, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, io.buildah.version=1.33.7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default, org.label-schema.license=GPLv2, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>) Apr 12 18:36:25 virt-master3 bash[1408104]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3 Apr 12 18:36:25 virt-master3 systemd[1]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service: Deactivated successfully. Apr 12 18:36:25 virt-master3 systemd[1]: Stopped ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338. Apr 12 18:36:25 virt-master3 systemd[1]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service: Consumed 1.540s CPU time. Apr 12 18:36:25 virt-master3 systemd[1]: Starting ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338... Apr 12 18:36:25 virt-master3 podman[1408259]: Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.498991532 +0200 CEST m=+0.071935136 container create bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.build-date=20250124, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>) Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.462816456 +0200 CEST m=+0.035760071 image pull quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.589479015 +0200 CEST m=+0.162422619 container init bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.build-date=20250124, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=squid, io.buildah.version=1.33.7) Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.595018205 +0200 CEST m=+0.167961840 container start bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20250124, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.vendor=CentOS, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers(a)ceph.io>, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) Apr 12 18:36:25 virt-master3 bash[1408259]: bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 Apr 12 18:36:25 virt-master3 systemd[1]: Started ceph-7644057a-00f6-11f0-9a0c-eac00fed9338(a)crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338. Apr 12 18:36:25 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: INFO:ceph-crash:pinging cluster to exercise our key Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 AuthRegistry(0x7f21a0068da0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 AuthRegistry(0x7f21a861bff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a5b91640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a6392640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: [errno 13] RADOS permission denied (error connecting to the cluster) Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s -- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher

3 5

v18.2.5 Reef released
by Yuri Weinstein 16 Apr '25

16 Apr '25

We're happy to announce the 5th point release in the Reef series. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/ Notable Changes --------------- * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default and is now deprecated. If the NBD netlink interface is not supported by the kernel, then the mapping is retried using the legacy ioctl interface. * RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired. Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1

5 9

Reef 18.2.5: upgraded OSDs are shown as crashed
by Eugen Block 16 Apr '25

16 Apr '25

Hi, I'm currently upgrading the first customer cluster to latest Reef (18.2.5). I haven't seen that in my lab clusters yet, but it might be due to insufficient crash configuration (or the shutdown_timeout is not triggered). I found this backport for 18.2.5: https://tracker.ceph.com/issues/63527 Am I interpreting correctly that the assert is triggered because the OSDs were shut down on 18.2.4, but this won't happen anymore on 18.2.5? Thanks! Eugen ---snip--- root@storage01:~# ceph crash info 2025-04-16T08:51:38.757577Z_bd34dfdd-183a-4274-8a40-a6cb3311ca56 { "archived": "2025-04-16 08:52:50.374393", "assert_condition": "end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/osd/OSD.cc", "assert_func": "int OSD::shutdown()", "assert_line": 4585, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/osd/OSD.cc: In function 'int OSD::shutdown()' thread 7f313de0a640 time 2025-04-16T08:51:38.751362+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/osd/OSD.cc: 4585: FAILED ceph_assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)\n", "assert_thread_name": "signal_handler", "backtrace": [ "/lib64/libc.so.6(+0x3e6f0) [0x7f31412d16f0]", "/lib64/libc.so.6(+0x8b94c) [0x7f314131e94c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56015bef62b2]", "/usr/bin/ceph-osd(+0x3c9413) [0x56015bef6413]", "/usr/bin/ceph-osd(+0x3591d7) [0x56015be861d7]", "/usr/bin/ceph-osd(+0xa1da57) [0x56015c54aa57]", "/lib64/libc.so.6(+0x89c02) [0x7f314131cc02]", "/lib64/libc.so.6(+0x10ec40) [0x7f31413a1c40]" ], "ceph_version": "18.2.4", "crash_id": "2025-04-16T08:51:38.757577Z_bd34dfdd-183a-4274-8a40-a6cb3311ca56", "entity_name": "osd.162", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-osd", "stack_sig": "434a67dd4e3c173992edcf6eb5dc6014d029ca333c4904eced28762dc1959db1", "timestamp": "2025-04-16T08:51:38.757577Z", "utsname_hostname": "storage04", "utsname_machine": "x86_64", "utsname_release": "5.15.0-130-generic", "utsname_sysname": "Linux", "utsname_version": "#140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024" } ---snip---

2 2

"BUG: soft lockup" with MDS
by Janek Bevendorff 16 Apr '25

16 Apr '25

Hi, Since the latest Reef update I have the problem that some of my hosts suddenly go into a state where all CPUs are stuck in kernel mode causing all daemons on that host to become unresponsive. When I connect to the IPMI console, I see a lot of messages like: watchdog: BUG: soft lockup - CPU#8 stuck for 47s! [cron:868840] (it's basically a list of all processes running on the machine). Usually, this resolves itself after several minutes, but sometimes I have to hardreset the host. When this happens, all daemons are marked as down and I cannot interact with the host at all. I don't know what causes this but, I think it happens primarily on the hosts where my MDS run and it seems to be triggered by events such as cluster rebalances, MDS restarts, or just randomly. I found a few reports about similar issues on the bug tracker and mailing list, but they are all very unspecific, unanswered, or more than 6 years old. Is there any way I can debug this? I upgraded to Squid already, but that didn't solve the problem. I also had massive issues with this during the upgrade. Particularly at the end when the MDS were upgraded, I had constant struggles with it. I had to set the noout flag and then literally sit next to it to resume the upgrade every few minutes until it finally went through, because random MDS hosts went intermittently dark all the time. All hosts run Ubuntu 22.04 with kernel 6.8.0. Any ideas? Thanks! Janek

1 0

Announcing go-ceph v0.33.0
by John Mulligan 15 Apr '25

15 Apr '25

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.33.0 Changes include additions to the rbd, rgw, and cephfs packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project.

1 0

All OSD_UNREACHABLE in IPv6 cluster - roleback or wait?
by Björn Lässig 15 Apr '25

15 Apr '25

Hi, after upgrading from 18.2.4 to 18.2.5, I have the result that the health status of all Ceph clusters that do not use legacyIP is always HEALTH_ERR for no useful reason. Do we get a new stable Release soon (including https://github.com/ceph/ceph/pull/61323 ) or should i roleback to 18.2.4? have a nice day Björn Lässig PS: attached is some output of the errors -- cluster: id: <myclusterUUID> health: HEALTH_ERR 44 osds(s) are not reachable services: mon: 5 daemons, quorum marvin06,marvin08,cephmon1,cephmon3,cephmon2 (age 2d) mgr: cephmgr2.zvtgjh(active, since 2d), standbys: cephmgr1.sxadnq mds: 2/2 daemons up, 2 standby osd: 44 osds: 44 up (since 9h), 44 in (since 6M) data: volumes: 2/2 healthy pools: 7 pools, 1953 pgs objects: 6.51M objects, 17 TiB usage: 50 TiB used, 174 TiB / 224 TiB avail pgs: 1951 active+clean 2 active+clean+scrubbing+deep [root@ceph:~] # ceph health detail [ERR] OSD_UNREACHABLE: 44 osds(s) are not reachable osd.0's public address is not in '2001:db8:0:2::/64' subnet [… every osd of this cluster …] osd.43's public address is not in '2001:db8:0:2::/64' subnet but this is all false [root@marvin09:~] 1 # ss -tlpne | grep ceph-<myclusterUUID>@osd.43.service | sed 's/users.*//' LISTEN 0 512 [2001:db8:0:3:1d::3f]:6812 [::]:* LISTEN 0 512 [2001:db8:0:3:1d::3f]:6813 [::]:* LISTEN 0 512 [2001:db8:0:3:1d::3f]:6814 [::]:* LISTEN 0 512 [2001:db8:0:3:1d::3f]:6815 [::]:* LISTEN 0 512 [2001:db8:0:2:1d::3f]:6818 [::]:* LISTEN 0 512 [2001:db8:0:2:1d::3f]:6819 [::]:* LISTEN 0 512 [2001:db8:0:2:1d::3f]:6816 [::]:* LISTEN 0 512 [2001:db8:0:2:1d::3f]:6817 [::]:* [root@marvin09:~] # ceph config get osd.43 | grep public_network global advanced public_network 2001:db8:0:2::/64 * global advanced public_network_interface rbdfenet *

2 3

Urgent help: I accidentally nuked all my Monitor
by Jonas Schwab 15 Apr '25

15 Apr '25

Hello everyone, I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup. I am very grateful for all help! Best regards, Jonas Schwab

4 18

Upcoming Ceph Days - CFP and Join Us
by Dan van der Ster 15 Apr '25

15 Apr '25

Dear Ceph Community, Just a quick note to highlight two upcoming Ceph Days events: * Seattle, May 15: https://ceph.io/en/community/events/2025/ceph-day-seattle/ * London, June 4: https://ceph.io/en/community/events/2025/ceph-days-london/ Hope to see you there! -- Dan

3 5

Unbalance between index pool osds in rgw
by Jaemin Joo 15 Apr '25

15 Apr '25

Hi all, I am testing rgw cluster which separated between index pool osd and data pool osd. After uploading a lot of objects, I found that index pool osd usage is unbalanced. I just know that index pool use rocksdb. rocksdb has object metadata, bucket metadata, multipart, versioning data for index pool. I thought that most of usage is object metadata. so I checked balance state of object metadata between osds of index pool. it's balanced well. (I know it though OMAP stat). Which part can make unbalance between index pool disks? (you can check osd.27, osd.29 which are bigger than other osds) ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 24 ssd 3.49309 1.00000 3.5 TiB 1.8 TiB 399 MiB 1.2 TiB 581 GiB 1.7 TiB 51.72 0.85 71 up 25 ssd 3.49309 1.00000 3.5 TiB 2.2 TiB 179 MiB 1.4 TiB 836 GiB 1.3 TiB 62.22 1.02 71 up 26 ssd 3.49309 1.00000 3.5 TiB 2.0 TiB 180 MiB 1.2 TiB 752 GiB 1.5 TiB 56.55 0.93 72 up *27 ssd 3.49309 1.00000 3.5 TiB 2.7 TiB 399 MiB 1.3 TiB 1.4 TiB 815 GiB 77.22 1.27 70 up* 28 ssd 3.49309 1.00000 3.5 TiB 1.5 TiB 179 MiB 1.2 TiB 321 GiB 2.0 TiB 43.57 0.71 72 up *29 ssd 3.49309 1.00000 3.5 TiB 2.8 TiB 179 MiB 1.4 TiB 1.4 TiB 748 GiB 79.08 1.30 73 up* 30 ssd 3.49309 1.00000 3.5 TiB 1.7 TiB 179 MiB 1.4 TiB 342 GiB 1.8 TiB 49.37 0.81 75 up 31 ssd 3.49309 1.00000 3.5 TiB 2.5 TiB 179 MiB 1.4 TiB 1.2 TiB 969 GiB 72.90 1.20 69 up 32 ssd 3.49309 1.00000 3.5 TiB 2.4 TiB 179 MiB 1.3 TiB 1.1 TiB 1.1 TiB 67.46 1.11 66 up 33 ssd 3.49309 1.00000 3.5 TiB 2.1 TiB 179 MiB 1.2 TiB 1015 GiB 1.3 TiB 61.40 1.01 68 up .... omission .... TOTAL 63 TiB 38 TiB 3.8 GiB 23 TiB 16 TiB 25 TiB 60.96 MIN/MAX VAR: 0.68/1.30 STDDEV: 9.98

2 1

MDS crashing with "Assertion `px != 0' failed"
by Simon Campion 15 Apr '25

15 Apr '25

We're trying to determine the root cause of a CephFS outage. We have three MDS ranks with active-standby. During the outage, several MDSs crashed. The timeline of the crashes was: 2025-04-13T14:19:45 mds.r-cephfs-hdd-f on node06.internal 2025-04-13T14:38:35 mds.r-cephfs-hdd-a on node02.internal 2025-04-13T14:38:37 mds.r-cephfs-hdd-b on node07.internal 2025-04-13T14:38:38 mds.r-cephfs-hdd-d on node05.internal 2025-04-13T14:48:52 mds.r-cephfs-hdd-e on node08.internal 2025-04-13T14:54:12 mds.r-cephfs-hdd-f on node06.internal At around 3pm, the file system recovered by itself. For all six crashes, the MDS logged this as the reason of the crash: ceph-mds: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.1/rpm/el9/BUILD/ceph-19.2.1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr<T>::operator->() const [with T = MDRequestImpl]: Assertion `px != 0' failed. For the earliest crash, here's the `ceph crash info` output: bash-5.1$ ceph crash info 2025-04-13T14:19:45.645607Z_7e6475e0-9a22-4e3f-a282-0ab02a7c972c { "backtrace": [ "/lib64/libc.so.6(+0x3e930) [0x7f4990b08930]", "/lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc]", "raise()", "abort()", "/lib64/libc.so.6(+0x2875b) [0x7f4990af275b]", "/lib64/libc.so.6(+0x375c6) [0x7f4990b015c6]", "ceph-mds(+0x1c2829) [0x559da9485829]", "ceph-mds(+0x3191a7) [0x559da95dc1a7]", "(MDSContext::complete(int)+0x5c) [0x559da971117c]", "(Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d]", "/lib64/libc.so.6(+0x8a292) [0x7f4990b54292]", "/lib64/libc.so.6(+0x10f300) [0x7f4990bd9300]" ], "ceph_version": "19.2.1", "crash_id": "2025-04-13T14:19:45.645607Z_7e6475e0-9a22-4e3f-a282-0ab02a7c972c", "entity_name": "mds.r-cephfs-hdd-f", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mds", "stack_sig": "8975f8e99bd02b53c8d37ce7cc9e85dc5d4898104a0949d0829819a753123f18", "timestamp": "2025-04-13T14:19:45.645607Z", "utsname_hostname": "node06.internal", "utsname_machine": "x86_64", "utsname_release": "6.6.83-flatcar", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Mon Mar 17 16:07:40 -00 2025" } And here's an excerpt of the logs, starting with the aforementioned assertion failure: ceph-mds: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.1/rpm/el9/BUILD/ceph-19.2.1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr<T>::operator->() const [with T = MDRequestImpl]: Assertion `px != 0' failed. *** Caught signal (Aborted) ** in thread 7f4984f30640 thread_name: ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable) 1: /lib64/libc.so.6(+0x3e930) [0x7f4990b08930] 2: /lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc] 3: raise() 4: abort() 5: /lib64/libc.so.6(+0x2875b) [0x7f4990af275b] 6: /lib64/libc.so.6(+0x375c6) [0x7f4990b015c6] 7: ceph-mds(+0x1c2829) [0x559da9485829] 8: ceph-mds(+0x3191a7) [0x559da95dc1a7] 9: (MDSContext::complete(int)+0x5c) [0x559da971117c] 10: (Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d] 11: /lib64/libc.so.6(+0x8a292) [0x7f4990b54292] 12: /lib64/libc.so.6(+0x10f300) [0x7f4990bd9300] debug 2025-04-13T14:19:45.645+0000 7f4984f30640 -1 *** Caught signal (Aborted) ** in thread 7f4984f30640 thread_name: ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable) 1: /lib64/libc.so.6(+0x3e930) [0x7f4990b08930] 2: /lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc] 3: raise() 4: abort() 5: /lib64/libc.so.6(+0x2875b) [0x7f4990af275b] 6: /lib64/libc.so.6(+0x375c6) [0x7f4990b015c6] 7: ceph-mds(+0x1c2829) [0x559da9485829] 8: ceph-mds(+0x3191a7) [0x559da95dc1a7] 9: (MDSContext::complete(int)+0x5c) [0x559da971117c] 10: (Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d] 11: /lib64/libc.so.6(+0x8a292) [0x7f4990b54292] 12: /lib64/libc.so.6(+0x10f300) [0x7f4990bd9300] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- debug -9999> 2025-04-13T14:17:22.696+0000 7f4986f34640 5 mds.2.log trim already expired LogSegment(79533796/0x1a5ea43a405 events=117) debug -9998> 2025-04-13T14:17:22.696+0000 7f4986f34640 5 mds.2.log trim already expired LogSegment(79533913/0x1a5ea8350c6 events=75) debug -9997> 2025-04-13T14:17:22.696+0000 7f4986f34640 5 mds.2.log trim already expired LogSegment(79533988/0x1a5eac5c3df events=95) ... Do you have ideas what could cause these crashes or how we could troubleshoot further? We're happy to provide more information if that'd help. Simon

2 1