[ceph-users] MDS crashing with "Assertion `px != 0' failed"

15 Apr 2025

      We're trying to determine the root cause of a CephFS outage. We have three
MDS ranks with active-standby.

During the outage, several MDSs crashed. The timeline of the crashes was:

2025-04-13T14:19:45 mds.r-cephfs-hdd-f on node06.internal
2025-04-13T14:38:35 mds.r-cephfs-hdd-a on node02.internal
2025-04-13T14:38:37 mds.r-cephfs-hdd-b on node07.internal
2025-04-13T14:38:38 mds.r-cephfs-hdd-d on node05.internal
2025-04-13T14:48:52 mds.r-cephfs-hdd-e on node08.internal
2025-04-13T14:54:12 mds.r-cephfs-hdd-f on node06.internal

At around 3pm, the file system recovered by itself.

For all six crashes, the MDS logged this as the reason of the crash:

ceph-mds:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.1/rpm/el9/BUILD/ceph-19.2.1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201:
T* boost::intrusive_ptr<T>::operator->() const [with T = MDRequestImpl]:
Assertion `px != 0' failed.

For the earliest crash, here's the `ceph crash info` output:

bash-5.1$ ceph crash info
2025-04-13T14:19:45.645607Z_7e6475e0-9a22-4e3f-a282-0ab02a7c972c
{
    "backtrace": [
        "/lib64/libc.so.6(+0x3e930) [0x7f4990b08930]",
        "/lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc]",
        "raise()",
        "abort()",
        "/lib64/libc.so.6(+0x2875b) [0x7f4990af275b]",
        "/lib64/libc.so.6(+0x375c6) [0x7f4990b015c6]",
        "ceph-mds(+0x1c2829) [0x559da9485829]",
        "ceph-mds(+0x3191a7) [0x559da95dc1a7]",
        "(MDSContext::complete(int)+0x5c) [0x559da971117c]",
        "(Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d]",
        "/lib64/libc.so.6(+0x8a292) [0x7f4990b54292]",
        "/lib64/libc.so.6(+0x10f300) [0x7f4990bd9300]"
    ],
    "ceph_version": "19.2.1",
    "crash_id":
"2025-04-13T14:19:45.645607Z_7e6475e0-9a22-4e3f-a282-0ab02a7c972c",
    "entity_name": "mds.r-cephfs-hdd-f",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "9",
    "os_version_id": "9",
    "process_name": "ceph-mds",
    "stack_sig":
"8975f8e99bd02b53c8d37ce7cc9e85dc5d4898104a0949d0829819a753123f18",
    "timestamp": "2025-04-13T14:19:45.645607Z",
    "utsname_hostname": "node06.internal",
    "utsname_machine": "x86_64",
    "utsname_release": "6.6.83-flatcar",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Mon Mar 17 16:07:40 -00 2025"
}

And here's an excerpt of the logs, starting with the aforementioned
assertion failure:

ceph-mds:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.1/rpm/el9/BUILD/ceph-19.2.1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201:
T* boost::intrusive_ptr<T>::operator->() const [with T = MDRequestImpl]:
Assertion `px != 0' failed.
*** Caught signal (Aborted) **
 in thread 7f4984f30640 thread_name:
 ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid
(stable)
 1: /lib64/libc.so.6(+0x3e930) [0x7f4990b08930]
 2: /lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc]
 3: raise()
 4: abort()
 5: /lib64/libc.so.6(+0x2875b) [0x7f4990af275b]
 6: /lib64/libc.so.6(+0x375c6) [0x7f4990b015c6]
 7: ceph-mds(+0x1c2829) [0x559da9485829]
 8: ceph-mds(+0x3191a7) [0x559da95dc1a7]
 9: (MDSContext::complete(int)+0x5c) [0x559da971117c]
 10: (Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d]
 11: /lib64/libc.so.6(+0x8a292) [0x7f4990b54292]
 12: /lib64/libc.so.6(+0x10f300) [0x7f4990bd9300]
debug 2025-04-13T14:19:45.645+0000 7f4984f30640 -1 *** Caught signal
(Aborted) **
 in thread 7f4984f30640 thread_name:
 ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid
(stable)
 1: /lib64/libc.so.6(+0x3e930) [0x7f4990b08930]
 2: /lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc]
 3: raise()
 4: abort()
 5: /lib64/libc.so.6(+0x2875b) [0x7f4990af275b]
 6: /lib64/libc.so.6(+0x375c6) [0x7f4990b015c6]
 7: ceph-mds(+0x1c2829) [0x559da9485829]
 8: ceph-mds(+0x3191a7) [0x559da95dc1a7]
 9: (MDSContext::complete(int)+0x5c) [0x559da971117c]
 10: (Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d]
 11: /lib64/libc.so.6(+0x8a292) [0x7f4990b54292]
 12: /lib64/libc.so.6(+0x10f300) [0x7f4990bd9300]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
--- begin dump of recent events ---
debug  -9999> 2025-04-13T14:17:22.696+0000 7f4986f34640  5 mds.2.log trim
already expired LogSegment(79533796/0x1a5ea43a405 events=117)
debug  -9998> 2025-04-13T14:17:22.696+0000 7f4986f34640  5 mds.2.log trim
already expired LogSegment(79533913/0x1a5ea8350c6 events=75)
debug  -9997> 2025-04-13T14:17:22.696+0000 7f4986f34640  5 mds.2.log trim
already expired LogSegment(79533988/0x1a5eac5c3df events=95)
...

Do you have ideas what could cause these crashes or how we could
troubleshoot further? We're happy to provide more information if that'd
help.

Simon

[ceph-users] MDS crashing with "Assertion `px != 0' failed"

Simon Campion