This looks at least similar to me, but I'not sure if it's actually related: https://tracker.ceph.com/issues/70624 Zitat von Simon Campion <simon.campion@deepl.com>:
We're trying to determine the root cause of a CephFS outage. We have three MDS ranks with active-standby.
During the outage, several MDSs crashed. The timeline of the crashes was:
2025-04-13T14:19:45 mds.r-cephfs-hdd-f on node06.internal 2025-04-13T14:38:35 mds.r-cephfs-hdd-a on node02.internal 2025-04-13T14:38:37 mds.r-cephfs-hdd-b on node07.internal 2025-04-13T14:38:38 mds.r-cephfs-hdd-d on node05.internal 2025-04-13T14:48:52 mds.r-cephfs-hdd-e on node08.internal 2025-04-13T14:54:12 mds.r-cephfs-hdd-f on node06.internal
At around 3pm, the file system recovered by itself.
For all six crashes, the MDS logged this as the reason of the crash:
ceph-mds: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.1/rpm/el9/BUILD/ceph-19.2.1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr<T>::operator->() const [with T = MDRequestImpl]: Assertion `px != 0' failed.
For the earliest crash, here's the `ceph crash info` output:
bash-5.1$ ceph crash info 2025-04-13T14:19:45.645607Z_7e6475e0-9a22-4e3f-a282-0ab02a7c972c { "backtrace": [ "/lib64/libc.so.6(+0x3e930) [0x7f4990b08930]", "/lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc]", "raise()", "abort()", "/lib64/libc.so.6(+0x2875b) [0x7f4990af275b]", "/lib64/libc.so.6(+0x375c6) [0x7f4990b015c6]", "ceph-mds(+0x1c2829) [0x559da9485829]", "ceph-mds(+0x3191a7) [0x559da95dc1a7]", "(MDSContext::complete(int)+0x5c) [0x559da971117c]", "(Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d]", "/lib64/libc.so.6(+0x8a292) [0x7f4990b54292]", "/lib64/libc.so.6(+0x10f300) [0x7f4990bd9300]" ], "ceph_version": "19.2.1", "crash_id": "2025-04-13T14:19:45.645607Z_7e6475e0-9a22-4e3f-a282-0ab02a7c972c", "entity_name": "mds.r-cephfs-hdd-f", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mds", "stack_sig": "8975f8e99bd02b53c8d37ce7cc9e85dc5d4898104a0949d0829819a753123f18", "timestamp": "2025-04-13T14:19:45.645607Z", "utsname_hostname": "node06.internal", "utsname_machine": "x86_64", "utsname_release": "6.6.83-flatcar", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Mon Mar 17 16:07:40 -00 2025" }
And here's an excerpt of the logs, starting with the aforementioned assertion failure:
ceph-mds: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.1/rpm/el9/BUILD/ceph-19.2.1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr<T>::operator->() const [with T = MDRequestImpl]: Assertion `px != 0' failed. *** Caught signal (Aborted) ** in thread 7f4984f30640 thread_name: ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable) 1: /lib64/libc.so.6(+0x3e930) [0x7f4990b08930] 2: /lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc] 3: raise() 4: abort() 5: /lib64/libc.so.6(+0x2875b) [0x7f4990af275b] 6: /lib64/libc.so.6(+0x375c6) [0x7f4990b015c6] 7: ceph-mds(+0x1c2829) [0x559da9485829] 8: ceph-mds(+0x3191a7) [0x559da95dc1a7] 9: (MDSContext::complete(int)+0x5c) [0x559da971117c] 10: (Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d] 11: /lib64/libc.so.6(+0x8a292) [0x7f4990b54292] 12: /lib64/libc.so.6(+0x10f300) [0x7f4990bd9300] debug 2025-04-13T14:19:45.645+0000 7f4984f30640 -1 *** Caught signal (Aborted) ** in thread 7f4984f30640 thread_name: ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable) 1: /lib64/libc.so.6(+0x3e930) [0x7f4990b08930] 2: /lib64/libc.so.6(+0x8bfdc) [0x7f4990b55fdc] 3: raise() 4: abort() 5: /lib64/libc.so.6(+0x2875b) [0x7f4990af275b] 6: /lib64/libc.so.6(+0x375c6) [0x7f4990b015c6] 7: ceph-mds(+0x1c2829) [0x559da9485829] 8: ceph-mds(+0x3191a7) [0x559da95dc1a7] 9: (MDSContext::complete(int)+0x5c) [0x559da971117c] 10: (Finisher::finisher_thread_entry()+0x17d) [0x7f499127a85d] 11: /lib64/libc.so.6(+0x8a292) [0x7f4990b54292] 12: /lib64/libc.so.6(+0x10f300) [0x7f4990bd9300] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- debug -9999> 2025-04-13T14:17:22.696+0000 7f4986f34640 5 mds.2.log trim already expired LogSegment(79533796/0x1a5ea43a405 events=117) debug -9998> 2025-04-13T14:17:22.696+0000 7f4986f34640 5 mds.2.log trim already expired LogSegment(79533913/0x1a5ea8350c6 events=75) debug -9997> 2025-04-13T14:17:22.696+0000 7f4986f34640 5 mds.2.log trim already expired LogSegment(79533988/0x1a5eac5c3df events=95) ...
Do you have ideas what could cause these crashes or how we could troubleshoot further? We're happy to provide more information if that'd help.
Simon _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io