Hi Eugen, that's a know issue - crash on OSD termination. Fixed in v18.2.5, see https://tracker.ceph.com/issues/66148 And your backtrace explicitly states that happened in 18.2.4:
"ceph_version": "18.2.4",
Thanks, Igor On 16.04.2025 12:07, Eugen Block wrote:
Hi,
I'm currently upgrading the first customer cluster to latest Reef (18.2.5). I haven't seen that in my lab clusters yet, but it might be due to insufficient crash configuration (or the shutdown_timeout is not triggered). I found this backport for 18.2.5:
https://tracker.ceph.com/issues/63527
Am I interpreting correctly that the assert is triggered because the OSDs were shut down on 18.2.4, but this won't happen anymore on 18.2.5?
Thanks! Eugen
---snip--- root@storage01:~# ceph crash info 2025-04-16T08:51:38.757577Z_bd34dfdd-183a-4274-8a40-a6cb3311ca56 { "archived": "2025-04-16 08:52:50.374393", "assert_condition": "end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/osd/OSD.cc", "assert_func": "int OSD::shutdown()", "assert_line": 4585, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/osd/OSD.cc: In function 'int OSD::shutdown()' thread 7f313de0a640 time 2025-04-16T08:51:38.751362+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/osd/OSD.cc: 4585: FAILED ceph_assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)\n", "assert_thread_name": "signal_handler", "backtrace": [ "/lib64/libc.so.6(+0x3e6f0) [0x7f31412d16f0]", "/lib64/libc.so.6(+0x8b94c) [0x7f314131e94c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56015bef62b2]", "/usr/bin/ceph-osd(+0x3c9413) [0x56015bef6413]", "/usr/bin/ceph-osd(+0x3591d7) [0x56015be861d7]", "/usr/bin/ceph-osd(+0xa1da57) [0x56015c54aa57]", "/lib64/libc.so.6(+0x89c02) [0x7f314131cc02]", "/lib64/libc.so.6(+0x10ec40) [0x7f31413a1c40]" ], "ceph_version": "18.2.4", "crash_id": "2025-04-16T08:51:38.757577Z_bd34dfdd-183a-4274-8a40-a6cb3311ca56", "entity_name": "osd.162", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-osd", "stack_sig": "434a67dd4e3c173992edcf6eb5dc6014d029ca333c4904eced28762dc1959db1", "timestamp": "2025-04-16T08:51:38.757577Z", "utsname_hostname": "storage04", "utsname_machine": "x86_64", "utsname_release": "5.15.0-130-generic", "utsname_sysname": "Linux", "utsname_version": "#140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024" } ---snip--- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io