ASSISTANCE REQUEST: OSDs Stability Issues Post-Upgrade
Dear Ceph Community, I hope this message finds you well. I am reaching out to seek assistance regarding a stability issue we have encountered after upgrading our Ceph cluster from version Pacific 16.2.3 to Quincy 17.2.8. Following the upgrade, we have observed that several of our Object Storage Daemons (OSDs) are experiencing erratic behavior. These OSDs frequently exhibit a "flapping" condition, where they unexpectedly go down and then come back up. This issue has predominantly affected the recently upgraded OSDs within the cluster. Upon reviewing the logs from the affected OSDs, we encountered the following messages: 2025-02-03T08:34:09.769+0000 7f0f11390780 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-02-03T08:38:22.920+0000 7feb9dd44780 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) In an attempt to resolve the issue, we executed the ceph-bluestore-tool fsck and repair commands. Although these commands executed successfully, they did not rectify the problem at hand. Additionally, we have captured the following crash information from the ceph logs: ceph crash info 2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039 { "assert_condition": "log.t.seq == log.seq_live", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/c eph-17.2.8/src/os/bluestore/BlueFS.cc", "assert_func": "uint64_t BlueFS::_log_advance_seq()", "assert_line": 3029, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ce ph-17.2.8/src/os/bluestore/BlueFS.cc: In function 'uint64_t BlueFS::_log_advance_seq()' thread 7ff983564640 time 2025-02-03T09:19:08.738781+0000\n/home/jenkins-build/build/workspace/ceph-bu ild/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/os/bluestore/BlueFS.cc: 3029: FAILED ceph_assert (log.t.seq == log.seq_live)\n", "assert_thread_name": "bstore_kv_sync", "backtrace": [ "/lib64/libc.so.6(+0x3e730) [0x7ff9930f5730]", "/lib64/libc.so.6(+0x8bbdc) [0x7ff993142bdc]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55882dfb7fdd]", "/usr/bin/ceph-osd(+0x36b13e) [0x55882dfb813e]", "/usr/bin/ceph-osd(+0x9cff3b) [0x55882e61cf3b]", "(BlueFS::_flush_and_sync_log_jump_D(unsigned long)+0x4e) [0x55882e6291ee]", "(BlueFS::_compact_log_async_LD_LNF_D()+0x59b) [0x55882e62e8fb]", "/usr/bin/ceph-osd(+0x9f2b15) [0x55882e63fb15]", "(BlueFS::fsync(BlueFS::FileWriter*)+0x1b9) [0x55882e631989]", "/usr/bin/ceph-osd(+0x9f4889) [0x55882e641889]", "/usr/bin/ceph-osd(+0xd74cd5) [0x55882e9c1cd5]", "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x483) [0x55882eade393]", "(rocksdb::WritableFileWriter::Sync(bool)+0x120) [0x55882eae0b60]", "(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x337) [0x55882ea00ab7]", "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb ::PreReleaseCallback*)+0x1935) [0x55882ea07675]", "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x35) [0x55882ea077c5]", "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x83) [0x55882e992593]", "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x99) [0x55882e992ee9]", "(BlueStore::_kv_sync_thread()+0xf64) [0x55882e578e24]", "/usr/bin/ceph-osd(+0x8afb81) [0x55882e4fcb81]", "/lib64/libc.so.6(+0x89e92) [0x7ff993140e92]", "/lib64/libc.so.6(+0x10ef20) [0x7ff9931c5f20]" ], "ceph_version": "17.2.8", "crash_id": "2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039", "entity_name": "osd.211", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-osd", "stack_sig": "ba90de24e2beba9c6a75249a4cce7c533987ca5127cfba5b835a3456174d6080", "timestamp": "2025-02-03T09:19:08.749233Z", "utsname_hostname": "afra-osd18", "utsname_machine": "x86_64", "utsname_release": "5.15.0-119-generic", "utsname_sysname": "Linux", "utsname_version": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024" } The above crash logs highlight an assertion failure in the BlueFS component, specifically within the function BlueFS::_log_advance_seq(). Despite our efforts to analyze and resolve the issue, we have reached an impasse. For completeness, we have verified the health of our disks using smartctl, and they have all been deemed healthy. We kindly request guidance from the community on how to address this issue or any recommended steps for deeper diagnostics. We appreciate your support and expertise during this troubleshooting process. OSD logs: https://paste.mozilla.org/6STm6eum Thank you for your attention and assistance. Best regards Nima
The question was posted here [0] as well. There is a tracker [1] with a fix [2] which will be backported to Reef, but Quincy is EOL. [0] https://serverfault.com/questions/1172161/osds-stability-issues-post-upgrade... [1] https://tracker.ceph.com/issues/69764 [2] https://github.com/ceph/ceph/pull/53732 Zitat von Nima AbolhassanBeigi <nima.abolhassanbeigi@gmail.com>:
Dear Ceph Community, I hope this message finds you well. I am reaching out to seek assistance regarding a stability issue we have encountered after upgrading our Ceph cluster from version Pacific 16.2.3 to Quincy 17.2.8.
Following the upgrade, we have observed that several of our Object Storage Daemons (OSDs) are experiencing erratic behavior. These OSDs frequently exhibit a "flapping" condition, where they unexpectedly go down and then come back up. This issue has predominantly affected the recently upgraded OSDs within the cluster.
Upon reviewing the logs from the affected OSDs, we encountered the following messages:
2025-02-03T08:34:09.769+0000 7f0f11390780 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-02-03T08:38:22.920+0000 7feb9dd44780 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file)
In an attempt to resolve the issue, we executed the ceph-bluestore-tool fsck and repair commands. Although these commands executed successfully, they did not rectify the problem at hand.
Additionally, we have captured the following crash information from the ceph logs: ceph crash info 2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039 { "assert_condition": "log.t.seq == log.seq_live", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/c eph-17.2.8/src/os/bluestore/BlueFS.cc", "assert_func": "uint64_t BlueFS::_log_advance_seq()", "assert_line": 3029, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ce ph-17.2.8/src/os/bluestore/BlueFS.cc: In function 'uint64_t BlueFS::_log_advance_seq()' thread 7ff983564640 time 2025-02-03T09:19:08.738781+0000\n/home/jenkins-build/build/workspace/ceph-bu ild/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/os/bluestore/BlueFS.cc: 3029: FAILED ceph_assert (log.t.seq == log.seq_live)\n", "assert_thread_name": "bstore_kv_sync", "backtrace": [ "/lib64/libc.so.6(+0x3e730) [0x7ff9930f5730]", "/lib64/libc.so.6(+0x8bbdc) [0x7ff993142bdc]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55882dfb7fdd]", "/usr/bin/ceph-osd(+0x36b13e) [0x55882dfb813e]", "/usr/bin/ceph-osd(+0x9cff3b) [0x55882e61cf3b]", "(BlueFS::_flush_and_sync_log_jump_D(unsigned long)+0x4e) [0x55882e6291ee]", "(BlueFS::_compact_log_async_LD_LNF_D()+0x59b) [0x55882e62e8fb]", "/usr/bin/ceph-osd(+0x9f2b15) [0x55882e63fb15]", "(BlueFS::fsync(BlueFS::FileWriter*)+0x1b9) [0x55882e631989]", "/usr/bin/ceph-osd(+0x9f4889) [0x55882e641889]", "/usr/bin/ceph-osd(+0xd74cd5) [0x55882e9c1cd5]", "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x483) [0x55882eade393]", "(rocksdb::WritableFileWriter::Sync(bool)+0x120) [0x55882eae0b60]",
"(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x337) [0x55882ea00ab7]", "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb ::PreReleaseCallback*)+0x1935) [0x55882ea07675]", "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x35) [0x55882ea077c5]", "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x83) [0x55882e992593]",
"(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x99) [0x55882e992ee9]", "(BlueStore::_kv_sync_thread()+0xf64) [0x55882e578e24]", "/usr/bin/ceph-osd(+0x8afb81) [0x55882e4fcb81]", "/lib64/libc.so.6(+0x89e92) [0x7ff993140e92]", "/lib64/libc.so.6(+0x10ef20) [0x7ff9931c5f20]" ], "ceph_version": "17.2.8", "crash_id": "2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039", "entity_name": "osd.211", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-osd", "stack_sig": "ba90de24e2beba9c6a75249a4cce7c533987ca5127cfba5b835a3456174d6080", "timestamp": "2025-02-03T09:19:08.749233Z", "utsname_hostname": "afra-osd18", "utsname_machine": "x86_64", "utsname_release": "5.15.0-119-generic", "utsname_sysname": "Linux", "utsname_version": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024" }
The above crash logs highlight an assertion failure in the BlueFS component, specifically within the function BlueFS::_log_advance_seq(). Despite our efforts to analyze and resolve the issue, we have reached an impasse.
For completeness, we have verified the health of our disks using smartctl, and they have all been deemed healthy.
We kindly request guidance from the community on how to address this issue or any recommended steps for deeper diagnostics. We appreciate your support and expertise during this troubleshooting process.
OSD logs: https://paste.mozilla.org/6STm6eum
Thank you for your attention and assistance.
Best regards Nima _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We didn't want to upgrade to reef, so we ended up downgrading to v17.2.7 and solved the problem
participants (2)
-
Eugen Block
-
Nima AbolhassanBeigi