Describe your hardware, please, and are you talking an orderly "shutdown -r" reboot, or a kernel / system crash or power loss? Often corruptions like this are a result of: * Using non-enterprise SSDs that lack power loss protection * Buggy / defective RAID HBAs * Enabling volatile write cache on drives
On Aug 5, 2024, at 4:54 AM, Reza Bakhshayeshi <reza.b2008@gmail.com> wrote:
Hello,
Whenever a node reboots in the cluster I get some corrupted OSDs, is there any config I should set to prevent this from happening that I am not aware of?
Here is the error log:
# kubectl logs rook-ceph-osd-1-5dcbd99cc7-2l5g2 -c expand-bluefs
ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f969977ce15] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f969977cfdb] 3: (BlueStore::expand_devices(std::ostream&)+0x5ff) [0x55ce89d1f3ff] 4: main() 5: __libc_start_main() 6: _start()
0> 2024-07-31T08:39:19.840+0000 7f969b1c0980 -1 *** Caught signal (Aborted) ** in thread 7f969b1c0980 thread_name:ceph-bluestore-
ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12d20) [0x7f969843fd20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f969977ce6f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f969977cfdb] 6: (BlueStore::expand_devices(std::ostream&)+0x5ff) [0x55ce89d1f3ff] 7: main() 8: __libc_start_main() 9: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io