[ceph-users] Re: 回复: Re: OSDs continuously restarting under load

Hi, Igor, thanks for the tip. Dmesg does not say any suspicious information. I will investigate whether hardware has any problem or not. best regards, samuel huxiaoyu(a)horebdata.cn 发件人： Igor Fedotov 发送时间： 2020-03-19 12:07 收件人： huxiaoyu(a)horebdata.cn; ceph-users; ceph-users 主题： Re: [ceph-users] OSDs continuously restarting under load Hi, Samuel, I've never seen that sort of signal in the real life: 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** I suppose this has some hardware roots. Have you checked dmesg output? Just in case, here is some info on "Bus Error" signal, may be it will provide some insight: https://en.wikipedia.org/wiki/Bus_error Thanks, Igor On 3/18/2020 5:06 PM, huxiaoyu(a)horebdata.cn wrote: reweight of newly-added OSD on the new node exceed 0.4 somewhere, the osd becomes unresponsive and restarting, eventually go down. ------------------------------------------------------------------------------------ 日志 opened 0 pgs pgs opened 202 pgs pgs opened 177 pgs pgs opened 170 pgs need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer ** luminous (stable) std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] [0x20001ecea74] ThreadPool::TPHandle&)+0x814) [0x20001de1384] boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] const&)+0xb8) [0x20001f98968] ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] [0x200024d60a4] [0x200024da278] needed to interpret this. register_command perfcounters_dump hook 0x2000ce09e40 register_command 1 hook 0x2000ce09e40 register_command perf dump hook 0x2000ce09e40 register_command perfcounters_schema hook 0x2000ce09e40 register_command perf histogram dump hook 0x2000ce09e40 register_command 2 hook 0x2000ce09e40 register_command perf schema hook 0x2000ce09e40 register_command perf histogram schema hook 0x2000ce09e40 register_command perf reset hook 0x2000ce09e40 register_command config show hook 0x2000ce09e40 register_command config help hook 0x2000ce09e40 register_command config set hook 0x2000ce09e40 register_command config get hook 0x2000ce09e40 register_command config diff hook 0x2000ce09e40 register_command config diff get hook 0x2000ce09e40 register_command log flush hook 0x2000ce09e40 register_command log dump hook 0x2000ce09e40 register_command log reopen hook 0x2000ce09e40 register_command dump_mempools hook 0x2000ce0d038 following dangerous and experimental features are enabled: bluestore,rocksdb following dangerous and experimental features are enabled: bluestore,rocksdb (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process ceph-osd, pid 1159687 empty --pid-file following dangerous and experimental features are enabled: bluestore,rocksdb lrc /var/lib/ceph/osd/ceph-6/block type kernel /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational /var/lib/ceph/osd/ceph-6/block) close /var/lib/ceph/osd/ceph-6/block type kernel /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational /var/lib/ceph/osd/ceph-6/block.db type kernel /var/lib/ceph/osd/ceph-6/block.db) open path /var/lib/ceph/osd/ceph-6/block.db /var/lib/ceph/osd/ceph-6/block.db) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway /var/lib/ceph/osd/ceph-6/block.db) open size 39998980096 (0x950200000, 38146 MB) block_size 4096 (4096 B) rotational /var/lib/ceph/osd/ceph-6/block type kernel /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational compaction_readahead_size = 2MB compaction_style = kCompactionStyleLevel compaction_threads = 32 compression = kNoCompression flusher_threads = 8 level0_file_num_compaction_trigger = 64 level0_slowdown_writes_trigger = 128 level0_stop_writes_trigger = 256 max_background_compactions = 64 max_bytes_for_level_base = 2GB max_write_buffer_number = 64 min_write_buffer_number_to_merge = 32 recycle_log_file_num = 64 target_file_size_base = 4MB write_buffer_size = 4MB compaction_readahead_size = 2MB compaction_style = kCompactionStyleLevel compaction_threads = 32 compression = kNoCompression flusher_threads = 8 level0_file_num_compaction_trigger = 64 level0_slowdown_writes_trigger = 128 level0_stop_writes_trigger = 256 max_background_compactions = 64 max_bytes_for_level_base = 2GB max_write_buffer_number = 64 min_write_buffer_number_to_merge = 32 recycle_log_file_num = 64 target_file_size_base = 4MB write_buffer_size = 4MB /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: loading cls_hello permitted to load lua permitted to load sdk permitted to load kvs /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs has features 432629239337189376, adjusting msgr requires for clients has features 432629239337189376 was 8705, adjusting msgr requires for mons has features 1009089991640629248, adjusting msgr requires for osds opened 170 pgs weightedpriority op queue with priority op cut off at 196. log_to_monitors {default=true} init, starting boot process Got map version 34 Active mgr is now 192.168.230.120:6808/44007 Starting new session with 192.168.230.120:6808/44007 handle_mgr_configure stats_period=5 handle_mgr_configure updated stats threshold: 5 192.168.240.122:6806/1159687 >> 192.168.240.121:6802/33592 conn(0x20014018040 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 192.168.240.122:6806/1159687 >> 192.168.240.121:6800/33864 conn(0x2001403ee60 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 192.168.240.122:6806/1159687 >> 192.168.240.120:6804/13410 conn(0x20014079940 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 192.168.240.122:6806/1159687 >> 192.168.240.120:6806/13743 conn(0x20014083980 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 192.168.240.122:6806/1159687 >> 192.168.240.121:6806/12423 conn(0x2001407e030 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 192.168.240.122:6806/1159687 >> 192.168.240.121:6804/11599 conn(0x2001408c660 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 192.168.240.122:6806/1159687 >> 192.168.240.120:6800/12205 conn(0x20014087ff0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 192.168.240.122:6806/1159687 >> 192.168.240.120:6802/12991 conn(0x2001409d300 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 192.168.240.122:6806/1159687 >> 192.168.240.122:6804/1159466 conn(0x200140b3420 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 192.168.230.122:6806/1159687 >> 192.168.230.202:0/3091162658 conn(0x200140cd900 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4289863819 conn(0x200140d8500 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer unexpected need for 4:b442b93f:::rbd_data.911772ae8944a.0000000000002aa7:head have 3097'452 flags = none tried to add 3097'452 flags = none unexpected need for 4:b445877b:::rbd_data.2fa7e6b8b4567.000000000000002d:head have 1915'212 flags = none tried to add 1915'212 flags = none unexpected need for 4:b44674a4:::rbd_data.110706b8b4567.0000000000000659:head have 1915'213 flags = none tried to add 1915'213 flags = none unexpected need for 4:b44757d1:::rbd_data.cd282238e1f29.0000000000009ea2:head have 5165'734 flags = none tried to add 5165'734 flags = none unexpected need for 4:b4490029:::rbd_data.8d8146b8b4567.00000000000080a0:head have 2855'272 flags = none tried to add 2855'272 flags = none unexpected need for 4:b44950ae:::rbd_data.5f15f625558ec.0000000000009fd2:head have 1915'214 flags = none tried to add 1915'214 flags = none unexpected need for 4:b449ce0d:::rbd_data.9117a327b23c6.00000000000074a6:head have 3798'560 flags = none tried to add 3798'560 flags = none unexpected need for 4:b44a69ad:::rbd_data.8b4a76b8b4567.000000000000017a:head have 2197'242 flags = none tried to add 2197'242 flags = none unexpected need for 4:b44bbb34:::rbd_data.8922074b0dc51.00000000000098a5:head have 3099'543 flags = delete tried to add 3099'543 flags = delete unexpected need for 4:b44be196:::rbd_data.cd8e219495cff.00000000000192a0:head have 5169'1101 flags = delete tried to add 5169'1101 flags = delete unexpected need for 4:b44cd1c7:::rbd_data.5f15f625558ec.000000000000820b:head have 1915'215 flags = none tried to add 1915'215 flags = none unexpected need for 4:b44e04c0:::rbd_data.cd66c46e87ccd.0000000000015aa4:head have 5175'1280 flags = none tried to add 5175'1280 flags = none unexpected need for 4:b44fa767:::rbd_data.ccf5e625558ec.00000000000150a0:head have 5166'844 flags = delete tried to add 5166'844 flags = delete unexpected need for 4:b4518cb0:::rbd_data.589572ae8944a.0000000000000433:head have 1805'174 flags = delete tried to add 1805'174 flags = delete unexpected need for 4:b4539e4a:::rbd_data.5f15f625558ec.000000000000031a:head have 1915'216 flags = none tried to add 1915'216 flags = none unexpected need for 4:b4540dfd:::rbd_data.cd66c46e87ccd.000000000001d6a6:head have 5175'1283 flags = none tried to add 5175'1283 flags = none unexpected need for 4:b45474ee:::rbd_data.2fa7e6b8b4567.000000000000003a:head have 1915'217 flags = none tried to add 1915'217 flags = none unexpected need for 4:b45506f1:::rbd_data.9177446e87ccd.00000000000036c3:head have 5175'1284 flags = none tried to add 5175'1284 flags = none unexpected need for 4:b4576e29:::rbd_data.589572ae8944a.0000000000001207:head have 1805'179 flags = delete tried to add 1805'179 flags = delete unexpected need for 4:b4578057:::rbd_data.ccf5e625558ec.00000000000096a5:head have 5166'785 flags = delete tried to add 5166'785 flags = delete unexpected need for 4:b457fe94:::rbd_data.589572ae8944a.0000000000004e27:head have 1805'200 flags = delete tried to add 1805'200 flags = delete unexpected need for 4:b4592563:::rbd_data.9117a327b23c6.000000000000b2a0:head have 3894'562 flags = none tried to add 3894'562 flags = none unexpected need for 4:b45bb3ff:::rbd_data.c6c99507ed7ab.000000000000aea0:head have 5172'1141 flags = none tried to add 5172'1141 flags = none unexpected need for 4:b45d7c66:::rbd_data.110706b8b4567.0000000000000e07:head have 1915'218 flags = none tried to add 1915'218 flags = none unexpected need for 4:b45e1a62:::rbd_data.589572ae8944a.0000000000004733:head have 1805'198 flags = delete tried to add 1805'198 flags = delete unexpected need for 4:b45f03a3:::rbd_data.914c22ae8944a.00000000000058a8:head have 3908'593 flags = delete tried to add 3908'593 flags = delete unexpected need for 4:b4604fdd:::rbd_data.c6c99507ed7ab.00000000000004ca:head have 5176'1523 flags = none tried to add 5176'1523 flags = none unexpected need for 4:b4636d5b:::rbd_data.cd282238e1f29.000000000000b6a1:head have 5165'735 flags = none tried to add 5165'735 flags = none unexpected need for 4:b463cd0a:::rbd_data.9117a327b23c6.000000000000c0a8:head have 3908'563 flags = none tried to add 3908'563 flags = none unexpected need for 4:b468ca37:::rbd_data.9177446e87ccd.000000000000040a:head have 4427'730 flags = none tried to add 4427'730 flags = none unexpected need for 4:b4696438:::rbd_data.9177446e87ccd.0000000000000c53:head have 4743'732 flags = none tried to add 4743'732 flags = none unexpected need for 4:b46be061:::rbd_data.c70aa2eb141f2.0000000000001ea6:head have 5175'1287 flags = none tried to add 5175'1287 flags = none unexpected need for 4:b46bfc4e:::rbd_data.cda772ae8944a.00000000000020a4:head have 5172'1144 flags = none tried to add 5172'1144 flags = none unexpected need for 4:b46e7efd:::rbd_data.110706b8b4567.0000000000001c01:head have 1915'219 flags = none tried to add 1915'219 flags = none unexpected need for 4:b46ece7b:::rbd_data.8ff1766334873.00000000000000ae:head have 3092'344 flags = delete tried to add 3092'344 flags = delete unexpected need for 4:b4786571:::rbd_data.8f6b3643c9869.00000000000040a0:head have 3092'381 flags = delete tried to add 3092'381 flags = delete unexpected need for 4:b47888b8:::rbd_data.589572ae8944a.0000000000004534:head have 1805'197 flags = delete tried to add 1805'197 flags = delete unexpected need for 4:b478b833:::rbd_data.589572ae8944a.00000000000048ff:head have 1805'199 flags = delete tried to add 1805'199 flags = delete unexpected need for 4:b47ec645:::rbd_data.8b4a76b8b4567.00000000000001d2:head have 2197'271 flags = none tried to add 2197'271 flags = none unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer (Bus error) ** luminous (stable) std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] [0x20001ecea74] ThreadPool::TPHandle&)+0x814) [0x20001de1384] boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] const&)+0xb8) [0x20001f98968] ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] [0x200024d60a4] [0x200024da278] needed to interpret this. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io