Hi, please if someone know how to help, I have an HDD pool in mycluster and after rebooting one server, my osds has started to crash. This pool is a backup pool and have OSD as failure domain with an size of 2. After rebooting one server, My osds started to crash, and the thing is only getting worse. I have then tried to run ceph-bluestore-tool repair and I receive what I think is the same error that shows on the osd logs: [root@cwvh13 ~]# ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-81 --log-level 10 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc: In function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f6467ffcec0 time 2021-03-11 12:13:12.121766 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc: 53: FAILED ceph_assert(r == 0) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f645e1a7b27] 2: (()+0x25ccef) [0x7f645e1a7cef] 3: (()+0x3cd57f) [0x5642e85c457f] 4: (HybridAllocator::~HybridAllocator()+0x17) [0x5642e85f3f37] 5: (BlueStore::_close_alloc()+0x42) [0x5642e84379d2] 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x5642e84bbac8] 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x293) [0x5642e84bbf13] 8: (main()+0x13cc) [0x5642e83caaec] 9: (__libc_start_main()+0xf5) [0x7f645ae24555] 10: (()+0x1fae9f) [0x5642e83f1e9f]
Hi Cassiano, the backtrace you've provided relates to the bug fixed by: https://github.com/ceph/ceph/pull/37793 This fix is going to be releases with the upcoming v14.2.17. But I doubt that your original crashes have the same root cause - this issue appears during shutdown only. Anyway you can work around it by using different allocator: avl or bitmap. Thanks, Igor On 3/11/2021 6:21 PM, Cassiano Pilipavicius wrote:
Hi, please if someone know how to help, I have an HDD pool in mycluster and after rebooting one server, my osds has started to crash.
This pool is a backup pool and have OSD as failure domain with an size of 2.
After rebooting one server, My osds started to crash, and the thing is only getting worse. I have then tried to run ceph-bluestore-tool repair and I receive what I think is the same error that shows on the osd logs:
[root@cwvh13 ~]# ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-81 --log-level 10 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc: In function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f6467ffcec0 time 2021-03-11 12:13:12.121766 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc: 53: FAILED ceph_assert(r == 0) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f645e1a7b27] 2: (()+0x25ccef) [0x7f645e1a7cef] 3: (()+0x3cd57f) [0x5642e85c457f] 4: (HybridAllocator::~HybridAllocator()+0x17) [0x5642e85f3f37] 5: (BlueStore::_close_alloc()+0x42) [0x5642e84379d2] 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x5642e84bbac8] 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x293) [0x5642e84bbf13] 8: (main()+0x13cc) [0x5642e83caaec] 9: (__libc_start_main()+0xf5) [0x7f645ae24555] 10: (()+0x1fae9f) [0x5642e83f1e9f] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, really this error was only showing up when I've tried to run ceph-bluestore-tool repair, In my 3 OSDs that keeps crashing, it show the following log... please let me know if there is something I can do to get the pool back to a functioning state. Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 21.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** -32> 2021-03-11 14:25:55.812 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 445022208 unmapped: 59588608 heap: 504610816 old mem: 134217728 new mem: 2564495564 -31> 2021-03-11 14:25:55.813 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 445210624 unmapped: 59400192 heap: 504610816 old mem: 2564495564 new mem: 2816296009 -30> 2021-03-11 14:25:55.813 7f50161b7700 5 bluestore.MempoolThread(0x558aa9a8ea98) _trim_shards cache_size: 2816296009 kv_alloc: 956301312 kv_used: 6321600 meta_alloc: 956301312 meta_used: 11680 data_alloc: 620756992 data_used: 151552 -29> 2021-03-11 14:25:55.838 7f502962ba80 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -28> 2021-03-11 14:25:55.839 7f502962ba80 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/cls/hello/cls_hello.cc:296: loading cls_hello -27> 2021-03-11 14:25:55.840 7f502962ba80 0 _get_class not permitted to load kvs -26> 2021-03-11 14:25:55.840 7f502962ba80 0 _get_class not permitted to load lua -25> 2021-03-11 14:25:55.852 7f502962ba80 0 _get_class not permitted to load sdk -24> 2021-03-11 14:25:55.853 7f502962ba80 0 osd.80 697960 crush map has features 283675107524608, adjusting msgr requires for clients -23> 2021-03-11 14:25:55.853 7f502962ba80 0 osd.80 697960 crush map has features 283675107524608 was 8705, adjusting msgr requires for mons -22> 2021-03-11 14:25:55.853 7f502962ba80 0 osd.80 697960 crush map has features 3026702624700514304, adjusting msgr requires for osds -21> 2021-03-11 14:25:56.814 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 501989376 unmapped: 21192704 heap: 523182080 old mem: 2816296009 new mem: 2842012351 -20> 2021-03-11 14:25:57.816 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 542867456 unmapped: 19210240 heap: 562077696 old mem: 2842012351 new mem: 2844985645 -19> 2021-03-11 14:25:58.327 7f502962ba80 0 osd.80 697960 load_pgs -18> 2021-03-11 14:25:58.818 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 596754432 unmapped: 229376 heap: 596983808 old mem: 2844985645 new mem: 2845356061 -17> 2021-03-11 14:25:59.818 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 679067648 unmapped: 753664 heap: 679821312 old mem: 2845356061 new mem: 2845406382 -16> 2021-03-11 14:26:00.820 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 729161728 unmapped: 991232 heap: 730152960 old mem: 2845406382 new mem: 2845414228 -15> 2021-03-11 14:26:00.820 7f50161b7700 5 bluestore.MempoolThread(0x558aa9a8ea98) _trim_shards cache_size: 2845414228 kv_alloc: 1174405120 kv_used: 166707616 meta_alloc: 1006632960 meta_used: 54703 data_alloc: 637534208 data_used: 745472 -14> 2021-03-11 14:26:01.735 7f502962ba80 0 osd.80 697960 load_pgs opened 63 pgs -13> 2021-03-11 14:26:01.735 7f502962ba80 0 osd.80 697960 using weightedpriority op queue with priority op cut off at 64. -12> 2021-03-11 14:26:01.736 7f502962ba80 -1 osd.80 697960 log_to_monitors {default=true} -11> 2021-03-11 14:26:01.743 7f502962ba80 -1 osd.80 697960 mon_cmd_maybe_osd_create fail: 'osd.80 has already bound to class 'backup', can not reset class to 'hdd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy -10> 2021-03-11 14:26:01.746 7f502962ba80 0 osd.80 697960 done with init, starting boot process -9> 2021-03-11 14:26:01.748 7f5013740700 4 mgrc handle_mgr_map Got map version 175738 -8> 2021-03-11 14:26:01.748 7f5013740700 4 mgrc handle_mgr_map Active mgr is now [v2:10.69.57.2:6802/3699587,v1:10.69.57.2:6803/3699587] -7> 2021-03-11 14:26:01.748 7f5013740700 4 mgrc reconnect Starting new session with [v2:10.69.57.2:6802/3699587,v1:10.69.57.2:6803/3699587] -6> 2021-03-11 14:26:01.752 7f5013740700 4 mgrc handle_mgr_configure stats_period=5 -5> 2021-03-11 14:26:01.752 7f5013740700 4 mgrc handle_mgr_configure updated stats threshold: 5 -4> 2021-03-11 14:26:01.822 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 802258944 unmapped: 245760 heap: 802504704 old mem: 2845414228 new mem: 2845415533 -3> 2021-03-11 14:26:02.826 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 815693824 unmapped: 6733824 heap: 822427648 old mem: 2845415533 new mem: 2845415776 -2> 2021-03-11 14:26:03.084 7f5005b9c700 0 log_channel(cluster) log [INF] : 0.23 continuing backfill to osd.41 from (31979'7979,561511'9571] MIN to 561511'9571 -1> 2021-03-11 14:26:03.088 7f5003b98700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/osd/PGLog.cc: In function 'void PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)' thread 7f5003b98700 time 2021-03-11 14:26:03.080329 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/osd/PGLog.cc: 368: FAILED ceph_assert(log.head >= olog.tail && olog.head >= log.tail) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x558a9db62c7d] 2: (()+0x4d8e45) [0x558a9db62e45] 3: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1c22) [0x558a9dd76a42] 4: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x558a9dcd8234] 5: (PG::RecoveryState::Stray::react(MLogRec const&)+0x22b) [0x558a9dd1adeb] 6: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x558a9dd693f5] 7: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x2dd) [0x558a9dd2d3ed] 8: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x558a9dc69f34] 9: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x558a9ded2291] 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x558a9dc5ea4f] 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x558a9e216e56] 12: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x558a9e219970] 13: (()+0x7ea5) [0x7f5026673ea5] 14: (clone()+0x6d) [0x7f50255369fd] 0> 2021-03-11 14:26:03.099 7f5003b98700 -1 *** Caught signal (Aborted) ** in thread 7f5003b98700 thread_name:tp_osd_tp ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (()+0xf630) [0x7f502667b630] 2: (gsignal()+0x37) [0x7f502546e3d7] 3: (abort()+0x148) [0x7f502546fac8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x558a9db62ccc] 5: (()+0x4d8e45) [0x558a9db62e45] 6: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1c22) [0x558a9dd76a42] 7: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x558a9dcd8234] 8: (PG::RecoveryState::Stray::react(MLogRec const&)+0x22b) [0x558a9dd1adeb] 9: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x558a9dd693f5] 10: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x2dd) [0x558a9dd2d3ed] 11: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x558a9dc69f34] 12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x558a9ded2291] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x558a9dc5ea4f] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x558a9e216e56] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x558a9e219970] 16: (()+0x7ea5) [0x7f5026673ea5] 17: (clone()+0x6d) [0x7f50255369fd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 0/ 0 mds 0/ 0 mds_balancer 0/ 0 mds_locker 0/ 0 mds_log 0/ 0 mds_log_expire 0/ 0 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 0 journaler 0/ 0 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 1/ 1 reserver 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.80.log --- end dump of recent events --- On Thu, Mar 11, 2021 at 1:23 PM Igor Fedotov <ifedotov@suse.de> wrote:
Hi Cassiano,
the backtrace you've provided relates to the bug fixed by: https://github.com/ceph/ceph/pull/37793
This fix is going to be releases with the upcoming v14.2.17.
But I doubt that your original crashes have the same root cause - this issue appears during shutdown only.
Anyway you can work around it by using different allocator: avl or bitmap.
Thanks,
Igor
On 3/11/2021 6:21 PM, Cassiano Pilipavicius wrote:
Hi, please if someone know how to help, I have an HDD pool in mycluster and after rebooting one server, my osds has started to crash.
This pool is a backup pool and have OSD as failure domain with an size of 2.
After rebooting one server, My osds started to crash, and the thing is only getting worse. I have then tried to run ceph-bluestore-tool repair and I receive what I think is the same error that shows on the osd logs:
[root@cwvh13 ~]# ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-81 --log-level 10
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc:
In function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f6467ffcec0 time 2021-03-11 12:13:12.121766
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc:
53: FAILED ceph_assert(r == 0) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f645e1a7b27] 2: (()+0x25ccef) [0x7f645e1a7cef] 3: (()+0x3cd57f) [0x5642e85c457f] 4: (HybridAllocator::~HybridAllocator()+0x17) [0x5642e85f3f37] 5: (BlueStore::_close_alloc()+0x42) [0x5642e84379d2] 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x5642e84bbac8] 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x293) [0x5642e84bbf13] 8: (main()+0x13cc) [0x5642e83caaec] 9: (__libc_start_main()+0xf5) [0x7f645ae24555] 10: (()+0x1fae9f) [0x5642e83f1e9f] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks again Igor, using the ceph-bluestore-tool with the CEPH_ARGS="--bluestore_hybrid_alloc_mem_cap=0" I was able to detect two OSDs returning IO errors. This OSDs crashing has caused backfills operations that triggered some OSDs marking others as down due to some kind of slow operations and then started a domino effect that has left the pool in a really bad shape. I think that incomplete backfills left a lot of PGs with 0 objects and when this pgs started to backfill/recover, the OSDs crashed. Now I have removed the failed ODS and have some degraded/incomplete PGs and unfound objects that I am inserting back to the cluster with ceph-objectstore-tool. On Thu, Mar 11, 2021 at 2:34 PM Cassiano Pilipavicius <cpilipav@gmail.com> wrote:
Hi, really this error was only showing up when I've tried to run ceph-bluestore-tool repair, In my 3 OSDs that keeps crashing, it show the following log... please let me know if there is something I can do to get the pool back to a functioning state.
Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 21.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
-32> 2021-03-11 14:25:55.812 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 445022208 unmapped: 59588608 heap: 504610816 old mem: 134217728 new mem: 2564495564 -31> 2021-03-11 14:25:55.813 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 445210624 unmapped: 59400192 heap: 504610816 old mem: 2564495564 new mem: 2816296009 -30> 2021-03-11 14:25:55.813 7f50161b7700 5 bluestore.MempoolThread(0x558aa9a8ea98) _trim_shards cache_size: 2816296009 kv_alloc: 956301312 kv_used: 6321600 meta_alloc: 956301312 meta_used: 11680 data_alloc: 620756992 data_used: 151552 -29> 2021-03-11 14:25:55.838 7f502962ba80 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -28> 2021-03-11 14:25:55.839 7f502962ba80 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/cls/hello/cls_hello.cc:296: loading cls_hello -27> 2021-03-11 14:25:55.840 7f502962ba80 0 _get_class not permitted to load kvs -26> 2021-03-11 14:25:55.840 7f502962ba80 0 _get_class not permitted to load lua -25> 2021-03-11 14:25:55.852 7f502962ba80 0 _get_class not permitted to load sdk -24> 2021-03-11 14:25:55.853 7f502962ba80 0 osd.80 697960 crush map has features 283675107524608, adjusting msgr requires for clients -23> 2021-03-11 14:25:55.853 7f502962ba80 0 osd.80 697960 crush map has features 283675107524608 was 8705, adjusting msgr requires for mons -22> 2021-03-11 14:25:55.853 7f502962ba80 0 osd.80 697960 crush map has features 3026702624700514304, adjusting msgr requires for osds -21> 2021-03-11 14:25:56.814 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 501989376 unmapped: 21192704 heap: 523182080 old mem: 2816296009 new mem: 2842012351 -20> 2021-03-11 14:25:57.816 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 542867456 unmapped: 19210240 heap: 562077696 old mem: 2842012351 new mem: 2844985645 -19> 2021-03-11 14:25:58.327 7f502962ba80 0 osd.80 697960 load_pgs -18> 2021-03-11 14:25:58.818 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 596754432 unmapped: 229376 heap: 596983808 old mem: 2844985645 new mem: 2845356061 -17> 2021-03-11 14:25:59.818 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 679067648 unmapped: 753664 heap: 679821312 old mem: 2845356061 new mem: 2845406382 -16> 2021-03-11 14:26:00.820 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 729161728 unmapped: 991232 heap: 730152960 old mem: 2845406382 new mem: 2845414228 -15> 2021-03-11 14:26:00.820 7f50161b7700 5 bluestore.MempoolThread(0x558aa9a8ea98) _trim_shards cache_size: 2845414228 kv_alloc: 1174405120 kv_used: 166707616 meta_alloc: 1006632960 meta_used: 54703 data_alloc: 637534208 data_used: 745472 -14> 2021-03-11 14:26:01.735 7f502962ba80 0 osd.80 697960 load_pgs opened 63 pgs -13> 2021-03-11 14:26:01.735 7f502962ba80 0 osd.80 697960 using weightedpriority op queue with priority op cut off at 64. -12> 2021-03-11 14:26:01.736 7f502962ba80 -1 osd.80 697960 log_to_monitors {default=true} -11> 2021-03-11 14:26:01.743 7f502962ba80 -1 osd.80 697960 mon_cmd_maybe_osd_create fail: 'osd.80 has already bound to class 'backup', can not reset class to 'hdd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy -10> 2021-03-11 14:26:01.746 7f502962ba80 0 osd.80 697960 done with init, starting boot process -9> 2021-03-11 14:26:01.748 7f5013740700 4 mgrc handle_mgr_map Got map version 175738 -8> 2021-03-11 14:26:01.748 7f5013740700 4 mgrc handle_mgr_map Active mgr is now [v2:10.69.57.2:6802/3699587,v1:10.69.57.2:6803/3699587] -7> 2021-03-11 14:26:01.748 7f5013740700 4 mgrc reconnect Starting new session with [v2:10.69.57.2:6802/3699587,v1:10.69.57.2:6803/3699587] -6> 2021-03-11 14:26:01.752 7f5013740700 4 mgrc handle_mgr_configure stats_period=5 -5> 2021-03-11 14:26:01.752 7f5013740700 4 mgrc handle_mgr_configure updated stats threshold: 5 -4> 2021-03-11 14:26:01.822 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 802258944 unmapped: 245760 heap: 802504704 old mem: 2845414228 new mem: 2845415533 -3> 2021-03-11 14:26:02.826 7f50161b7700 5 prioritycache tune_memory target: 4294967296 mapped: 815693824 unmapped: 6733824 heap: 822427648 old mem: 2845415533 new mem: 2845415776 -2> 2021-03-11 14:26:03.084 7f5005b9c700 0 log_channel(cluster) log [INF] : 0.23 continuing backfill to osd.41 from (31979'7979,561511'9571] MIN to 561511'9571 -1> 2021-03-11 14:26:03.088 7f5003b98700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/osd/PGLog.cc: In function 'void PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)' thread 7f5003b98700 time 2021-03-11 14:26:03.080329 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/osd/PGLog.cc: 368: FAILED ceph_assert(log.head >= olog.tail && olog.head >= log.tail)
ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x558a9db62c7d] 2: (()+0x4d8e45) [0x558a9db62e45] 3: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1c22) [0x558a9dd76a42] 4: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x558a9dcd8234] 5: (PG::RecoveryState::Stray::react(MLogRec const&)+0x22b) [0x558a9dd1adeb] 6: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x558a9dd693f5] 7: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x2dd) [0x558a9dd2d3ed] 8: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x558a9dc69f34] 9: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x558a9ded2291] 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x558a9dc5ea4f] 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x558a9e216e56] 12: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x558a9e219970] 13: (()+0x7ea5) [0x7f5026673ea5] 14: (clone()+0x6d) [0x7f50255369fd]
0> 2021-03-11 14:26:03.099 7f5003b98700 -1 *** Caught signal (Aborted) ** in thread 7f5003b98700 thread_name:tp_osd_tp
ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (()+0xf630) [0x7f502667b630] 2: (gsignal()+0x37) [0x7f502546e3d7] 3: (abort()+0x148) [0x7f502546fac8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x558a9db62ccc] 5: (()+0x4d8e45) [0x558a9db62e45] 6: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1c22) [0x558a9dd76a42] 7: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x558a9dcd8234] 8: (PG::RecoveryState::Stray::react(MLogRec const&)+0x22b) [0x558a9dd1adeb] 9: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x558a9dd693f5] 10: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x2dd) [0x558a9dd2d3ed] 11: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x558a9dc69f34] 12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x558a9ded2291] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x558a9dc5ea4f] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x558a9e216e56] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x558a9e219970] 16: (()+0x7ea5) [0x7f5026673ea5] 17: (clone()+0x6d) [0x7f50255369fd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 0/ 0 mds 0/ 0 mds_balancer 0/ 0 mds_locker 0/ 0 mds_log 0/ 0 mds_log_expire 0/ 0 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 0 journaler 0/ 0 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 1/ 1 reserver 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.80.log --- end dump of recent events ---
On Thu, Mar 11, 2021 at 1:23 PM Igor Fedotov <ifedotov@suse.de> wrote:
Hi Cassiano,
the backtrace you've provided relates to the bug fixed by: https://github.com/ceph/ceph/pull/37793
This fix is going to be releases with the upcoming v14.2.17.
But I doubt that your original crashes have the same root cause - this issue appears during shutdown only.
Anyway you can work around it by using different allocator: avl or bitmap.
Thanks,
Igor
On 3/11/2021 6:21 PM, Cassiano Pilipavicius wrote:
Hi, please if someone know how to help, I have an HDD pool in mycluster and after rebooting one server, my osds has started to crash.
This pool is a backup pool and have OSD as failure domain with an size of 2.
After rebooting one server, My osds started to crash, and the thing is only getting worse. I have then tried to run ceph-bluestore-tool repair and I receive what I think is the same error that shows on the osd logs:
[root@cwvh13 ~]# ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-81 --log-level 10
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc:
In function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f6467ffcec0 time 2021-03-11 12:13:12.121766
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc:
53: FAILED ceph_assert(r == 0) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f645e1a7b27] 2: (()+0x25ccef) [0x7f645e1a7cef] 3: (()+0x3cd57f) [0x5642e85c457f] 4: (HybridAllocator::~HybridAllocator()+0x17) [0x5642e85f3f37] 5: (BlueStore::_close_alloc()+0x42) [0x5642e84379d2] 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x5642e84bbac8] 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x293) [0x5642e84bbf13] 8: (main()+0x13cc) [0x5642e83caaec] 9: (__libc_start_main()+0xf5) [0x7f645ae24555] 10: (()+0x1fae9f) [0x5642e83f1e9f] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Cassiano Pilipavicius
-
Igor Fedotov