- ceph-users - lists.ceph.io

Re: Questions on Ceph cluster without OS disks
by Marc Roos 21 Mar '20

21 Mar '20

I would say it is not a 'proven technology' otherwise you would see a wide spread implementation and adaptation of this method. However if you really need the physical disk space, it is a solution. Although I also would have questions on creating an extra redundant environment to service remote booting, just to spare a os disk position. Maybe this makes more sence in really big environments. -----Original Message----- From: huxiaoyu(a)horebdata.cn [mailto:huxiaoyu@horebdata.cn] Sent: 21 March 2020 13:54 To: Martin Verges; ceph-users Subject: [ceph-users] Questions on Ceph cluster without OS disks Hello， Martin， I notice that Croit advocate the use of ceph cluster without OS disks, but with PXE boot. Do you use a NFS server to serve the root file system for each node? such as hosting configuration files, user and password, log files, etc. My question is, will the NFS server be a single point of failure? If the NFS server goes down, the network experience any outage, ceph nodes may not be able to write to the local file systems, possibly leading to service outage. How do you deal with the above potential issues in production? I am a bit worried... best regards, samuel

2 1

New 3 node Ceph cluster
by Amudhan P 21 Mar '20

21 Mar '20

Hi, I am planning to create a new 3 node ceph storage cluster. I will be using Cephfs + with samba for max 10 clients for upload and download. Storage Node HW is Intel Xeon E5v2 8 core single Proc, 32GB RAM and 10Gb Nic 2 nos., 6TB SATA HDD 24 Nos. each node, OS separate SSD disk. Earlier I have tested orchestration using ceph-deploy in the test setup. now, is there any other alternative to ceph-deploy? Can I restrict folder access to the user using cephfs + vfs samba or should I use ceph client + samba? Ubuntu or Centos? Any block size consideration for object size, metadata when using cephfs? Idea or suggestion from existing users. I am also going to start to explore all the above. regards Amudhan

6 8

ceph objecy storage client gui
by Ignazio Cassano 21 Mar '20

21 Mar '20

Hello All, I am looking for object storage freee/opensource client gui (linux and windows) for end users . I tried swiftstack but it is only for personal use. Help, please ? Ignazio

2 2

Docs@RSS
by Guilherme Geronimo 20 Mar '20

20 Mar '20

Hey guys, do you know who takes care of the docs.ceph.com? I mean, the tool, not the content. I would suggest to enable its FEED (RSS/ATOM). That would facilitate a lot of things... -- []'s Arthur (aKa Guilherme Geronimo)

1 0

crush rule question
by Philippe Van Hecke 20 Mar '20

20 Mar '20

Hi , i have a question regarding crush rules. I try to have the following: i have domain faillure on rack. currently we have 2 racks what i want to make is a crush rules for 3 replicas where at least 2 replicas are put on 2 differents rack and the third one can be on any of the 2. Is this possible and if yes how to write the crrush rule. Kr Philippe Van Hecke.

1 0

Full OSD's on cephfs_metadata pool
by Robert Ruge 20 Mar '20

20 Mar '20

Hi All. Nautilus 14.2.8. I came in this morning to find that six of my eight NVME OSD's that were housing the cephfs_metadata pool had mysteriously filled up and crashed overnight and they won't come back up. These OSD's are all single logical volume devices with no separate WAL or DB. I have tried extending the LV of one of the OSD's but it can't make use of it and I have added a separate db volume but that didn't help. In the meantime I have told the cluster to move cephfs_metadata back to HDD which it has kindly done and emptied my two live OSD's but I am left with 10 pgs inactive. BLUEFS_SPILLOVER BlueFS spillover detected on 6 OSD(s) osd.93 spilled over 521 MiB metadata from 'db' device (26 GiB used of 50 GiB) to slow device osd.95 spilled over 456 MiB metadata from 'db' device (26 GiB used of 50 GiB) to slow device osd.100 spilled over 2.1 GiB metadata from 'db' device (26 GiB used of 50 GiB) to slow device osd.107 spilled over 782 MiB metadata from 'db' device (26 GiB used of 50 GiB) to slow device osd.112 spilled over 1.3 GiB metadata from 'db' device (27 GiB used of 50 GiB) to slow device osd.115 spilled over 1.4 GiB metadata from 'db' device (27 GiB used of 50 GiB) to slow device PG_AVAILABILITY Reduced data availability: 10 pgs inactive, 10 pgs down pg 2.4e is down, acting [60,6,120] pg 2.60 is down, acting [105,132,15] pg 2.61 is down, acting [8,13,112] pg 2.72 is down, acting [93,112,0] pg 2.9f is down, acting [117,1,35] pg 2.b9 is down, acting [95,25,6] pg 2.c3 is down, acting [97,139,5] pg 2.c6 is down, acting [95,7,127] pg 2.d1 is down, acting [36,107,17] pg 2.f4 is down, acting [23,117,138] Can I backup and recreate an OSD on a larger volume? Can I remove a good pg from an offline OSD to remove some space? Ceph-bluestore-tool repair fails. "bluefs enospc" seems to be the critical error. So currently my cephfs is unavailable so any help would be greatly appreciated. Regards Robert Ruge Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

4 5

Re: OSDs continuously restarting under load
by Igor Fedotov 19 Mar '20

19 Mar '20

Hi, Samuel, I've never seen that sort of signal in the real life: 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** I suppose this has some hardware roots. Have you checked dmesg output? Just in case, here is some info on "Bus Error" signal, may be it will provide some insight: https://en.wikipedia.org/wiki/Bus_error Thanks, Igor On 3/18/2020 5:06 PM, huxiaoyu(a)horebdata.cn wrote: > Hello， folks, > > I am trying to add a ceph node into an existing ceph cluster. Once the reweight of newly-added OSD on the new node exceed 0.4 somewhere, the osd becomes unresponsive and restarting, eventually go down. > > What could be the problem? Any suggestion would be highly appreciated. > > best regards, > > samuel > > **************************************************** > root@node81:/var/log/ceph# > root@node81:/var/log/ceph# > root@node81:/var/log/ceph# > root@node81:/var/log/ceph# ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 12 hybrid 1.00000 1.00000 3.81TiB 38.3GiB 3.77TiB 0.98 1.32 316 > 13 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 308 > 14 hybrid 1.00000 1.00000 3.81TiB 36.9GiB 3.77TiB 0.95 1.27 301 > 15 hybrid 1.00000 1.00000 3.81TiB 37.1GiB 3.77TiB 0.95 1.28 297 > 0 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 305 > 1 hybrid 1.00000 1.00000 3.81TiB 38.2GiB 3.77TiB 0.98 1.31 309 > 2 hybrid 1.00000 1.00000 3.81TiB 37.4GiB 3.77TiB 0.96 1.29 296 > 3 hybrid 1.00000 1.00000 3.81TiB 37.9GiB 3.77TiB 0.97 1.30 303 > 4 hdd 0.20000 1.00000 3.42TiB 10.5GiB 3.41TiB 0.30 0.40 0 > 5 hdd 0.20000 1.00000 3.42TiB 9.63GiB 3.41TiB 0.28 0.37 87 > 6 hdd 0.20000 1.00000 3.42TiB 1.91GiB 3.42TiB 0.05 0.07 0 > 7 hdd 0.20000 1.00000 3.42TiB 11.3GiB 3.41TiB 0.32 0.43 83 > 16 hdd 0.39999 1.00000 1.79TiB 16.3GiB 1.78TiB 0.89 1.19 142 > TOTAL 45.9TiB 351GiB 45.6TiB 0.75 > > ------------------------------------------------------------------------------------ 日志 > > root@node81:/var/log/ceph# cat ceph-osd.6.log | grep load_pgs > 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs > -1324> 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > -1323> 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs > 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs > -466> 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > -465> 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs > 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs > -422> 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > -421> 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs > 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs > > > 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none > 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none > 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete > 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** > in thread 201e35fdb40 thread_name:tp_osd_tp > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) > 1: (()+0x145882c) [0x2000245882c] > 2: (()+0x19890) [0x2000c54b890] > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] > 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] > 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > 19: (()+0x80fc) [0x2000c53a0fc] > 20: (()+0x119854) [0x2000f2ad854] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- begin dump of recent events --- > -147> 2020-03-18 18:37:51.039443 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_dump hook 0x2000ce09e40 > -146> 2020-03-18 18:37:51.039716 2000f374000 5 asok(0x2000cd7f230) register_command 1 hook 0x2000ce09e40 > -145> 2020-03-18 18:37:51.039736 2000f374000 5 asok(0x2000cd7f230) register_command perf dump hook 0x2000ce09e40 > -144> 2020-03-18 18:37:51.039769 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_schema hook 0x2000ce09e40 > -143> 2020-03-18 18:37:51.039789 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram dump hook 0x2000ce09e40 > -142> 2020-03-18 18:37:51.039807 2000f374000 5 asok(0x2000cd7f230) register_command 2 hook 0x2000ce09e40 > -141> 2020-03-18 18:37:51.039823 2000f374000 5 asok(0x2000cd7f230) register_command perf schema hook 0x2000ce09e40 > -140> 2020-03-18 18:37:51.039843 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram schema hook 0x2000ce09e40 > -139> 2020-03-18 18:37:51.039863 2000f374000 5 asok(0x2000cd7f230) register_command perf reset hook 0x2000ce09e40 > -138> 2020-03-18 18:37:51.039881 2000f374000 5 asok(0x2000cd7f230) register_command config show hook 0x2000ce09e40 > -137> 2020-03-18 18:37:51.039899 2000f374000 5 asok(0x2000cd7f230) register_command config help hook 0x2000ce09e40 > -136> 2020-03-18 18:37:51.039928 2000f374000 5 asok(0x2000cd7f230) register_command config set hook 0x2000ce09e40 > -135> 2020-03-18 18:37:51.039949 2000f374000 5 asok(0x2000cd7f230) register_command config get hook 0x2000ce09e40 > -134> 2020-03-18 18:37:51.039967 2000f374000 5 asok(0x2000cd7f230) register_command config diff hook 0x2000ce09e40 > -133> 2020-03-18 18:37:51.039985 2000f374000 5 asok(0x2000cd7f230) register_command config diff get hook 0x2000ce09e40 > -132> 2020-03-18 18:37:51.040005 2000f374000 5 asok(0x2000cd7f230) register_command log flush hook 0x2000ce09e40 > -131> 2020-03-18 18:37:51.040021 2000f374000 5 asok(0x2000cd7f230) register_command log dump hook 0x2000ce09e40 > -130> 2020-03-18 18:37:51.040038 2000f374000 5 asok(0x2000cd7f230) register_command log reopen hook 0x2000ce09e40 > -129> 2020-03-18 18:37:51.040189 2000f374000 5 asok(0x2000cd7f230) register_command dump_mempools hook 0x2000ce0d038 > -128> 2020-03-18 18:37:51.099580 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > -127> 2020-03-18 18:37:51.102046 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > -126> 2020-03-18 18:37:51.102148 2000f374000 0 ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process ceph-osd, pid 1159687 > -125> 2020-03-18 18:37:51.109860 2000f374000 0 pidfile_write: ignore empty --pid-file > -124> 2020-03-18 18:37:51.130256 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > -123> 2020-03-18 18:37:51.183798 2000f374000 0 load: jerasure load: lrc > -122> 2020-03-18 18:37:51.184330 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel > -121> 2020-03-18 18:37:51.184492 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > -120> 2020-03-18 18:37:51.184741 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -119> 2020-03-18 18:37:51.185841 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational > -118> 2020-03-18 18:37:51.186232 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) close > -117> 2020-03-18 18:37:51.498718 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel > -116> 2020-03-18 18:37:51.498748 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > -115> 2020-03-18 18:37:51.498811 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -114> 2020-03-18 18:37:51.499234 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational > -113> 2020-03-18 18:37:51.500074 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block.db type kernel > -112> 2020-03-18 18:37:51.500096 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open path /var/lib/ceph/osd/ceph-6/block.db > -111> 2020-03-18 18:37:51.500170 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -110> 2020-03-18 18:37:51.500815 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open size 39998980096 (0x950200000, 38146 MB) block_size 4096 (4096 B) rotational > -109> 2020-03-18 18:37:51.502625 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel > -108> 2020-03-18 18:37:51.502651 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > -107> 2020-03-18 18:37:51.502718 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -106> 2020-03-18 18:37:51.503137 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational > -105> 2020-03-18 18:37:51.549269 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB > -104> 2020-03-18 18:37:51.549349 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel > -103> 2020-03-18 18:37:51.552610 2000f374000 0 set rocksdb option compaction_threads = 32 > -102> 2020-03-18 18:37:51.552652 2000f374000 0 set rocksdb option compression = kNoCompression > -101> 2020-03-18 18:37:51.553442 2000f374000 0 set rocksdb option flusher_threads = 8 > -100> 2020-03-18 18:37:51.553508 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 > -99> 2020-03-18 18:37:51.553536 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 > -98> 2020-03-18 18:37:51.553559 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 > -97> 2020-03-18 18:37:51.553579 2000f374000 0 set rocksdb option max_background_compactions = 64 > -96> 2020-03-18 18:37:51.553601 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB > -95> 2020-03-18 18:37:51.553624 2000f374000 0 set rocksdb option max_write_buffer_number = 64 > -94> 2020-03-18 18:37:51.553646 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 > -93> 2020-03-18 18:37:51.553665 2000f374000 0 set rocksdb option recycle_log_file_num = 64 > -92> 2020-03-18 18:37:51.553687 2000f374000 0 set rocksdb option target_file_size_base = 4MB > -91> 2020-03-18 18:37:51.553708 2000f374000 0 set rocksdb option write_buffer_size = 4MB > -90> 2020-03-18 18:37:51.553892 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB > -89> 2020-03-18 18:37:51.553923 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel > -88> 2020-03-18 18:37:51.553948 2000f374000 0 set rocksdb option compaction_threads = 32 > -87> 2020-03-18 18:37:51.553973 2000f374000 0 set rocksdb option compression = kNoCompression > -86> 2020-03-18 18:37:51.553994 2000f374000 0 set rocksdb option flusher_threads = 8 > -85> 2020-03-18 18:37:51.554016 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 > -84> 2020-03-18 18:37:51.554043 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 > -83> 2020-03-18 18:37:51.554065 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 > -82> 2020-03-18 18:37:51.554084 2000f374000 0 set rocksdb option max_background_compactions = 64 > -81> 2020-03-18 18:37:51.554106 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB > -80> 2020-03-18 18:37:51.554133 2000f374000 0 set rocksdb option max_write_buffer_number = 64 > -79> 2020-03-18 18:37:51.554154 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 > -78> 2020-03-18 18:37:51.554174 2000f374000 0 set rocksdb option recycle_log_file_num = 64 > -77> 2020-03-18 18:37:51.554196 2000f374000 0 set rocksdb option target_file_size_base = 4MB > -76> 2020-03-18 18:37:51.554232 2000f374000 0 set rocksdb option write_buffer_size = 4MB > -75> 2020-03-18 18:37:56.382110 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: loading cls_hello > -74> 2020-03-18 18:37:56.383845 2000f374000 0 _get_class not permitted to load lua > -73> 2020-03-18 18:37:56.386594 2000f374000 0 _get_class not permitted to load sdk > -72> 2020-03-18 18:37:56.395800 2000f374000 0 _get_class not permitted to load kvs > -71> 2020-03-18 18:37:56.398226 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs > -70> 2020-03-18 18:37:56.433293 2000f374000 0 osd.6 5247 crush map has features 432629239337189376, adjusting msgr requires for clients > -69> 2020-03-18 18:37:56.433330 2000f374000 0 osd.6 5247 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons > -68> 2020-03-18 18:37:56.433357 2000f374000 0 osd.6 5247 crush map has features 1009089991640629248, adjusting msgr requires for osds > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs > -65> 2020-03-18 18:39:03.377040 2000f374000 0 osd.6 5247 using weightedpriority op queue with priority op cut off at 196. > -64> 2020-03-18 18:39:03.413901 2000f374000 -1 osd.6 5247 log_to_monitors {default=true} > -63> 2020-03-18 18:39:03.663128 2000f374000 0 osd.6 5247 done with init, starting boot process > -62> 2020-03-18 18:39:03.663856 201d65fdb40 4 mgrc handle_mgr_map Got map version 34 > -61> 2020-03-18 18:39:03.663947 201d65fdb40 4 mgrc handle_mgr_map Active mgr is now 192.168.230.120:6808/44007 > -60> 2020-03-18 18:39:03.663972 201d65fdb40 4 mgrc reconnect Starting new session with 192.168.230.120:6808/44007 > -59> 2020-03-18 18:39:03.667814 201d65fdb40 4 mgrc handle_mgr_configure stats_period=5 > -58> 2020-03-18 18:39:03.667831 201d65fdb40 4 mgrc handle_mgr_configure updated stats threshold: 5 > -57> 2020-03-18 18:39:03.752253 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6802/33592 conn(0x20014018040 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -56> 2020-03-18 18:39:03.754040 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6800/33864 conn(0x2001403ee60 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -55> 2020-03-18 18:39:03.762441 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6804/13410 conn(0x20014079940 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -54> 2020-03-18 18:39:03.762860 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6806/13743 conn(0x20014083980 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -53> 2020-03-18 18:39:03.765775 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6806/12423 conn(0x2001407e030 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -52> 2020-03-18 18:39:03.767284 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6804/11599 conn(0x2001408c660 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -51> 2020-03-18 18:39:03.769434 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6800/12205 conn(0x20014087ff0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -50> 2020-03-18 18:39:03.775190 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6802/12991 conn(0x2001409d300 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -49> 2020-03-18 18:39:04.009755 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.122:6804/1159466 conn(0x200140b3420 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -48> 2020-03-18 18:39:05.177544 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.202:0/3091162658 conn(0x200140cd900 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -47> 2020-03-18 18:39:05.402465 2000dae5b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4289863819 conn(0x200140d8500 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -46> 2020-03-18 18:39:09.483237 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b442b93f:::rbd_data.911772ae8944a.0000000000002aa7:head have 3097'452 flags = none tried to add 3097'452 flags = none > -45> 2020-03-18 18:39:09.483318 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b445877b:::rbd_data.2fa7e6b8b4567.000000000000002d:head have 1915'212 flags = none tried to add 1915'212 flags = none > -44> 2020-03-18 18:39:09.483336 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44674a4:::rbd_data.110706b8b4567.0000000000000659:head have 1915'213 flags = none tried to add 1915'213 flags = none > -43> 2020-03-18 18:39:09.483351 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44757d1:::rbd_data.cd282238e1f29.0000000000009ea2:head have 5165'734 flags = none tried to add 5165'734 flags = none > -42> 2020-03-18 18:39:09.483366 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4490029:::rbd_data.8d8146b8b4567.00000000000080a0:head have 2855'272 flags = none tried to add 2855'272 flags = none > -41> 2020-03-18 18:39:09.483381 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44950ae:::rbd_data.5f15f625558ec.0000000000009fd2:head have 1915'214 flags = none tried to add 1915'214 flags = none > -40> 2020-03-18 18:39:09.483395 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b449ce0d:::rbd_data.9117a327b23c6.00000000000074a6:head have 3798'560 flags = none tried to add 3798'560 flags = none > -39> 2020-03-18 18:39:09.483409 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44a69ad:::rbd_data.8b4a76b8b4567.000000000000017a:head have 2197'242 flags = none tried to add 2197'242 flags = none > -38> 2020-03-18 18:39:09.483423 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44bbb34:::rbd_data.8922074b0dc51.00000000000098a5:head have 3099'543 flags = delete tried to add 3099'543 flags = delete > -37> 2020-03-18 18:39:09.483438 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44be196:::rbd_data.cd8e219495cff.00000000000192a0:head have 5169'1101 flags = delete tried to add 5169'1101 flags = delete > -36> 2020-03-18 18:39:09.483454 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44cd1c7:::rbd_data.5f15f625558ec.000000000000820b:head have 1915'215 flags = none tried to add 1915'215 flags = none > -35> 2020-03-18 18:39:09.483469 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44e04c0:::rbd_data.cd66c46e87ccd.0000000000015aa4:head have 5175'1280 flags = none tried to add 5175'1280 flags = none > -34> 2020-03-18 18:39:09.483483 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44fa767:::rbd_data.ccf5e625558ec.00000000000150a0:head have 5166'844 flags = delete tried to add 5166'844 flags = delete > -33> 2020-03-18 18:39:09.483497 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4518cb0:::rbd_data.589572ae8944a.0000000000000433:head have 1805'174 flags = delete tried to add 1805'174 flags = delete > -32> 2020-03-18 18:39:09.483511 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4539e4a:::rbd_data.5f15f625558ec.000000000000031a:head have 1915'216 flags = none tried to add 1915'216 flags = none > -31> 2020-03-18 18:39:09.483525 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4540dfd:::rbd_data.cd66c46e87ccd.000000000001d6a6:head have 5175'1283 flags = none tried to add 5175'1283 flags = none > -30> 2020-03-18 18:39:09.483539 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45474ee:::rbd_data.2fa7e6b8b4567.000000000000003a:head have 1915'217 flags = none tried to add 1915'217 flags = none > -29> 2020-03-18 18:39:09.483553 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45506f1:::rbd_data.9177446e87ccd.00000000000036c3:head have 5175'1284 flags = none tried to add 5175'1284 flags = none > -28> 2020-03-18 18:39:09.483567 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4576e29:::rbd_data.589572ae8944a.0000000000001207:head have 1805'179 flags = delete tried to add 1805'179 flags = delete > -27> 2020-03-18 18:39:09.483582 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4578057:::rbd_data.ccf5e625558ec.00000000000096a5:head have 5166'785 flags = delete tried to add 5166'785 flags = delete > -26> 2020-03-18 18:39:09.483595 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b457fe94:::rbd_data.589572ae8944a.0000000000004e27:head have 1805'200 flags = delete tried to add 1805'200 flags = delete > -25> 2020-03-18 18:39:09.483610 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4592563:::rbd_data.9117a327b23c6.000000000000b2a0:head have 3894'562 flags = none tried to add 3894'562 flags = none > -24> 2020-03-18 18:39:09.483625 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45bb3ff:::rbd_data.c6c99507ed7ab.000000000000aea0:head have 5172'1141 flags = none tried to add 5172'1141 flags = none > -23> 2020-03-18 18:39:09.483639 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45d7c66:::rbd_data.110706b8b4567.0000000000000e07:head have 1915'218 flags = none tried to add 1915'218 flags = none > -22> 2020-03-18 18:39:09.483654 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45e1a62:::rbd_data.589572ae8944a.0000000000004733:head have 1805'198 flags = delete tried to add 1805'198 flags = delete > -21> 2020-03-18 18:39:09.483667 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45f03a3:::rbd_data.914c22ae8944a.00000000000058a8:head have 3908'593 flags = delete tried to add 3908'593 flags = delete > -20> 2020-03-18 18:39:09.483681 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4604fdd:::rbd_data.c6c99507ed7ab.00000000000004ca:head have 5176'1523 flags = none tried to add 5176'1523 flags = none > -19> 2020-03-18 18:39:09.483695 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4636d5b:::rbd_data.cd282238e1f29.000000000000b6a1:head have 5165'735 flags = none tried to add 5165'735 flags = none > -18> 2020-03-18 18:39:09.483710 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b463cd0a:::rbd_data.9117a327b23c6.000000000000c0a8:head have 3908'563 flags = none tried to add 3908'563 flags = none > -17> 2020-03-18 18:39:09.483724 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b468ca37:::rbd_data.9177446e87ccd.000000000000040a:head have 4427'730 flags = none tried to add 4427'730 flags = none > -16> 2020-03-18 18:39:09.483738 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4696438:::rbd_data.9177446e87ccd.0000000000000c53:head have 4743'732 flags = none tried to add 4743'732 flags = none > -15> 2020-03-18 18:39:09.483752 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46be061:::rbd_data.c70aa2eb141f2.0000000000001ea6:head have 5175'1287 flags = none tried to add 5175'1287 flags = none > -14> 2020-03-18 18:39:09.483766 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46bfc4e:::rbd_data.cda772ae8944a.00000000000020a4:head have 5172'1144 flags = none tried to add 5172'1144 flags = none > -13> 2020-03-18 18:39:09.483782 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46e7efd:::rbd_data.110706b8b4567.0000000000001c01:head have 1915'219 flags = none tried to add 1915'219 flags = none > -12> 2020-03-18 18:39:09.483796 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46ece7b:::rbd_data.8ff1766334873.00000000000000ae:head have 3092'344 flags = delete tried to add 3092'344 flags = delete > -11> 2020-03-18 18:39:09.483811 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4786571:::rbd_data.8f6b3643c9869.00000000000040a0:head have 3092'381 flags = delete tried to add 3092'381 flags = delete > -10> 2020-03-18 18:39:09.483825 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47888b8:::rbd_data.589572ae8944a.0000000000004534:head have 1805'197 flags = delete tried to add 1805'197 flags = delete > -9> 2020-03-18 18:39:09.483840 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b478b833:::rbd_data.589572ae8944a.00000000000048ff:head have 1805'199 flags = delete tried to add 1805'199 flags = delete > -8> 2020-03-18 18:39:09.483854 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47ec645:::rbd_data.8b4a76b8b4567.00000000000001d2:head have 2197'271 flags = none tried to add 2197'271 flags = none > -7> 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none > -6> 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none > -5> 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete > -4> 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -3> 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -2> 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -1> 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 0> 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** > in thread 201e35fdb40 thread_name:tp_osd_tp > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) > 1: (()+0x145882c) [0x2000245882c] > 2: (()+0x19890) [0x2000c54b890] > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] > 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] > 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > 19: (()+0x80fc) [0x2000c53a0fc] > 20: (()+0x119854) [0x2000f2ad854] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

OSDs continuously restarting under load
by huxiaoyu＠horebdata.cn 18 Mar '20

18 Mar '20

Hello， folks, I am trying to add a ceph node into an existing ceph cluster. Once the reweight of newly-added OSD on the new node exceed 0.4 somewhere, the osd becomes unresponsive and restarting, eventually go down. What could be the problem? Any suggestion would be highly appreciated. best regards, samuel **************************************************** root@node81:/var/log/ceph# root@node81:/var/log/ceph# root@node81:/var/log/ceph# root@node81:/var/log/ceph# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 12 hybrid 1.00000 1.00000 3.81TiB 38.3GiB 3.77TiB 0.98 1.32 316 13 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 308 14 hybrid 1.00000 1.00000 3.81TiB 36.9GiB 3.77TiB 0.95 1.27 301 15 hybrid 1.00000 1.00000 3.81TiB 37.1GiB 3.77TiB 0.95 1.28 297 0 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 305 1 hybrid 1.00000 1.00000 3.81TiB 38.2GiB 3.77TiB 0.98 1.31 309 2 hybrid 1.00000 1.00000 3.81TiB 37.4GiB 3.77TiB 0.96 1.29 296 3 hybrid 1.00000 1.00000 3.81TiB 37.9GiB 3.77TiB 0.97 1.30 303 4 hdd 0.20000 1.00000 3.42TiB 10.5GiB 3.41TiB 0.30 0.40 0 5 hdd 0.20000 1.00000 3.42TiB 9.63GiB 3.41TiB 0.28 0.37 87 6 hdd 0.20000 1.00000 3.42TiB 1.91GiB 3.42TiB 0.05 0.07 0 7 hdd 0.20000 1.00000 3.42TiB 11.3GiB 3.41TiB 0.32 0.43 83 16 hdd 0.39999 1.00000 1.79TiB 16.3GiB 1.78TiB 0.89 1.19 142 TOTAL 45.9TiB 351GiB 45.6TiB 0.75 ------------------------------------------------------------------------------------ 日志 root@node81:/var/log/ceph# cat ceph-osd.6.log | grep load_pgs 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs -1324> 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs -1323> 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs -466> 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs -465> 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs -422> 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs -421> 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** in thread 201e35fdb40 thread_name:tp_osd_tp ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (()+0x145882c) [0x2000245882c] 2: (()+0x19890) [0x2000c54b890] 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] 19: (()+0x80fc) [0x2000c53a0fc] 20: (()+0x119854) [0x2000f2ad854] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -147> 2020-03-18 18:37:51.039443 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_dump hook 0x2000ce09e40 -146> 2020-03-18 18:37:51.039716 2000f374000 5 asok(0x2000cd7f230) register_command 1 hook 0x2000ce09e40 -145> 2020-03-18 18:37:51.039736 2000f374000 5 asok(0x2000cd7f230) register_command perf dump hook 0x2000ce09e40 -144> 2020-03-18 18:37:51.039769 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_schema hook 0x2000ce09e40 -143> 2020-03-18 18:37:51.039789 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram dump hook 0x2000ce09e40 -142> 2020-03-18 18:37:51.039807 2000f374000 5 asok(0x2000cd7f230) register_command 2 hook 0x2000ce09e40 -141> 2020-03-18 18:37:51.039823 2000f374000 5 asok(0x2000cd7f230) register_command perf schema hook 0x2000ce09e40 -140> 2020-03-18 18:37:51.039843 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram schema hook 0x2000ce09e40 -139> 2020-03-18 18:37:51.039863 2000f374000 5 asok(0x2000cd7f230) register_command perf reset hook 0x2000ce09e40 -138> 2020-03-18 18:37:51.039881 2000f374000 5 asok(0x2000cd7f230) register_command config show hook 0x2000ce09e40 -137> 2020-03-18 18:37:51.039899 2000f374000 5 asok(0x2000cd7f230) register_command config help hook 0x2000ce09e40 -136> 2020-03-18 18:37:51.039928 2000f374000 5 asok(0x2000cd7f230) register_command config set hook 0x2000ce09e40 -135> 2020-03-18 18:37:51.039949 2000f374000 5 asok(0x2000cd7f230) register_command config get hook 0x2000ce09e40 -134> 2020-03-18 18:37:51.039967 2000f374000 5 asok(0x2000cd7f230) register_command config diff hook 0x2000ce09e40 -133> 2020-03-18 18:37:51.039985 2000f374000 5 asok(0x2000cd7f230) register_command config diff get hook 0x2000ce09e40 -132> 2020-03-18 18:37:51.040005 2000f374000 5 asok(0x2000cd7f230) register_command log flush hook 0x2000ce09e40 -131> 2020-03-18 18:37:51.040021 2000f374000 5 asok(0x2000cd7f230) register_command log dump hook 0x2000ce09e40 -130> 2020-03-18 18:37:51.040038 2000f374000 5 asok(0x2000cd7f230) register_command log reopen hook 0x2000ce09e40 -129> 2020-03-18 18:37:51.040189 2000f374000 5 asok(0x2000cd7f230) register_command dump_mempools hook 0x2000ce0d038 -128> 2020-03-18 18:37:51.099580 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb -127> 2020-03-18 18:37:51.102046 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb -126> 2020-03-18 18:37:51.102148 2000f374000 0 ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process ceph-osd, pid 1159687 -125> 2020-03-18 18:37:51.109860 2000f374000 0 pidfile_write: ignore empty --pid-file -124> 2020-03-18 18:37:51.130256 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb -123> 2020-03-18 18:37:51.183798 2000f374000 0 load: jerasure load: lrc -122> 2020-03-18 18:37:51.184330 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel -121> 2020-03-18 18:37:51.184492 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block -120> 2020-03-18 18:37:51.184741 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway -119> 2020-03-18 18:37:51.185841 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational -118> 2020-03-18 18:37:51.186232 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) close -117> 2020-03-18 18:37:51.498718 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel -116> 2020-03-18 18:37:51.498748 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block -115> 2020-03-18 18:37:51.498811 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway -114> 2020-03-18 18:37:51.499234 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational -113> 2020-03-18 18:37:51.500074 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block.db type kernel -112> 2020-03-18 18:37:51.500096 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open path /var/lib/ceph/osd/ceph-6/block.db -111> 2020-03-18 18:37:51.500170 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway -110> 2020-03-18 18:37:51.500815 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open size 39998980096 (0x950200000, 38146 MB) block_size 4096 (4096 B) rotational -109> 2020-03-18 18:37:51.502625 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel -108> 2020-03-18 18:37:51.502651 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block -107> 2020-03-18 18:37:51.502718 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway -106> 2020-03-18 18:37:51.503137 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational -105> 2020-03-18 18:37:51.549269 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB -104> 2020-03-18 18:37:51.549349 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel -103> 2020-03-18 18:37:51.552610 2000f374000 0 set rocksdb option compaction_threads = 32 -102> 2020-03-18 18:37:51.552652 2000f374000 0 set rocksdb option compression = kNoCompression -101> 2020-03-18 18:37:51.553442 2000f374000 0 set rocksdb option flusher_threads = 8 -100> 2020-03-18 18:37:51.553508 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 -99> 2020-03-18 18:37:51.553536 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 -98> 2020-03-18 18:37:51.553559 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 -97> 2020-03-18 18:37:51.553579 2000f374000 0 set rocksdb option max_background_compactions = 64 -96> 2020-03-18 18:37:51.553601 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB -95> 2020-03-18 18:37:51.553624 2000f374000 0 set rocksdb option max_write_buffer_number = 64 -94> 2020-03-18 18:37:51.553646 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 -93> 2020-03-18 18:37:51.553665 2000f374000 0 set rocksdb option recycle_log_file_num = 64 -92> 2020-03-18 18:37:51.553687 2000f374000 0 set rocksdb option target_file_size_base = 4MB -91> 2020-03-18 18:37:51.553708 2000f374000 0 set rocksdb option write_buffer_size = 4MB -90> 2020-03-18 18:37:51.553892 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB -89> 2020-03-18 18:37:51.553923 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel -88> 2020-03-18 18:37:51.553948 2000f374000 0 set rocksdb option compaction_threads = 32 -87> 2020-03-18 18:37:51.553973 2000f374000 0 set rocksdb option compression = kNoCompression -86> 2020-03-18 18:37:51.553994 2000f374000 0 set rocksdb option flusher_threads = 8 -85> 2020-03-18 18:37:51.554016 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 -84> 2020-03-18 18:37:51.554043 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 -83> 2020-03-18 18:37:51.554065 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 -82> 2020-03-18 18:37:51.554084 2000f374000 0 set rocksdb option max_background_compactions = 64 -81> 2020-03-18 18:37:51.554106 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB -80> 2020-03-18 18:37:51.554133 2000f374000 0 set rocksdb option max_write_buffer_number = 64 -79> 2020-03-18 18:37:51.554154 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 -78> 2020-03-18 18:37:51.554174 2000f374000 0 set rocksdb option recycle_log_file_num = 64 -77> 2020-03-18 18:37:51.554196 2000f374000 0 set rocksdb option target_file_size_base = 4MB -76> 2020-03-18 18:37:51.554232 2000f374000 0 set rocksdb option write_buffer_size = 4MB -75> 2020-03-18 18:37:56.382110 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: loading cls_hello -74> 2020-03-18 18:37:56.383845 2000f374000 0 _get_class not permitted to load lua -73> 2020-03-18 18:37:56.386594 2000f374000 0 _get_class not permitted to load sdk -72> 2020-03-18 18:37:56.395800 2000f374000 0 _get_class not permitted to load kvs -71> 2020-03-18 18:37:56.398226 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -70> 2020-03-18 18:37:56.433293 2000f374000 0 osd.6 5247 crush map has features 432629239337189376, adjusting msgr requires for clients -69> 2020-03-18 18:37:56.433330 2000f374000 0 osd.6 5247 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons -68> 2020-03-18 18:37:56.433357 2000f374000 0 osd.6 5247 crush map has features 1009089991640629248, adjusting msgr requires for osds -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs -65> 2020-03-18 18:39:03.377040 2000f374000 0 osd.6 5247 using weightedpriority op queue with priority op cut off at 196. -64> 2020-03-18 18:39:03.413901 2000f374000 -1 osd.6 5247 log_to_monitors {default=true} -63> 2020-03-18 18:39:03.663128 2000f374000 0 osd.6 5247 done with init, starting boot process -62> 2020-03-18 18:39:03.663856 201d65fdb40 4 mgrc handle_mgr_map Got map version 34 -61> 2020-03-18 18:39:03.663947 201d65fdb40 4 mgrc handle_mgr_map Active mgr is now 192.168.230.120:6808/44007 -60> 2020-03-18 18:39:03.663972 201d65fdb40 4 mgrc reconnect Starting new session with 192.168.230.120:6808/44007 -59> 2020-03-18 18:39:03.667814 201d65fdb40 4 mgrc handle_mgr_configure stats_period=5 -58> 2020-03-18 18:39:03.667831 201d65fdb40 4 mgrc handle_mgr_configure updated stats threshold: 5 -57> 2020-03-18 18:39:03.752253 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6802/33592 conn(0x20014018040 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -56> 2020-03-18 18:39:03.754040 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6800/33864 conn(0x2001403ee60 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -55> 2020-03-18 18:39:03.762441 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6804/13410 conn(0x20014079940 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -54> 2020-03-18 18:39:03.762860 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6806/13743 conn(0x20014083980 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -53> 2020-03-18 18:39:03.765775 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6806/12423 conn(0x2001407e030 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -52> 2020-03-18 18:39:03.767284 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6804/11599 conn(0x2001408c660 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -51> 2020-03-18 18:39:03.769434 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6800/12205 conn(0x20014087ff0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -50> 2020-03-18 18:39:03.775190 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6802/12991 conn(0x2001409d300 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -49> 2020-03-18 18:39:04.009755 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.122:6804/1159466 conn(0x200140b3420 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -48> 2020-03-18 18:39:05.177544 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.202:0/3091162658 conn(0x200140cd900 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -47> 2020-03-18 18:39:05.402465 2000dae5b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4289863819 conn(0x200140d8500 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -46> 2020-03-18 18:39:09.483237 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b442b93f:::rbd_data.911772ae8944a.0000000000002aa7:head have 3097'452 flags = none tried to add 3097'452 flags = none -45> 2020-03-18 18:39:09.483318 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b445877b:::rbd_data.2fa7e6b8b4567.000000000000002d:head have 1915'212 flags = none tried to add 1915'212 flags = none -44> 2020-03-18 18:39:09.483336 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44674a4:::rbd_data.110706b8b4567.0000000000000659:head have 1915'213 flags = none tried to add 1915'213 flags = none -43> 2020-03-18 18:39:09.483351 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44757d1:::rbd_data.cd282238e1f29.0000000000009ea2:head have 5165'734 flags = none tried to add 5165'734 flags = none -42> 2020-03-18 18:39:09.483366 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4490029:::rbd_data.8d8146b8b4567.00000000000080a0:head have 2855'272 flags = none tried to add 2855'272 flags = none -41> 2020-03-18 18:39:09.483381 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44950ae:::rbd_data.5f15f625558ec.0000000000009fd2:head have 1915'214 flags = none tried to add 1915'214 flags = none -40> 2020-03-18 18:39:09.483395 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b449ce0d:::rbd_data.9117a327b23c6.00000000000074a6:head have 3798'560 flags = none tried to add 3798'560 flags = none -39> 2020-03-18 18:39:09.483409 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44a69ad:::rbd_data.8b4a76b8b4567.000000000000017a:head have 2197'242 flags = none tried to add 2197'242 flags = none -38> 2020-03-18 18:39:09.483423 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44bbb34:::rbd_data.8922074b0dc51.00000000000098a5:head have 3099'543 flags = delete tried to add 3099'543 flags = delete -37> 2020-03-18 18:39:09.483438 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44be196:::rbd_data.cd8e219495cff.00000000000192a0:head have 5169'1101 flags = delete tried to add 5169'1101 flags = delete -36> 2020-03-18 18:39:09.483454 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44cd1c7:::rbd_data.5f15f625558ec.000000000000820b:head have 1915'215 flags = none tried to add 1915'215 flags = none -35> 2020-03-18 18:39:09.483469 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44e04c0:::rbd_data.cd66c46e87ccd.0000000000015aa4:head have 5175'1280 flags = none tried to add 5175'1280 flags = none -34> 2020-03-18 18:39:09.483483 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44fa767:::rbd_data.ccf5e625558ec.00000000000150a0:head have 5166'844 flags = delete tried to add 5166'844 flags = delete -33> 2020-03-18 18:39:09.483497 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4518cb0:::rbd_data.589572ae8944a.0000000000000433:head have 1805'174 flags = delete tried to add 1805'174 flags = delete -32> 2020-03-18 18:39:09.483511 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4539e4a:::rbd_data.5f15f625558ec.000000000000031a:head have 1915'216 flags = none tried to add 1915'216 flags = none -31> 2020-03-18 18:39:09.483525 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4540dfd:::rbd_data.cd66c46e87ccd.000000000001d6a6:head have 5175'1283 flags = none tried to add 5175'1283 flags = none -30> 2020-03-18 18:39:09.483539 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45474ee:::rbd_data.2fa7e6b8b4567.000000000000003a:head have 1915'217 flags = none tried to add 1915'217 flags = none -29> 2020-03-18 18:39:09.483553 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45506f1:::rbd_data.9177446e87ccd.00000000000036c3:head have 5175'1284 flags = none tried to add 5175'1284 flags = none -28> 2020-03-18 18:39:09.483567 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4576e29:::rbd_data.589572ae8944a.0000000000001207:head have 1805'179 flags = delete tried to add 1805'179 flags = delete -27> 2020-03-18 18:39:09.483582 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4578057:::rbd_data.ccf5e625558ec.00000000000096a5:head have 5166'785 flags = delete tried to add 5166'785 flags = delete -26> 2020-03-18 18:39:09.483595 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b457fe94:::rbd_data.589572ae8944a.0000000000004e27:head have 1805'200 flags = delete tried to add 1805'200 flags = delete -25> 2020-03-18 18:39:09.483610 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4592563:::rbd_data.9117a327b23c6.000000000000b2a0:head have 3894'562 flags = none tried to add 3894'562 flags = none -24> 2020-03-18 18:39:09.483625 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45bb3ff:::rbd_data.c6c99507ed7ab.000000000000aea0:head have 5172'1141 flags = none tried to add 5172'1141 flags = none -23> 2020-03-18 18:39:09.483639 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45d7c66:::rbd_data.110706b8b4567.0000000000000e07:head have 1915'218 flags = none tried to add 1915'218 flags = none -22> 2020-03-18 18:39:09.483654 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45e1a62:::rbd_data.589572ae8944a.0000000000004733:head have 1805'198 flags = delete tried to add 1805'198 flags = delete -21> 2020-03-18 18:39:09.483667 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45f03a3:::rbd_data.914c22ae8944a.00000000000058a8:head have 3908'593 flags = delete tried to add 3908'593 flags = delete -20> 2020-03-18 18:39:09.483681 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4604fdd:::rbd_data.c6c99507ed7ab.00000000000004ca:head have 5176'1523 flags = none tried to add 5176'1523 flags = none -19> 2020-03-18 18:39:09.483695 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4636d5b:::rbd_data.cd282238e1f29.000000000000b6a1:head have 5165'735 flags = none tried to add 5165'735 flags = none -18> 2020-03-18 18:39:09.483710 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b463cd0a:::rbd_data.9117a327b23c6.000000000000c0a8:head have 3908'563 flags = none tried to add 3908'563 flags = none -17> 2020-03-18 18:39:09.483724 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b468ca37:::rbd_data.9177446e87ccd.000000000000040a:head have 4427'730 flags = none tried to add 4427'730 flags = none -16> 2020-03-18 18:39:09.483738 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4696438:::rbd_data.9177446e87ccd.0000000000000c53:head have 4743'732 flags = none tried to add 4743'732 flags = none -15> 2020-03-18 18:39:09.483752 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46be061:::rbd_data.c70aa2eb141f2.0000000000001ea6:head have 5175'1287 flags = none tried to add 5175'1287 flags = none -14> 2020-03-18 18:39:09.483766 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46bfc4e:::rbd_data.cda772ae8944a.00000000000020a4:head have 5172'1144 flags = none tried to add 5172'1144 flags = none -13> 2020-03-18 18:39:09.483782 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46e7efd:::rbd_data.110706b8b4567.0000000000001c01:head have 1915'219 flags = none tried to add 1915'219 flags = none -12> 2020-03-18 18:39:09.483796 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46ece7b:::rbd_data.8ff1766334873.00000000000000ae:head have 3092'344 flags = delete tried to add 3092'344 flags = delete -11> 2020-03-18 18:39:09.483811 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4786571:::rbd_data.8f6b3643c9869.00000000000040a0:head have 3092'381 flags = delete tried to add 3092'381 flags = delete -10> 2020-03-18 18:39:09.483825 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47888b8:::rbd_data.589572ae8944a.0000000000004534:head have 1805'197 flags = delete tried to add 1805'197 flags = delete -9> 2020-03-18 18:39:09.483840 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b478b833:::rbd_data.589572ae8944a.00000000000048ff:head have 1805'199 flags = delete tried to add 1805'199 flags = delete -8> 2020-03-18 18:39:09.483854 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47ec645:::rbd_data.8b4a76b8b4567.00000000000001d2:head have 2197'271 flags = none tried to add 2197'271 flags = none -7> 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none -6> 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none -5> 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete -4> 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -3> 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -2> 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -1> 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 0> 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** in thread 201e35fdb40 thread_name:tp_osd_tp ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (()+0x145882c) [0x2000245882c] 2: (()+0x19890) [0x2000c54b890] 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] 19: (()+0x80fc) [0x2000c53a0fc] 20: (()+0x119854) [0x2000f2ad854] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. huxiaoyu(a)horebdata.cn

1 0

New Ceph Cluster Setup
by adhobale8＠gmail.com 18 Mar '20

18 Mar '20

Dear Team, We are creating new ceph cluster with three OSD node each node will have 38TB disk space, unfortunately we are having two servers with us third server will get delivered after 3 weeks. I need your help to plan the cluster setup, please suggest which approach will be the right to initiate the setup. 1) Will create the ceph cluster with two osd node by doing "osd pool default size = 2" after that we can add the third node in live cluster and change the replication factor of pool from 2 to 3 by doing # "ceph osd pool set <Pool> size 3". 2) Can we create ceph cluster with 2 osd node by keeping default replication factor 3, and add third osd node in cluster whenever we receive it ?

2 1

bluefs enospc
by Derek Yarnell 18 Mar '20

18 Mar '20

Hi, We have a production cluster that just suffered an issue with multiple of our NVMe OSDs. Multiple of them died (>12) with errors that they no longer had space with a 'ENOSPC from bluestore, misconfigured cluster' error over 4 nodes. These are all simple one device bluestore osds. ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) This is an example[0] of one of the logs. In this case each of 8 NVMe OSDs on a node have 106GB of space allocated to each bluestore NVMe OSD. The ceph-bluestore-tool bluefs-bdev-sizes output only lists 22GiB for osd 681. I extended the space of bluestore on a few of the OSDs via LVM and then the bluefs-bdev-expand command. This worked for a few and not for others. Some of the ones that it did work for recovered for a bit then re-entered the error state. Trying to extend the allocation didn't work after that. When they failed again I ran the fsck which reported that it found 1 error and then running repair I got a rather long stack trace[1]. # ceph-bluestore-tool --log-level 30 --command bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-681 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-681/block -> /dev/dm-33 1 : device size 0x1a80000000 : own 0x[2480000~10000,24a0000~10000,2520000~60000,25f0000~c0000,2720000~50000,28a0000~110000,2a20000~230000,2cc0000~260000,2f30000~220000,31c0000~6b0000,38a0000~10000,3990000~3e0000,3d80000~530000,42d0000~590000,48d0000~400000,4d00000~7d0000,54f0000~c50000,6150000~10000,6190000~150000,6350000~c0000,6480000~160000,6640000~1e0000,6870000~c0000,6a00000~30000,6a40000~240000,6dd0000~310000,7210000~b0000,73a0000~b0000,76a0000~180000,7830000~80000,78e0000~240000,7b70000~90000,7c50000~b0000,7ef0000~140000,8040000~30000,8180000~250000,8440000~50000,84b0000~110000,8610000~c0000,9e20000~20000,9e50000~b0000,9f10000~60000,9f80000~30000,dd80000~180000,df70000~6a0000,e620000~5ae0000,15200000~3510000,187f0000~bf0000,19490000~1070000,1ab70000~4c0000,1b400000~7d0000,1bbe0000~c20000,1cd10000~340000,1d3a0000~860000,1dd00000~2e00000,20c00000~3f00000,24d00000~700000,25600000~700000,26100000~200000,26400000~300000,26b00000~600000,27400000~400000,27ba0000~6e0000,28500000~1d00000,2a400000~700000,2ac00000~100000,2b100000~300000,2b470000~120000,2b700000~500000,2c000000~200000,2c400000~400000,2ca00000~100000,2cf00000~300000,2d340000~39b0000,30d00000~1f00000,32e00000~4bf0000,380a0000~3c0000,38500000~c0000,38bd0000~400000,390b0000~340000,39400000~100000,39900000~1000000,3ac00000~5d00000,40b90000~400000,41280000~db50000,4ee00000~700000,4f900000~4500000,54390000~100000,54e00000~18400000,6d800000~20d0000,6f8f0000~1a10000,71400000~4500000,76100000~300000,766e0000~6860000,7dd00000~c00000,7eac0000~a0000,7ef90000~f190000,8e1f0000~80000,8e410000~60000,8e480000~20000,8e4b0000~20000,8e5c0000~50000,8e7e0000~50000,8f160000~60000,8f240000~a0000,90000000~15e90000,a6200000~c3a0000,b25d0000~630000,b3000000~c00000,b3ee0000~90000,b4200000~d00000,b5a70000~160000,b63f0000~2a0000,b6720000~2820000,bab00000~400000,bbf60000~10ad0000,ccb90000~2300000,cf000000~2b00000,d1ca0000~10000,d1e00000~1400000,d3230000~1df0000,d5200000~1a00000,d6d00000~800000,d75e0000~6f0000,d7f00000~d00000,d9100000~400000,d9900000~d00000,da800000~600000,daf10000~400000,db700000~1600000,dd280000~20000,dd670000~390000,dda30000~400000,de190000~70000,de2a0000~370000,de660000~20000,de700000~14770000,f3600000~700000,f3db0000~960000,f49e0000~5b00000,fa600000~c00000,fb300000~510000,fbb00000~100000,fbeb0000~450000,fc400000~2b0000,fd400000~400000,fde00000~c00000,ff0b0000~50000,ff200000~800000,ffd60000~10000,fff00000~a0000,100200000~300000,101600000~100000,101750000~300000,102120000~1e0000,1027f0000~a00000,103600000~330000,103b00000~200000,103e60000~4a0000,104310000~c00000,105030000~1200000,106800000~100000,106b20000~400000,107000000~300000,1073e0000~400000,107950000~86b0000,110140000~d0000,110350000~2e0000,110e20000~20000,110eb0000~a0000,110f60000~60000,110fd0000~1f0000,1112a0000~f0000,111420000~30000,1115b0000~30000,111620000~150000,111790000~40000,112560000~180000,112730000~180000,1129b0000~50000,112f90000~4a0000,113840000~c0000,113ea0000~40000,113fb0000~130000,114100000~310000,114470000~10000,114620000~120000,114810000~120000,114a00000~20000,114a90000~f0000,114c60000~e0000,114e80000~20000,114f70000~140000,1150c0000~50000,1151f0000~320000,1155f0000~10000,115670000~226f0000,137e30000~800000,138b50000~400000,139400000~1500000,13ae00000~4500000,13f480000~400000,13f950000~1b6e0000,15b700000~400000,15c000000~300000,15c600000~700000,15d9e0000~1820000,15f400000~c00000,160400000~d00000,1613a0000~630000,1619e0000~d20000,162800000~1b00000,164600000~7550000,170d00000~1800000,172580000~100000,172d70000~1c190000,18f100000~40e0000,193700000~400000,193c90000~6970000,19aa90000~188d0000,1b3770000~c220000,1bff00000~1200000,1c12f0000~400000,1c2c00000~400000,1c3f80000~400000,1c5300000~4200000,1c95c0000~b9a40000,283800000~1800000,289000000~3800000,28d000000~6000000,293800000~2000000,296000000~e000000,2a4800000~465d0000,4b6100000~4c560000,502800000~32500000,cb8500000~10f600000,1139800000~7a000000,1668d30000~3d290000,1794000000~3e800000,191c000000~3ee00000,1a49800000~4110000,1a4f800000~40f0000,1a70800000~4100000,1a76000000~3c10000] = 0x582550000 : using 0x56d090000(22 GiB) Any help here would be appreciated, I have stopped out CephFS file system but our radosgw is also impacted. [0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.681.log [1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.repair Thanks, derek -- Derek T. Yarnell Director of Computing Facilities University of Maryland Institute for Advanced Computer Studies

2 6