- ceph-users - lists.ceph.io

How to recover from active+clean+inconsistent+failed_repair?
by Sagara Wijetunga 03 Nov '20

03 Nov '20

Hi all I have a Ceph cluster (Nautilus 14.2.11) with 3 Ceph nodes. A crash happened and all 3 Ceph nodes went down. One (1) PG turned "active+clean+inconsistent", I tried to repair it. After the repair, now shows "active+clean+inconsistent+failed_repair" for the PG in the question and cannot bring the cluster to "active+clean". How do I rescue the cluster? Is this a false positive? Here are the detail: All three Ceph nodes run ceph-mon, ceph-mgr, ceph-osd and ceph-mds. 1. ceph -s health: HEALTH_ERR 3 scrub errors Possible data damage: 1 pg inconsistent pgs: 191 active+clean 1 active+clean+inconsistent 2. ceph health detailHEALTH_ERR 3 scrub errors; Possible data damage: 1 pg inconsistentOSD_SCRUB_ERRORS 3 scrub errorsPG_DAMAGED Possible data damage: 1 pg inconsistent pg 3.b is active+clean+inconsistent, acting [0,1,2] 3. rados list-inconsistent-pg rbd[] 4. ceph pg deep-scrub 3.b 5. ceph pg repair 3.b 6. ceph health detailHEALTH_ERR 3 scrub errors; Possible data damage: 1 pg inconsistentOSD_SCRUB_ERRORS 3 scrub errorsPG_DAMAGED Possible data damage: 1 pg inconsistent pg 3.b is active+clean+inconsistent+failed_repair, acting [0,1,2] 7. rados list-inconsistent-obj 3.b --format=json-pretty{ "epoch": 4769, "inconsistents": []} 8. ceph pg 3.b list_unfound { "num_missing": 0, "num_unfound": 0, "objects": [], "more": false} Appreciate your help. ThanksSagara

2 17

Monitor persistently out-of-quorum
by Ki Wong 02 Nov '20

02 Nov '20

Hello, I am at my wit's end. So I made a mistake in the configuration of my router and one of the monitors (out of 3) dropped out of the quorum and nothing I’ve done allow it to rejoin. That includes reinstalling the monitor with ceph-ansible. The connectivity issue is fixed. I’ve tested it using “nc” and the host can connect to both port 3300 and 6789 of the other monitors. But the wayward monitor continue to stay out of quorum. What is wrong? I see a bunch of “EBUSY” errors in the log, with the message: e1 handle_auth_request haven't formed initial quorum, EBUSY How do I fix this? Any help would be greatly appreciated. Many thanks, -kc With debug_mon at 1/10, I got these log snippets: 2020-10-28 15:40:05.961 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x564050353ec0 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x564050347ce0 after 2 seconds 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:07.961 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x564050347ce0 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x564050360660 after 2 seconds 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:09.107 7fb79253a700 -1 mon.mgmt03@2(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 1 at 2020-10-27 23:03:41.586915) 2020-10-28 15:40:09.961 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x564050360660 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x5640503606c0 after 2 seconds 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:11.962 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x5640503606c0 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x564050360900 after 2 seconds 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:12.354 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404cf25400 (start) method 2 payload 32 2020-10-28 15:40:12.354 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:40:12.354 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404cf25400 - … 2020-10-28 15:40:59.968 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:41:00.110 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404cae1000 (start) method 2 payload 22 2020-10-28 15:41:00.110 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:41:00.110 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404cae1000 - 2020-10-28 15:41:00.110 7fb793d3d700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404c8d4c00 (start) method 2 payload 22 2020-10-28 15:41:00.110 7fb793d3d700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:41:00.110 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404c8d4c00 - 2020-10-28 15:41:00.117 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404c630800 (start) method 2 payload 22 2020-10-28 15:41:00.117 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:41:00.117 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404c630800 - … 2020-10-28 15:42:42.379 7fb78d530700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2020-10-28 15:42:42.379 7fb78d530700 4 rocksdb: [db/db_impl.cc:778] ** DB Stats ** Uptime(secs): 60000.0 total, 600.0 interval Cumulative writes: 6 writes, 7 keys, 6 commit groups, 0.9 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 6 writes, 6 syncs, 0.86 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 3.02 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0.00 0.00 1 0.001 0 0 Sum 2/0 3.02 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.7 0.00 0.00 1 0.001 0 0 Uptime(secs): 60000.0 total, 600.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count … 2020-10-28 17:17:11.781 7f5f694821c0 0 using public_addr v2:10.2.1.1:0/0 -> [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] 2020-10-28 17:17:11.781 7f5f694821c0 0 starting mon.mgmt03 rank -1 at public addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] at bind addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] mon_data /var/lib/ceph/mon/ceph-mgmt03 fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:11.783 7f5f694821c0 1 mon.mgmt03@-1(???) e2 preinit fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:11.783 7f5f694821c0 1 mon.mgmt03@-1(???) e2 initial_members mgmt01,mgmt02,mgmt03, filtering seed monmap 2020-10-28 17:17:11.783 7f5f694821c0 1 mon.mgmt03@-1(???) e2 preinit clean up potentially inconsistent store state 2020-10-28 17:17:11.785 7f5f694821c0 0 -- [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] send_to message mon_probe(probe 374aed9e-5fc1-47e1-8d29-4416f7425e76 name mgmt03 new mon_release 14) v7 with empty dest 2020-10-28 17:17:13.191 7f5f5170d700 0 mon.mgmt03@-1(probing) e3 monmap addrs for rank 2 changed, i am [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0], monmap is v2:10.2.1.1:3300/0, respawning 2020-10-28 17:17:13.191 7f5f5170d700 -1 mon.mgmt03@-1(probing) e3 stashing newest monmap 3 for next startup 2020-10-28 17:17:13.191 7f5f5170d700 0 mon.mgmt03@-1(probing) e3 respawn 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 e: '/usr/bin/ceph-mon' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 0: '/usr/bin/ceph-mon' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 1: '-f' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 2: '--cluster' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 3: 'ceph' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 4: '--id' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 5: 'mgmt03' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 6: '--setuser' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 7: 'ceph' 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 8: '--setgroup' 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 9: 'ceph' 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 respawning with exe /usr/bin/ceph-mon 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 exe_path /proc/self/exe 2020-10-28 17:17:13.217 7eff1f7cd1c0 0 ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable), process ceph-mon, pid 24265 2020-10-28 17:17:13.217 7eff1f7cd1c0 0 pidfile_write: ignore empty --pid-file 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 load: jerasure load: lrc load: isa 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option compression = kNoCompression 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option write_buffer_size = 33554432 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option compression = kNoCompression 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option write_buffer_size = 33554432 2020-10-28 17:17:13.246 7eff1f7cd1c0 1 rocksdb: do_open column families: [default] 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: RocksDB version: 6.1.2 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: Compile date Aug 13 2020 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: DB SUMMARY 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: CURRENT file: CURRENT 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: IDENTITY file: IDENTITY 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: MANIFEST file: MANIFEST-009877 size: 194 Bytes 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-mgmt03/store.db dir, Total Num: 2, files: 009874.sst 009876.sst 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-mgmt03/store.db: 009878.log size: 451 ; 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.error_if_exists: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.create_if_missing: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.paranoid_checks: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.env: 0x55beb27a5780 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.info_log: 0x55beb3d8e300 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_file_opening_threads: 16 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.statistics: (nil) 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_fsync: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_log_file_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.log_file_time_to_roll: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.keep_log_file_num: 1000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.recycle_log_file_num: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_fallocate: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_mmap_reads: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_mmap_writes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_direct_reads: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.create_missing_column_families: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.db_log_dir: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-mgmt03/store.db 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.table_cache_numshardbits: 6 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_subcompactions: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_background_flushes: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.WAL_ttl_seconds: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.WAL_size_limit_MB: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.is_fd_close_on_exec: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.advise_random_on_open: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.db_write_buffer_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_buffer_manager: 0x55beb3d98b40 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_adaptive_mutex: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.rate_limiter: (nil) 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_recovery_mode: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.enable_thread_tracking: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.enable_pipelined_write: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.row_cache: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_filter: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_ingest_behind: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.preserve_deletes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.two_write_queues: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.manual_wal_flush: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.atomic_flush: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_background_jobs: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_background_compactions: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.delayed_write_rate : 16777216 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_total_wal_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.stats_dump_period_sec: 600 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.stats_persist_period_sec: 600 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_open_files: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bytes_per_sync: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_bytes_per_sync: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_readahead_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Compression algorithms supported: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kZSTD supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kXpressCompression supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kLZ4HCCompression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kLZ4Compression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kBZip2Compression supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kZlibCompression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kSnappyCompression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Fast CRC32 supported: Supported on x86 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3543] Recovering from manifest file: /var/lib/ceph/mon/ceph-mgmt03/store.db/MANIFEST-009877 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: [db/column_family.cc:477] --------------- Options for column family [default]: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.merge_operator: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_filter: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_filter_factory: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_factory: SkipListFactory 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.table_factory: BlockBasedTable 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55beb30daac0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55beb30f7010 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_buffer_size: 33554432 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_write_buffer_number: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression: NoCompression 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression: Disabled 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.prefix_extractor: nullptr 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.num_levels: 7 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.window_bits: -14 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.level: 32767 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.strategy: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.enabled: false 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.target_file_size_base: 67108864 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.target_file_size_multiplier: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.arena_block_size: 4194304 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.disable_auto_compactions: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.table_properties_collectors: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.inplace_update_support: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.inplace_update_num_locks: 10000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_huge_page_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bloom_locality: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_successive_merges: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.optimize_filters_for_hits: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.paranoid_file_checks: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.force_consistency_checks: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.report_bg_io_stats: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.ttl: 0 2020-10-28 17:17:13.248 7eff1f7cd1c0 3 rocksdb: [db/version_set.cc:2581] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:/var/lib/ceph/mon/ceph-mgmt03/store.db/MANIFEST-009877 succeeded,manifest_file_number is 9877, next_file_number is 9879, last_sequence is 2479, log_number is 9872,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 9872 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1603930633249084, "job": 1, "event": "recovery_started", "log_files": [9878]} 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #9878 mode 2 2020-10-28 17:17:13.249 7eff1f7cd1c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1603930633250074, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 9879, "file_size": 1336, "table_properties": {"data_size": 434, "index_size": 28, "filter_size": 69, "raw_key_size": 34, "raw_average_key_size": 34, "raw_value_size": 383, "raw_average_value_size": 383, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2020-10-28 17:17:13.249 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3036] Creating manifest 9880 2020-10-28 17:17:13.249 7eff1f7cd1c0 3 rocksdb: [db/version_set.cc:2581] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2020-10-28 17:17:13.250 7eff1f7cd1c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1603930633251509, "job": 1, "event": "recovery_finished"} 2020-10-28 17:17:13.253 7eff1f7cd1c0 4 rocksdb: DB pointer 0x55beb3d26400 2020-10-28 17:17:13.253 7eff05253700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2020-10-28 17:17:13.253 7eff05253700 4 rocksdb: [db/db_impl.cc:778] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 2.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 L6 1/0 2.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 5.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 2.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 L6 1/0 2.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 5.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2020-10-28 17:17:13.253 7eff1f7cd1c0 0 mon.mgmt03 does not exist in monmap, will attempt to join an existing cluster 2020-10-28 17:17:13.254 7eff1f7cd1c0 0 using public_addr v2:10.2.1.1:0/0 -> [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] 2020-10-28 17:17:13.254 7eff1f7cd1c0 0 starting mon.mgmt03 rank -1 at public addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] at bind addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] mon_data /var/lib/ceph/mon/ceph-mgmt03 fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:13.256 7eff1f7cd1c0 1 mon.mgmt03@-1(???) e2 preinit fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:13.256 7eff1f7cd1c0 1 mon.mgmt03@-1(???) e2 initial_members mgmt01,mgmt02,mgmt03, filtering seed monmap 2020-10-28 17:17:13.256 7eff1f7cd1c0 1 mon.mgmt03@-1(???) e2 preinit clean up potentially inconsistent store state 2020-10-28 17:17:13.258 7eff1f7cd1c0 0 -- [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] send_to message mon_probe(probe 374aed9e-5fc1-47e1-8d29-4416f7425e76 name mgmt03 new mon_release 14) v7 with empty dest

3 4

read latency
by Tony Liu 02 Nov '20

02 Nov '20

Hi, AWIK, the read latency primarily depends on HW latency, not much can be tuned in SW. Is that right? I ran a fio random read with iodepth 1 within a VM backed by Ceph with HDD OSD and here is what I got. ================= read: IOPS=282, BW=1130KiB/s (1157kB/s)(33.1MiB/30001msec) slat (usec): min=4, max=181, avg=14.04, stdev=10.16 clat (usec): min=178, max=393831, avg=3521.86, stdev=5771.35 lat (usec): min=188, max=393858, avg=3536.38, stdev=5771.51 ================= I checked HDD average latency is 2.9 ms. Looks like the test result makes perfect sense, isn't it? If I want to get shorter latency (more IOPS), I will have to go for better disk, eg. SSD. Right? Thanks! Tony

2 4

v14.2.13 Nautilus released
by Abhishek Lekshmanan 02 Nov '20

02 Nov '20

This is the 13th backport release in the Nautilus series. This release fixes a regression introduced in v14.2.12, and a few ceph-volume & RGW fixes. We recommend users to update to this release. Notable Changes --------------- * Fixed a regression that caused breakage in clusters that referred to ceph-mon hosts using dns names instead of ip addresses in the `mon_host` param in `ceph.conf` (issue#47951) * ceph-volume: the ``lvm batch`` subcommand received a major rewrite Changelog --------- * ceph-volume: major batch refactor (pr#37522, Jan Fajerski) * mgr/dashboard: Proper format iSCSI target portals (pr#37060, Volker Theile) * rpm: move python-enum34 into rhel 7 conditional (pr#37747, Nathan Cutler) * mon/MonMap: fix unconditional failure for init_with_hosts (pr#37816, Nathan Cutler, Patrick Donnelly) * rgw: allow rgw-orphan-list to note when rados objects are in namespace (pr#37799, J. Eric Ivancich) * rgw: fix setting of namespace in ordered and unordered bucket listing (pr#37798, J. Eric Ivancich) -- Abhishek

1 0

Re: Seriously degraded performance after update to Octopus
by Vladimir Prokofev 02 Nov '20

02 Nov '20

Just shooting in the dark here, but you may be affected by similar issue I had a while back, it was discussed here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZOPBOY6XQOY… In short - they've changed setting bluefs_buffered_io to false in the recent Nautilus release. I guess the same was applied to newer releases. That lead to severe performance issues and similar symptoms, i.e. lower memory usage on OSD nodes. Worth checking out. Of course, it may be something completely different. You should look into monitoring all your OSDs separately, checking their utilization, await, and other parameters, at the same time comparing them to pre-upgrade values, to find the root cause. пн, 2 нояб. 2020 г. в 11:55, Marc Roos <M.Roos(a)f1-outsourcing.eu>: > > I am advocating already a long time for publishing testing data of some > basic test cluster against different ceph releases. Just a basic ceph > cluster that covers most configs and run the same tests, so you can > compare just ceph performance. That would mean a lot for smaller > companies that do not have access to a good test environment. I have > asked also about this at some ceph seminar. > > > > -----Original Message----- > From: Martin Rasmus Lundquist Hansen [mailto:hansen@imada.sdu.dk] > Sent: Monday, November 02, 2020 7:53 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Seriously degraded performance after update to > Octopus > > Two weeks ago we updated our Ceph cluster from Nautilus (14.2.0) to > Octopus (15.2.5), an update that was long overdue. We used the Ansible > playbooks to perform a rolling update and except from a few minor > problems with the Ansible code, the update went well. The Ansible > playbooks were also used for setting up the cluster in the first place. > Before updating the Ceph software we also performed a full update of > CentOS and the Linux kernel (this part of the update had already been > tested on one of the OSD nodes the week before and we didn't notice any > problems). > > However, after the update we are seeing a serious decrease in > performance, more than a factor of 10x in some cases. I spend a week > trying to come up with an explantion or solution, but I am completely > blank. Independently of Ceph I tested the network performance and the > performance of the OSD disks, and I am not really seeing any problems > here. > > The specifications of the cluster is: > - 3x Monitor nodes running mgr+mon+mds (Intel(R) Xeon(R) Silver 4108 CPU > @ 1.80GHz, 16 cores, 196 GB RAM) > - 14x OSD nodes, each with 18 HDDs and 1 NVME (Intel(R) Xeon(R) Gold > 6126 CPU @ 2.60GHz, 24 cores, 384 GB RAM) > - CentOS 7.8 and Kernel 5.4.51 > - 100 Gbps Infiniband > > We are collecting various metrics using Prometheus, and on the OSD nodes > we are seeing some clear differences when it comes to CPU and Memory > usage. I collected some graphs here: http://mitsted.dk/ceph . After the > update the system load is highly reduced, there is almost no longer any > iowait for the CPU, and the free memory is no longer used for Buffers (I > can confirm that the changes in these metrics are not due to the update > of CentOS or the Linux kernel). All in all, now the OSD nodes are almost > completely idle all the time (and so are the monitors). On the linked > page I also attached two RADOS benchmarks. The first benchmark was > performed when the cluster was initially configured, and the second is > the same benchmark after the update to Octopus. When comparing these > two, it is clear that the performance has changed dramatically. For > example, in the write test the bandwidth is reduced from 320 MB/s to 21 > MB/s and the number of IOPS has also dropped significantly. > > I temporarily tried to disable the firewall and SELinux on all nodes to > see if it made any difference, but it didnt look like it (I did not > restart any services during this test, I am not sure if that could be > necessary). > > Any suggestions for finding the root cause of this performance decrease > would be greatly appreciated. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

cephfs cannot write
by Patrick 02 Nov '20

02 Nov '20

Hi all, My ceph cluster is HEALTH_OK, but I cannot write on cephfs. OS: Ubuntu 20.04, ceph version 15.2.5, deploy with cephadm. root@RK01-OSD-A001:~# ceph -s   cluster:     id:     9091b472-1bdb-11eb-b217-abff3468259e     health: HEALTH_OK     services:     mon: 3 daemons, quorum RK01-OSD-A001,RK02-OSD-A002,RK03-OSD-A003 (age 18s)     mgr: RK01-OSD-A001.jwrjgj(active, since 51m), standbys: RK03-OSD-A003.tulrii     mds: cephfs:1 {0=cephfs.RK02-OSD-A002.lwpgaw=up:active} 1 up:standby     osd: 6 osds: 6 up (since 44m), 6 in (since 44m)     task status:     scrub status:         mds.cephfs.RK02-OSD-A002.lwpgaw: idle     data:     pools:   3 pools, 65 pgs     objects: 24 objects, 67 KiB     usage:   6.0 GiB used, 44 TiB / 44 TiB avail     pgs:     65 active+clean root@RK01-OSD-A001:~# ceph fs status cephfs - 1 clients ====== RANK  STATE               MDS                 ACTIVITY     DNS    INOS    0    active  cephfs.RK02-OSD-A002.lwpgaw  Reqs:    0 /s    13     15           POOL           TYPE     USED  AVAIL   cephfs.cephfs.meta  metadata  1152k  20.7T   cephfs.cephfs.data    data       0   20.7T           STANDBY MDS           cephfs.RK03-OSD-A003.xchwqj   MDS version: ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable) root@RK05-FRP-A001:~# df -h|grep "ceph-test" 172.16.65.1,172.16.65.2,172.16.65.3:6789:/   21T     0   21T   0% /ceph-test root@RK05-FRP-A001:~# echo 123 > /ceph-test/1.txt -bash: echo: write error: Operation not permitted root@RK05-FRP-A001:~# ls -l /ceph-test/1.txt -rw-r--r-- 1 root root 0 Nov  1 09:40 /ceph-test/1.txt root@RK05-FRP-A001:~# ls -ld /ceph-test/ drwxr-xr-x 2 root root 1 Nov  1 09:40 /ceph-test/ root@RK01-OSD-A001:~# cd /var/log/ceph/`ceph fsid` root@RK01-OSD-A001:/var/log/ceph/9091b472-1bdb-11eb-b217-abff3468259e# cat ceph-volume.log | grep err | grep sdx [2020-11-01 08:53:51,384][ceph_volume.process][INFO  ] stderr Failed to find physical volume "/dev/sdx". [2020-11-01 08:53:51,417][ceph_volume.process][INFO  ] stderr unable to read label for /dev/sdx: (2) No such file or directory [2020-11-01 08:53:51,445][ceph_volume.process][INFO  ] stderr unable to read label for /dev/sdx: (2) No such file or directory root@RK01-OSD-A001:~# pvs|grep sdx   /dev/sdx   ceph-41b09a52-e44b-43c5-ad86-0eada11b48b6 lvm2 a--  <7.28t    0  root@RK01-OSD-A001:~# lsblk|grep sdx sdx                                                                                                   65:112  0   7.3T  0 disk  root@RK01-OSD-A001:~# parted -s /dev/sdx print Error: /dev/sdx: unrecognised disk label Model: LSI MR9261-8i (scsi) Disk /dev/sdx: 8001GB Sector size (logical/physical): 512B/4096B Partition Table: unknown Disk Flags:  root@RK01-OSD-A001:~# 

2 1

RGW seems to not clean up after some requests
by Denis Krienbühl 02 Nov '20

02 Nov '20

Hi everyone We have faced some RGW outages recently, with the RGW returning HTTP 503. First for a few, then for most, then all requests - in the course of 1-2 hours. This seems to have started since we have updated from 15.2.4 to 15.2.5. The line that accompanies these outages in the log is the following: s3:list_bucket Scheduling request failed with -2218 It first pops up a few times here and there, until it eventually applies to all requests. It seems to indicate that the throttler has reached the limit of open connections. As we run a pair of HAProxy instances in front of RGW, which limit the number of connections to the two RGW instances to 400, this limit should never be reached. We do use RGW metadata sync between the instances, which could account for some extra connections, but if I look at open TCP connections between the instances I can count no more than 20 at any given time. I also noticed that some connections in the RGW log seem to never complete. That is, I can find a ‘starting new request’ line, but no associated ‘req done’ or ‘beast’ line. I don’t think there are any hung connections around, as they are killed by HAProxy after a short timeout. Looking at the code, it seems as if the throttler in use (SimpleThrottler), eventually reaches the maximum count of 1024 connections (outstanding_requests), and never recovers. I believe that the request_complete function is not called in all cases, but I am not familiar with the Ceph codebase, so I am not sure. See https://github.com/ceph/ceph/blob/cc17681b478594aa39dd80437256a54e388432f0/… <https://github.com/ceph/ceph/blob/cc17681b478594aa39dd80437256a54e388432f0/…> Does anyone see the same phenomenon? Could this be a bug in the request handling of RGW, or am I wrong in my assumptions? For now we’re just restarting our RGWs regularly, which seems to keep the problem at bay. Thanks for any hints. Denis

2 2

pgs stuck backfill_toofull
by Mark Johnson 02 Nov '20

02 Nov '20

I've been struggling with this one for a few days now. We had an OSD report as near full a few days ago. Had this happen a couple of times before and a reweight-by-utilization has sorted it out in the past. Tried the same again but this time we ended up with a couple of pgs in a state of backfill_toofull and a handful of misplaced objects as a result. Tried doing the reweight a few more times and it's been moving data around. We did have another osd trigger the near full alert but running the reweight a couple more times seems to have moved some of that data around a bit better. However, the original near_full osd doesn't seem to have changed much and the backfill_toofull pgs are still there. I'd keep doing the reweight-by-utilization but I'm not sure if I'm heading down the right path and if it will eventually sort it out. We have 14 pools, but the vast majority of data resides in just one of those pools (pool 20). The pgs in the backfill state are in pool 2 (as far as I can tell). That particular pool is used for some cephfs stuff and has a handful of large files in there (not sure if this is significant to the problem). All up, our utilization is showing as 55.13% but some of our OSDs are showing as 76% in use with this one problem sitting at 85.02%. Right now, I'm just not sure what the proper corrective action is. The last couple of reweights I've run have been a bit more targetted in that I've set it to only function on two OSDs at a time. If I run a test-reweight targetting only one osd, it does say it will reweight OSD 9 (the one at 85.02%). I gather this will move data away from this OSD and potentially get it below the threshold. However, at one point in the past couple of days, it's shown as no OSDs in a near full state, yet the two pgs in backfill_toofull didn't change. So, that's why I'm not sure continually reweighting is going to solve this issue. I'm a long way from knowledgable on Ceph so I'm not really sure what information is useful here. Here's a bit of info on what I'm seeing. Can provide anything else that might help. Basically, we have a three node cluster but only two have OSDs. The third is there simply to enable a quorum to be established. The OSDs are evenly spread across these two needs and the configuration of each is identical. We are running Jewel and are not in a position to upgrade at this stage. # ceph --version ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e) # ceph health detail HEALTH_WARN 2 pgs backfill_toofull; 2 pgs stuck unclean; recovery 33/62099566 objects misplaced (0.000%); 1 near full osd(s) pg 2.52 is stuck unclean for 201822.031280, current state active+remapped+backfill_toofull, last acting [17,3] pg 2.18 is stuck unclean for 202114.617682, current state active+remapped+backfill_toofull, last acting [18,2] pg 2.18 is active+remapped+backfill_toofull, acting [18,2] pg 2.52 is active+remapped+backfill_toofull, acting [17,3] recovery 33/62099566 objects misplaced (0.000%) osd.9 is near full at 85% # ceph osd df ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 2 1.37790 1.00000 1410G 842G 496G 59.75 1.08 33 3 1.37790 0.45013 1410G 1079G 259G 76.49 1.39 21 4 1.37790 0.95001 1410G 1086G 253G 76.98 1.40 44 5 1.37790 1.00000 1410G 617G 722G 43.74 0.79 43 6 1.37790 0.65009 1410G 616G 722G 43.69 0.79 39 7 1.37790 0.95001 1410G 495G 844G 35.10 0.64 40 8 1.37790 1.00000 1410G 732G 606G 51.93 0.94 52 9 1.37790 0.70007 1410G 1199G 139G 85.02 1.54 37 10 1.37790 1.00000 1410G 611G 727G 43.35 0.79 41 11 1.37790 0.75006 1410G 495G 843G 35.11 0.64 32 0 1.37790 1.00000 1410G 731G 608G 51.82 0.94 43 12 1.37790 1.00000 1410G 851G 487G 60.36 1.09 44 13 1.37790 1.00000 1410G 378G 960G 26.82 0.49 38 14 1.37790 1.00000 1410G 969G 370G 68.68 1.25 37 15 1.37790 1.00000 1410G 724G 614G 51.35 0.93 35 16 1.37790 1.00000 1410G 491G 847G 34.84 0.63 43 17 1.37790 1.00000 1410G 862G 476G 61.16 1.11 50 18 1.37790 0.80005 1410G 1083G 255G 76.78 1.39 26 19 1.37790 0.65009 1410G 963G 375G 68.29 1.24 23 20 1.37790 1.00000 1410G 724G 614G 51.38 0.93 42 TOTAL 28219G 15557G 11227G 55.13 MIN/MAX VAR: 0.49/1.54 STDDEV: 15.57 # ceph pg ls backfill_toofull pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 2.18 9 0 0 18 0 0 3653 3653 active+remapped+backfill_toofull 2020-10-29 05:31:20.429912 610'549153 656:390372 [9,12] 9 [18,2] 18 594'547482 2020-10-25 20:28:39.680744 594'543841 2020-10-21 21:21:33.092868 2.52 15 0 0 15 0 0 4883 4883 active+remapped+backfill_toofull 2020-10-29 05:31:28.277898 652'502085 656:367288 [17,9] 17 [17,3] 17 594'499108 2020-10-26 11:06:48.417825 594'499108 2020-10-26 11:06:48.417825 pool : 17 18 19 11 20 21 12 13 0 14 1 15 2 16 | SUM -------------------------------------------------------------------------------------------------------------------------------- osd.4 3 0 0 0 9 2 0 0 12 1 9 0 7 1 | 44 osd.17 1 0 0 0 7 3 1 0 8 1 17 1 11 0 | 50 osd.18 0 0 0 0 9 0 0 0 4 0 7 0 5 0 | 25 osd.5 0 0 0 2 5 1 1 0 5 0 16 0 11 2 | 43 osd.6 0 1 0 1 5 2 0 0 9 0 13 1 7 0 | 39 osd.19 0 0 1 0 8 2 0 1 2 0 6 0 3 0 | 23 osd.7 0 0 0 0 4 1 1 0 3 0 12 0 19 0 | 40 osd.8 0 1 0 0 6 3 0 2 10 1 13 1 15 0 | 52 osd.9 1 0 2 0 10 2 0 0 4 1 6 1 10 0 | 37 osd.10 0 0 1 1 5 2 0 1 7 0 12 0 11 1 | 41 osd.20 1 0 0 0 6 1 0 1 7 0 8 1 17 0 | 42 osd.11 0 0 0 0 4 1 1 1 5 0 11 0 9 0 | 32 osd.12 0 0 1 1 7 1 0 0 5 1 12 1 14 1 | 44 osd.13 0 2 0 0 3 1 0 0 10 1 11 0 10 0 | 38 osd.0 0 1 0 1 6 3 0 1 7 0 11 0 13 0 | 43 osd.14 1 0 0 0 8 1 1 0 4 1 12 0 9 0 | 37 osd.15 1 0 2 1 6 1 1 0 8 0 7 0 6 2 | 35 osd.2 0 2 1 0 7 2 1 0 7 1 4 1 6 0 | 32 osd.3 0 0 0 0 9 0 0 0 2 0 4 0 5 0 | 20 osd.16 0 1 0 1 4 3 1 1 9 0 9 1 12 1 | 43 -------------------------------------------------------------------------------------------------------------------------------- SUM : 8 8 8 8 128 32 8 8 128 8 200 8 200 8 |

4 8

Fix PGs states
by Ing. Luis Felipe Domínguez Vega 02 Nov '20

02 Nov '20

Hi: I have this ceph status: ----------------------------------------------------------------------------- cluster: id: 039bf268-b5a6-11e9-bbb7-d06726ca4a78 health: HEALTH_WARN noout flag(s) set 1 osds down Reduced data availability: 191 pgs inactive, 2 pgs down, 35 pgs incomplete, 290 pgs stale 5 pgs not deep-scrubbed in time 7 pgs not scrubbed in time 327 slow ops, oldest one blocked for 233398 sec, daemons [osd.12,osd.36,osd.5] have slow ops. services: mon: 1 daemons, quorum fond-beagle (age 23h) mgr: fond-beagle(active, since 7h) osd: 48 osds: 45 up (since 95s), 46 in (since 8h); 4 remapped pgs flags noout data: pools: 7 pools, 2305 pgs objects: 350.37k objects, 1.5 TiB usage: 3.0 TiB used, 38 TiB / 41 TiB avail pgs: 6.681% pgs unknown 1.605% pgs not active 1835 active+clean 279 stale+active+clean 154 unknown 22 incomplete 10 stale+incomplete 2 down 2 remapped+incomplete 1 stale+remapped+incomplete -------------------------------------------------------------------------------------------- How can i fix all of unknown, incomplete, remmaped+incomplete, etc... i dont care if i need remove PGs

5 9

Intel SSD firmware guys contacts, if any
by vitalif＠yourcmc.ru 02 Nov '20

02 Nov '20

Hi! I have an interesting question regarding SSDs and I'll try to ask about it here. During my testing of Ceph & Vitastor & Linstor on servers equipped with Intel D3-4510 SSDs I discovered a very funny problem with these SSDs: They don't like overwrites of the same sector. That is, if you overwrite the same sector over and over again you get very low iops: $ fio -direct=1 -rw=write -bs=4k -size=4k -loops=100000 -iodepth=1 write: IOPS=3142, BW=12.3MiB/s (12.9MB/s)(97.9MiB/7977msec) And if you overwrite at least ~128k of other sectors between overwriting the same sector you get normal results: $ fio -direct=1 -rw=write -bs=4k -size=128k -loops=100000 -iodepth=1 write: IOPS=20.8k, BW=81.4MiB/s (85.3MB/s)(543MiB/6675msec) This slowdown almost doesn't hurt Ceph, slightly hurts Vitastor (the impact was greater before I added a fix), and MASSIVELY hurts Linstor/DRBD9 because of its "bitmap". By now I've only seen it on this particular model of SSD. For example, Intel P4500, Micron 9300 Pro, Samsung PM983 don't have this issue. Do you have any contacts of Intel SSD firmware guys to ask them about this bug-o-feature? :-) -- With best regards, Vitaliy Filippov

1 0