[ceph-users] Problem with OSD - stuck in CPU loop after rbd snapshot mount

3 Feb 2020

Hi all,

I have small cluster and yesterday I tried to mount older RBD snapshot torecover data. (I
have approx. 230 daily snapshots of one RBD image 
on my small ceph).

After I did mount and ls operation, cluster was stuck and I notice that 2of my OSD's
eaten CPU and raise in memory usage (more than 4x more 
memory than usual). Cluster was unusable after that and it didn't help stopping rbd
mount operation and restarting OSD's etc.

Strange is, that it affects only 2 of my OSD's - one is on NVME and has cache PG's
mapped, other one is rotational HDD. IOPS are normal, 
only CPU and memory usage are unusual.

I noticed hundred thousands of slow requests (on idle cluster) so I startto debug OSD
operations. You can find OSD 8 log below. OSD 9 
contains nvme cached PG's and there is nothing unusual in standard log. In the middle
of log I raised debug level to 10.

It look's to me, that it is trying to do one operation with one snapshotted RBD object
in PG 2.b 
(rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3) in a loop.

Is there any way how to stop that? What is causing the problem? Can I somehow flush
OSD's "journals" or "queues" to stop current operation 
and make cluster usable again? Is there any way how I can access old snapshot data?

Problem started on Luminous 12.2.12 and upgrade to 12.2.13 didn't help.

Thank you
Jan Pekar

2020-02-03 15:09:22.470277 7f51f6af7700  0 log_channel(cluster) log [WRN] : 30966 slow
requests, 5 included below; oldest blocked for > 
37.505482 secs
2020-02-03 15:09:22.470336 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.503404 seconds old, received at 2020-02-03 
15:08:44.956422: osd_op(osd.9.59923:2670991 2.b 2.162d8f8b (undecoded) 
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
2020-02-03 15:09:22.470360 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.500695 seconds old, received at 2020-02-03 
15:08:44.959132: osd_op(osd.9.59923:2671023 2.b 2.162d8f8b (undecoded) 
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
2020-02-03 15:09:22.470366 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.498013 seconds old, received at 2020-02-03 
15:08:44.961814: osd_op(osd.9.59923:2671055 2.b 2.162d8f8b (undecoded) 
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
2020-02-03 15:09:22.470371 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.494836 seconds old, received at 2020-02-03 
15:08:44.964990: osd_op(osd.9.59923:2671087 2.b 2.162d8f8b (undecoded) 
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
2020-02-03 15:09:22.470376 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.491950 seconds old, received at 2020-02-03 
15:08:44.967877: osd_op(osd.9.59923:2671119 2.b 2.162d8f8b (undecoded) 
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
...
2020-02-03 15:09:35.537259 7f51ddac5700  0 osd.8 59939 do_command r=0
2020-02-03 15:09:35.588753 7f51ddac5700  0 osd.8 59939 do_command r=0 Set debug_osd to
10/10
2020-02-03 15:09:35.588814 7f51ddac5700  0 log_channel(cluster) log [INF] : Set debug_osd
to 10/10
2020-02-03 15:09:35.597667 7f51f0945700  2 osd.8 59939 ms_handle_reset con 0x5647736ac800
session 0x564785519e00
2020-02-03 15:09:35.599056 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean]  dropping ondisk_read_lock
2020-02-03 15:09:35.599209 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5800 finish
2020-02-03 15:09:35.599250 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d59c0 prio 63
cost 29 latency 22.224501 
osd_op(osd.9.59923:2779946 2.b 2.162d8f8b (undecoded)
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 
e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227]
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 
59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod
0'0 mlcod 0'0 active+clean]
2020-02-03 15:09:35.599293 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] _handle_message: 0x5647873d59c0
2020-02-03 15:09:35.599347 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] do_op osd_op(osd.9.59923:2779946 2.b
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get 
max 8388608] snapc 0=[]
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) v8 may_read -> 
write-ordered flags
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
2020-02-03 15:09:35.599398 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

map_snapid_to_clone=true
2020-02-03 15:09:35.599506 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3
2020-02-03 15:09:35.599606 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] get_object_context: found obc in cache:
0x564785fb3880
2020-02-03 15:09:35.613724 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] get_object_context: 0x564785fb3880
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none 
n=0 w=0) oi: 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046
osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 
uv 7633387 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160
snapset: 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}
2020-02-03 15:09:35.613881 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT
2020-02-03 15:09:35.614024 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] execute_ctx 0x5647843a3000
2020-02-03 15:09:35.614053 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] execute_ctx
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] ov
58682'4215046
2020-02-03 15:09:35.614077 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean]  taking ondisk_read_lock
2020-02-03 15:09:35.614095 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] do_osd_op
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608]
2020-02-03 15:09:35.614112 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max 8388608
2020-02-03 15:09:35.637123 7f51f72f8700 10 osd.8 59939 tick
2020-02-03 15:09:35.637149 7f51f72f8700 10 osd.8 59939 do_waiters -- start
2020-02-03 15:09:35.637152 7f51f72f8700 10 osd.8 59939 do_waiters -- finish
2020-02-03 15:09:35.645761 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean]  dropping ondisk_read_lock
2020-02-03 15:09:35.645887 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d59c0 finish
2020-02-03 15:09:35.645924 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5b80 prio 63
cost 29 latency 22.271117 
osd_op(osd.9.59923:2779947 2.b 2.162d8f8b (undecoded)
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 
e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227]
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 
59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod
0'0 mlcod 0'0 active+clean]
2020-02-03 15:09:35.645960 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] _handle_message: 0x5647873d5b80
2020-02-03 15:09:35.646007 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] do_op osd_op(osd.9.59923:2779947 2.b
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get 
max 8388608] snapc 0=[]
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) v8 may_read -> 
write-ordered flags
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
2020-02-03 15:09:35.646047 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

map_snapid_to_clone=true
2020-02-03 15:09:35.646145 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3
2020-02-03 15:09:35.646239 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] get_object_context: found obc in cache:
0x564785fb3880
2020-02-03 15:09:35.646255 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] get_object_context: 0x564785fb3880
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none 
n=0 w=0) oi: 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046
osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 
uv 7633387 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160
snapset: 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}
2020-02-03 15:09:35.646359 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT
2020-02-03 15:09:35.646469 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] execute_ctx 0x5647843a3000
2020-02-03 15:09:35.646494 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] execute_ctx
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] ov
58682'4215046
2020-02-03 15:09:35.646513 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean]  taking ondisk_read_lock
2020-02-03 15:09:35.646530 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] do_osd_op
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608]
2020-02-03 15:09:35.646547 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max 8388608
2020-02-03 15:09:35.681653 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean]  dropping ondisk_read_lock
2020-02-03 15:09:35.681780 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5b80 finish
2020-02-03 15:09:35.681817 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5d40 prio 63
cost 29 latency 22.306952 
osd_op(osd.9.59923:2779948 2.b 2.162d8f8b (undecoded)
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 
e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227]
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 
59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod
0'0 mlcod 0'0 active+clean]
2020-02-03 15:09:35.681851 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] _handle_message: 0x5647873d5d40
2020-02-03 15:09:35.681898 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] do_op osd_op(osd.9.59923:2779948 2.b
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get 
max 8388608] snapc 0=[]
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) v8 may_read -> 
write-ordered flags
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
2020-02-03 15:09:35.681937 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

map_snapid_to_clone=true
2020-02-03 15:09:35.682035 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3
2020-02-03 15:09:35.682128 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] get_object_context: found obc in cache:
0x564785fb3880
2020-02-03 15:09:35.682144 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] get_object_context: 0x564785fb3880
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none 
n=0 w=0) oi: 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046
osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 
uv 7633387 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160
snapset: 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}
2020-02-03 15:09:35.682245 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset 
c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT
2020-02-03 15:09:35.682356 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] execute_ctx 0x5647843a3000
2020-02-03 15:09:35.682385 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] execute_ctx
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] ov
58682'4215046
2020-02-03 15:09:35.682404 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean]  taking ondisk_read_lock
2020-02-03 15:09:35.682420 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] do_osd_op
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608]
2020-02-03 15:09:35.682437 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max 8388608
2020-02-03 15:09:35.714076 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 
lcod 0'0 mlcod 0'0 active+clean]  dropping ondisk_read_lock
2020-02-03 15:09:35.714196 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5d40 finish

2024

2023

2022

2021

2020

2019

[ceph-users] Problem with OSD - stuck in CPU loop after rbd snapshot mount