[ceph-users] Re: Problem with OSD - stuck in CPU loop after rbd snapshot mount

5 Feb 2020

Hi,

because load to my OSD's stopped approximately after 12 hours and everything returned
to normal, I did another test.

I have chosen another rbd image and some old snapshot and I tried to mount it - now
without cloning it.

rbd-nbd map succeeded, but situation from my first test repeated, only different OSD's
(PG) were affected.

It looks to me, that it is trying to do some operation on one snapshotted object, that
object is in cache so primary OSD that is holding 
object, and OSD that is holding cached object are affected with CPU high usage (loop).

One loop operation with debug=20 is included below. This same operation is executed again
and again etc.. and I don't know how to stop that.

When I stop affected osd, another one which is holding replica of affected object
(affected PG) starts the same bombing of cache and 
consumes CPU and memory. Whenever affected PG is down, nothing unusual is happening in the
cluster.

Flushing object from cache didn't help.

For this time, object mentioned in logs below is first sector of snapshoted RBD image,
which I was trying to map.

Any ideas or hints?

With regards
Jan Pekar

2020-02-05 22:46:53.786394 7ff1c3749700 10 osd.1 60042 dequeue_op 0x561362c80a00 finish
2020-02-05 22:46:53.786434 7ff1c3749700 20 osd.1 op_wq(2) _process 2.c item
PGQueueable(0x561362c80bc0 prio 63 cost 29 e60042) queued
2020-02-05 22:46:53.786444 7ff1c3749700 20 osd.1 op_wq(2) _process 2.c item
PGQueueable(0x561362c80bc0 prio 63 cost 29 e60042) pg 0x561324afc000
2020-02-05 22:46:53.786450 7ff1c3749700 10 osd.1 60042 dequeue_op 0x561362c80bc0 prio 63
cost 29 latency 19.605707 osd_op(osd.9.60026:836317 
2.c 2.4011388c (undecoded)
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e60042) v8 pg pg[2.c( v 
60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112
ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod
60042'3048390 active+clean]
2020-02-05 22:46:53.786474 7ff1c3749700 20 osd.1 60042 share_map osd.9 10.4.4.10:0/1678416
60042
2020-02-05 22:46:53.786482 7ff1c3749700 20 osd.1 60042 should_share_map osd.9
10.4.4.10:0/1678416 60042
2020-02-05 22:46:53.786494 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] _handle_message:
0x561362c80bc0
2020-02-05 22:46:53.786522 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_op: op
osd_op(osd.9.60026:836317 2.c 
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 8388608] snapc
0=[] 
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e60042) v8
2020-02-05 22:46:53.786554 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] op_has_sufficient_caps
session=0x561363328580 pool=2 (rbd ) owner=0 need_read_cap=1 
need_write_cap=0 classes=[] -> yes
2020-02-05 22:46:53.786588 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_op
osd_op(osd.9.60026:836317 2.c 
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 8388608] snapc
0=[] 
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e60042) v8 may_read -> write-ordered flags 
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
2020-02-05 22:46:53.786628 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] find_object_context
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 @7b5 
snapset 
c0f=[c0f,c0c,c09,c06,c03,c00,bfb,bf8,bf5,bf2,bef,bec,be9,be6,be3,be0,bdd,bd8,bd5,bd2,bcf,bcc,bc9,bc6,bc3,bc0,bbd,bba,bb7,bad,ba7,ba1,b9b,b95,b8f,b89,b83,b7d,b77,b71,b6b,b65,b5f,b59,b53,b4d,b47,b41,b3b,b35,b2f,b29,b23,b1d,b17,b0f,b09,b03,afd,af7,af1,aeb,ae5,adf,ad9,ad3,acd,ac7,ac1,abb,ab5,aaf,aa9,aa3,a9d,a97,a91,a8b,a85,a7f,a79,a73,a6d,a67,a61,a5b,a55,a4f,a49,a43,a3d,a37,a31,a2b,a25,a1f,a19,a13,a0d,a07,a01,9fb,9f5,9ef,9e9,9e3,9dd,9d7,9d1,9cb,9c5,9bf,9b9,9b3,9ad,9a7,9a1,99b,995,98f,989,983,97d,977,971,96b,965,95f,959,953,94d,947,941,93b,935,92f,929,923,91d,917,911,90b,905,8ff,8f9,8f3,8ed,8e7,8e1,8db,8d5,8cf,8c9,8c3,8bd,8b7,8b1,8ab,8a5,89f,899,893,88d,887,881,87b,875,86f,869,863,85d,857,851,84b,845,83f,839,833,82d,827,821,81b,815,80f,809,803,7fd,7f7,7f1,7eb,7e5,7df,7d9,7d3,7cd,7c7,7c1,7bb,7b5,7af,7a9,7a3,79d,797,791,78b,785,77f,779,773,76d,767,761,75b,755,74f,749,743,73d,737,731,72b,725,71f,719,713,70d]:{70d=[70d],719=[719,713],71f=[71f],72b=[72b,725],731=[731],73d=[73d,737],743=[743],74f=[74f,749],755=[755],761=[761,75b],767=[767],773=[773,76d],77f=[77f,779],785=[785],78b=[78b],791=[791],79d=[79d,797],7a9=[7a9,7a3],7b5=[7b5,7af],7c1=[7c1,7bb],7cd=[7cd,7c7],7d3=[7d3],7df=[7df,7d9],7eb=[7eb,7e5],7f1=[7f1],7fd=[7fd,7f7],803=[803],80f=[80f,809],81b=[81b,815],827=[827,821],82d=[82d],833=[833],839=[839],83f=[83f],84b=[84b,845],851=[851],857=[857],85d=[85d],863=[863],869=[869],875=[875,86f],87b=[87b],881=[881],88d=[88d,887],899=[899,893],89f=[89f],8a5=[8a5],8ab=[8ab],8b1=[8b1],8bd=[8bd,8b7],8c9=[8c9,8c3],8cf=[8cf],8d5=[8d5],8db=[8db],8e1=[8e1],8ed=[8ed,8e7],8f9=[8f9,8f3],8ff=[8ff],905=[905],911=[911,90b],917=[917],923=[923,91d],92f=[92f,929],935=[935],941=[941,93b],94d=[94d,947],959=[959,953],95f=[95f],96b=[96b,965],971=[971],977=[977],983=[983,97d],989=[989],995=[995,98f],99b=[99b],9a1=[9a1],9ad=[9ad,9a7],9b9=[9b9,9b3],9c5=[9c5,9bf],9cb=[9cb],9d7=[9d7,9d1],9dd=[9dd],9e9=[9e9,9e3],9f5=[9f5,9ef],a01=[a01,9fb],a07=[a07],a0d=[a0d],a19=[a19,a13],a1f=[a1f],a25=[a25],a2b=[a2b],a31=[a31],a37=[a37],a43=[a43,a3d],a49=[a49],a55=[a55,a4f],a5b=[a5b],a61=[a61],a67=[a67],a6d=[a6d],a79=[a79,a73],a7f=[a7f],a8b=[a8b,a85],a91=[a91],a9d=[a9d,a97],aa3=[aa3],aa9=[aa9],aaf=[aaf],abb=[abb,ab5],ac7=[ac7,ac1],acd=[acd],ad9=[ad9,ad3],adf=[adf],ae5=[ae5],af1=[af1,aeb],afd=[afd,af7],b03=[b03],b0f=[b0f,b09],b1d=[b1d,b17],b23=[b23],b2f=[b2f,b29],b35=[b35],b41=[b41,b3b],b47=[b47],b53=[b53,b4d],b59=[b59],b5f=[b5f],b6b=[b6b,b65],b71=[b71],b77=[b77],b7d=[b7d],b83=[b83],b89=[b89],b95=[b95,b8f],b9b=[b9b],ba1=[ba1],ba7=[ba7],bad=[bad],bb7=[bb7],bbd=[bbd,bba],bc3=[bc3,bc0],bc9=[bc9,bc6],bcc=[bcc],bcf=[bcf],bd2=[bd2],bd5=[bd5],bd8=[bd8],be0=[be0,bdd],be6=[be6,be3],bec=[bec,be9],bef=[bef],bf2=[bf2],bf8=[bf8,bf5],c00=[c00,bfb],c03=[c03],c09=[c09,c06],c0c=[c0c],c0f=[c0f]}

map_snapid_to_clone=true
2020-02-05 22:46:53.786790 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] find_object_context
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 @7b5 
snapset 
c0f=[c0f,c0c,c09,c06,c03,c00,bfb,bf8,bf5,bf2,bef,bec,be9,be6,be3,be0,bdd,bd8,bd5,bd2,bcf,bcc,bc9,bc6,bc3,bc0,bbd,bba,bb7,bad,ba7,ba1,b9b,b95,b8f,b89,b83,b7d,b77,b71,b6b,b65,b5f,b59,b53,b4d,b47,b41,b3b,b35,b2f,b29,b23,b1d,b17,b0f,b09,b03,afd,af7,af1,aeb,ae5,adf,ad9,ad3,acd,ac7,ac1,abb,ab5,aaf,aa9,aa3,a9d,a97,a91,a8b,a85,a7f,a79,a73,a6d,a67,a61,a5b,a55,a4f,a49,a43,a3d,a37,a31,a2b,a25,a1f,a19,a13,a0d,a07,a01,9fb,9f5,9ef,9e9,9e3,9dd,9d7,9d1,9cb,9c5,9bf,9b9,9b3,9ad,9a7,9a1,99b,995,98f,989,983,97d,977,971,96b,965,95f,959,953,94d,947,941,93b,935,92f,929,923,91d,917,911,90b,905,8ff,8f9,8f3,8ed,8e7,8e1,8db,8d5,8cf,8c9,8c3,8bd,8b7,8b1,8ab,8a5,89f,899,893,88d,887,881,87b,875,86f,869,863,85d,857,851,84b,845,83f,839,833,82d,827,821,81b,815,80f,809,803,7fd,7f7,7f1,7eb,7e5,7df,7d9,7d3,7cd,7c7,7c1,7bb,7b5,7af,7a9,7a3,79d,797,791,78b,785,77f,779,773,76d,767,761,75b,755,74f,749,743,73d,737,731,72b,725,71f,719,713,70d]:{70d=[70d],719=[719,713],71f=[71f],72b=[72b,725],731=[731],73d=[73d,737],743=[743],74f=[74f,749],755=[755],761=[761,75b],767=[767],773=[773,76d],77f=[77f,779],785=[785],78b=[78b],791=[791],79d=[79d,797],7a9=[7a9,7a3],7b5=[7b5,7af],7c1=[7c1,7bb],7cd=[7cd,7c7],7d3=[7d3],7df=[7df,7d9],7eb=[7eb,7e5],7f1=[7f1],7fd=[7fd,7f7],803=[803],80f=[80f,809],81b=[81b,815],827=[827,821],82d=[82d],833=[833],839=[839],83f=[83f],84b=[84b,845],851=[851],857=[857],85d=[85d],863=[863],869=[869],875=[875,86f],87b=[87b],881=[881],88d=[88d,887],899=[899,893],89f=[89f],8a5=[8a5],8ab=[8ab],8b1=[8b1],8bd=[8bd,8b7],8c9=[8c9,8c3],8cf=[8cf],8d5=[8d5],8db=[8db],8e1=[8e1],8ed=[8ed,8e7],8f9=[8f9,8f3],8ff=[8ff],905=[905],911=[911,90b],917=[917],923=[923,91d],92f=[92f,929],935=[935],941=[941,93b],94d=[94d,947],959=[959,953],95f=[95f],96b=[96b,965],971=[971],977=[977],983=[983,97d],989=[989],995=[995,98f],99b=[99b],9a1=[9a1],9ad=[9ad,9a7],9b9=[9b9,9b3],9c5=[9c5,9bf],9cb=[9cb],9d7=[9d7,9d1],9dd=[9dd],9e9=[9e9,9e3],9f5=[9f5,9ef],a01=[a01,9fb],a07=[a07],a0d=[a0d],a19=[a19,a13],a1f=[a1f],a25=[a25],a2b=[a2b],a31=[a31],a37=[a37],a43=[a43,a3d],a49=[a49],a55=[a55,a4f],a5b=[a5b],a61=[a61],a67=[a67],a6d=[a6d],a79=[a79,a73],a7f=[a7f],a8b=[a8b,a85],a91=[a91],a9d=[a9d,a97],aa3=[aa3],aa9=[aa9],aaf=[aaf],abb=[abb,ab5],ac7=[ac7,ac1],acd=[acd],ad9=[ad9,ad3],adf=[adf],ae5=[ae5],af1=[af1,aeb],afd=[afd,af7],b03=[b03],b0f=[b0f,b09],b1d=[b1d,b17],b23=[b23],b2f=[b2f,b29],b35=[b35],b41=[b41,b3b],b47=[b47],b53=[b53,b4d],b59=[b59],b5f=[b5f],b6b=[b6b,b65],b71=[b71],b77=[b77],b7d=[b7d],b83=[b83],b89=[b89],b95=[b95,b8f],b9b=[b9b],ba1=[ba1],ba7=[ba7],bad=[bad],bb7=[bb7],bbd=[bbd,bba],bc3=[bc3,bc0],bc9=[bc9,bc6],bcc=[bcc],bcf=[bcf],bd2=[bd2],bd5=[bd5],bd8=[bd8],be0=[be0,bdd],be6=[be6,be3],bec=[bec,be9],bef=[bef],bf2=[bf2],bf8=[bf8,bf5],c00=[c00,bfb],c03=[c03],c09=[c09,c06],c0c=[c0c],c0f=[c0f]}

maps to 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5
2020-02-05 22:46:53.786936 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] get_object_context: found obc
in cache: 0x56137068c000
2020-02-05 22:46:53.786951 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] get_object_context:
0x56137068c000 
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 rwstate(none n=0 w=0) oi: 
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5(56424'2944715
osd.1.0:1001912 dirty|data_digest|omap_digest s 4194304 uv 9888802 
dd 915b865a od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x56136841dfa0 snapset: 
c0f=[c0f,c0c,c09,c06,c03,c00,bfb,bf8,bf5,bf2,bef,bec,be9,be6,be3,be0,bdd,bd8,bd5,bd2,bcf,bcc,bc9,bc6,bc3,bc0,bbd,bba,bb7,bad,ba7,ba1,b9b,b95,b8f,b89,b83,b7d,b77,b71,b6b,b65,b5f,b59,b53,b4d,b47,b41,b3b,b35,b2f,b29,b23,b1d,b17,b0f,b09,b03,afd,af7,af1,aeb,ae5,adf,ad9,ad3,acd,ac7,ac1,abb,ab5,aaf,aa9,aa3,a9d,a97,a91,a8b,a85,a7f,a79,a73,a6d,a67,a61,a5b,a55,a4f,a49,a43,a3d,a37,a31,a2b,a25,a1f,a19,a13,a0d,a07,a01,9fb,9f5,9ef,9e9,9e3,9dd,9d7,9d1,9cb,9c5,9bf,9b9,9b3,9ad,9a7,9a1,99b,995,98f,989,983,97d,977,971,96b,965,95f,959,953,94d,947,941,93b,935,92f,929,923,91d,917,911,90b,905,8ff,8f9,8f3,8ed,8e7,8e1,8db,8d5,8cf,8c9,8c3,8bd,8b7,8b1,8ab,8a5,89f,899,893,88d,887,881,87b,875,86f,869,863,85d,857,851,84b,845,83f,839,833,82d,827,821,81b,815,80f,809,803,7fd,7f7,7f1,7eb,7e5,7df,7d9,7d3,7cd,7c7,7c1,7bb,7b5,7af,7a9,7a3,79d,797,791,78b,785,77f,779,773,76d,767,761,75b,755,74f,749,743,73d,737,731,72b,725,71f,719,713,70d]:{70d=[70d],719=[719,713],71f=[71f],72b=[72b,725],731=[731],73d=[73d,737],743=[743],74f=[74f,749],755=[755],761=[761,75b],767=[767],773=[773,76d],77f=[77f,779],785=[785],78b=[78b],791=[791],79d=[79d,797],7a9=[7a9,7a3],7b5=[7b5,7af],7c1=[7c1,7bb],7cd=[7cd,7c7],7d3=[7d3],7df=[7df,7d9],7eb=[7eb,7e5],7f1=[7f1],7fd=[7fd,7f7],803=[803],80f=[80f,809],81b=[81b,815],827=[827,821],82d=[82d],833=[833],839=[839],83f=[83f],84b=[84b,845],851=[851],857=[857],85d=[85d],863=[863],869=[869],875=[875,86f],87b=[87b],881=[881],88d=[88d,887],899=[899,893],89f=[89f],8a5=[8a5],8ab=[8ab],8b1=[8b1],8bd=[8bd,8b7],8c9=[8c9,8c3],8cf=[8cf],8d5=[8d5],8db=[8db],8e1=[8e1],8ed=[8ed,8e7],8f9=[8f9,8f3],8ff=[8ff],905=[905],911=[911,90b],917=[917],923=[923,91d],92f=[92f,929],935=[935],941=[941,93b],94d=[94d,947],959=[959,953],95f=[95f],96b=[96b,965],971=[971],977=[977],983=[983,97d],989=[989],995=[995,98f],99b=[99b],9a1=[9a1],9ad=[9ad,9a7],9b9=[9b9,9b3],9c5=[9c5,9bf],9cb=[9cb],9d7=[9d7,9d1],9dd=[9dd],9e9=[9e9,9e3],9f5=[9f5,9ef],a01=[a01,9fb],a07=[a07],a0d=[a0d],a19=[a19,a13],a1f=[a1f],a25=[a25],a2b=[a2b],a31=[a31],a37=[a37],a43=[a43,a3d],a49=[a49],a55=[a55,a4f],a5b=[a5b],a61=[a61],a67=[a67],a6d=[a6d],a79=[a79,a73],a7f=[a7f],a8b=[a8b,a85],a91=[a91],a9d=[a9d,a97],aa3=[aa3],aa9=[aa9],aaf=[aaf],abb=[abb,ab5],ac7=[ac7,ac1],acd=[acd],ad9=[ad9,ad3],adf=[adf],ae5=[ae5],af1=[af1,aeb],afd=[afd,af7],b03=[b03],b0f=[b0f,b09],b1d=[b1d,b17],b23=[b23],b2f=[b2f,b29],b35=[b35],b41=[b41,b3b],b47=[b47],b53=[b53,b4d],b59=[b59],b5f=[b5f],b6b=[b6b,b65],b71=[b71],b77=[b77],b7d=[b7d],b83=[b83],b89=[b89],b95=[b95,b8f],b9b=[b9b],ba1=[ba1],ba7=[ba7],bad=[bad],bb7=[bb7],bbd=[bbd,bba],bc3=[bc3,bc0],bc9=[bc9,bc6],bcc=[bcc],bcf=[bcf],bd2=[bd2],bd5=[bd5],bd8=[bd8],be0=[be0,bdd],be6=[be6,be3],bec=[bec,be9],bef=[bef],bf2=[bf2],bf8=[bf8,bf5],c00=[c00,bfb],c03=[c03],c09=[c09,c06],c0c=[c0c],c0f=[c0f]}
2020-02-05 22:46:53.787106 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] find_object_context
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 @7b5 
snapset 
c0f=[c0f,c0c,c09,c06,c03,c00,bfb,bf8,bf5,bf2,bef,bec,be9,be6,be3,be0,bdd,bd8,bd5,bd2,bcf,bcc,bc9,bc6,bc3,bc0,bbd,bba,bb7,bad,ba7,ba1,b9b,b95,b8f,b89,b83,b7d,b77,b71,b6b,b65,b5f,b59,b53,b4d,b47,b41,b3b,b35,b2f,b29,b23,b1d,b17,b0f,b09,b03,afd,af7,af1,aeb,ae5,adf,ad9,ad3,acd,ac7,ac1,abb,ab5,aaf,aa9,aa3,a9d,a97,a91,a8b,a85,a7f,a79,a73,a6d,a67,a61,a5b,a55,a4f,a49,a43,a3d,a37,a31,a2b,a25,a1f,a19,a13,a0d,a07,a01,9fb,9f5,9ef,9e9,9e3,9dd,9d7,9d1,9cb,9c5,9bf,9b9,9b3,9ad,9a7,9a1,99b,995,98f,989,983,97d,977,971,96b,965,95f,959,953,94d,947,941,93b,935,92f,929,923,91d,917,911,90b,905,8ff,8f9,8f3,8ed,8e7,8e1,8db,8d5,8cf,8c9,8c3,8bd,8b7,8b1,8ab,8a5,89f,899,893,88d,887,881,87b,875,86f,869,863,85d,857,851,84b,845,83f,839,833,82d,827,821,81b,815,80f,809,803,7fd,7f7,7f1,7eb,7e5,7df,7d9,7d3,7cd,7c7,7c1,7bb,7b5,7af,7a9,7a3,79d,797,791,78b,785,77f,779,773,76d,767,761,75b,755,74f,749,743,73d,737,731,72b,725,71f,719,713,70d]:{70d=[70d],719=[719,713],71f=[71f],72b=[72b,725],731=[731],73d=[73d,737],743=[743],74f=[74f,749],755=[755],761=[761,75b],767=[767],773=[773,76d],77f=[77f,779],785=[785],78b=[78b],791=[791],79d=[79d,797],7a9=[7a9,7a3],7b5=[7b5,7af],7c1=[7c1,7bb],7cd=[7cd,7c7],7d3=[7d3],7df=[7df,7d9],7eb=[7eb,7e5],7f1=[7f1],7fd=[7fd,7f7],803=[803],80f=[80f,809],81b=[81b,815],827=[827,821],82d=[82d],833=[833],839=[839],83f=[83f],84b=[84b,845],851=[851],857=[857],85d=[85d],863=[863],869=[869],875=[875,86f],87b=[87b],881=[881],88d=[88d,887],899=[899,893],89f=[89f],8a5=[8a5],8ab=[8ab],8b1=[8b1],8bd=[8bd,8b7],8c9=[8c9,8c3],8cf=[8cf],8d5=[8d5],8db=[8db],8e1=[8e1],8ed=[8ed,8e7],8f9=[8f9,8f3],8ff=[8ff],905=[905],911=[911,90b],917=[917],923=[923,91d],92f=[92f,929],935=[935],941=[941,93b],94d=[94d,947],959=[959,953],95f=[95f],96b=[96b,965],971=[971],977=[977],983=[983,97d],989=[989],995=[995,98f],99b=[99b],9a1=[9a1],9ad=[9ad,9a7],9b9=[9b9,9b3],9c5=[9c5,9bf],9cb=[9cb],9d7=[9d7,9d1],9dd=[9dd],9e9=[9e9,9e3],9f5=[9f5,9ef],a01=[a01,9fb],a07=[a07],a0d=[a0d],a19=[a19,a13],a1f=[a1f],a25=[a25],a2b=[a2b],a31=[a31],a37=[a37],a43=[a43,a3d],a49=[a49],a55=[a55,a4f],a5b=[a5b],a61=[a61],a67=[a67],a6d=[a6d],a79=[a79,a73],a7f=[a7f],a8b=[a8b,a85],a91=[a91],a9d=[a9d,a97],aa3=[aa3],aa9=[aa9],aaf=[aaf],abb=[abb,ab5],ac7=[ac7,ac1],acd=[acd],ad9=[ad9,ad3],adf=[adf],ae5=[ae5],af1=[af1,aeb],afd=[afd,af7],b03=[b03],b0f=[b0f,b09],b1d=[b1d,b17],b23=[b23],b2f=[b2f,b29],b35=[b35],b41=[b41,b3b],b47=[b47],b53=[b53,b4d],b59=[b59],b5f=[b5f],b6b=[b6b,b65],b71=[b71],b77=[b77],b7d=[b7d],b83=[b83],b89=[b89],b95=[b95,b8f],b9b=[b9b],ba1=[ba1],ba7=[ba7],bad=[bad],bb7=[bb7],bbd=[bbd,bba],bc3=[bc3,bc0],bc9=[bc9,bc6],bcc=[bcc],bcf=[bcf],bd2=[bd2],bd5=[bd5],bd8=[bd8],be0=[be0,bdd],be6=[be6,be3],bec=[bec,be9],bef=[bef],bf2=[bf2],bf8=[bf8,bf5],c00=[c00,bfb],c03=[c03],c09=[c09,c06],c0c=[c0c],c0f=[c0f]}

2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 HIT
2020-02-05 22:46:53.787323 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_op obc
obc(2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 rwstate(excl 
n=1 w=0))
2020-02-05 22:46:53.787347 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] execute_ctx 0x561330e4c000
2020-02-05 22:46:53.787412 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] execute_ctx
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 
8388608] ov 56424'2944715
2020-02-05 22:46:53.787432 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean]  taking ondisk_read_lock
2020-02-05 22:46:53.787449 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_osd_op
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 8388608]
2020-02-05 22:46:53.787483 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_osd_op  copy-get max
8388608
2020-02-05 22:46:53.787515 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean]  got attrs
2020-02-05 22:46:53.807853 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean]  got data
2020-02-05 22:46:53.808034 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean]  got reqids
2020-02-05 22:46:53.808051 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean]  cursor.is_complete=1 0 attrs
4194304 bytes 0 omap header bytes 0 omap data bytes in 0 
keys 0 reqids
2020-02-05 22:46:53.808085 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean]  dropping ondisk_read_lock
2020-02-05 22:46:53.808107 7ff1c3749700 15 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_osd_op_effects osd.9 con
0x56135d9e5000
2020-02-05 22:46:53.808132 7ff1c3749700 15 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] log_op_stats
osd_op(osd.9.60026:836317 2.c 
2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 8388608] snapc
0=[] 
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e60042) v8 inb 0 outb 4194423 lat 19.627387
2020-02-05 22:46:53.808171 7ff1c3749700 15 osd.1 pg_epoch: 60042 pg[2.c( v
60042'3048392 (58743'3046874,60042'3048392] 
local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237
60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 
lcod 60042'3048390 mlcod 60042'3048390 active+clean] publish_stats_to_osd
60042:3424975
2020-02-05 22:46:53.808292 7ff1c3749700 10 osd.1 60042 dequeue_op 0x561362c80bc0 finish

On 03/02/2020 15.48, Jan Pekař - Imatic wrote:
...
  Hi all,

 I have small cluster and yesterday I tried to mount older RBD snapshot torecover data. (I
have approx. 230 daily snapshots of one RBD 
 image on my small ceph).

 After I did mount and ls operation, cluster was stuck and I notice that 2of my OSD's
eaten CPU and raise in memory usage (more than 4x 
 more memory than usual). Cluster was unusable after that and it didn't help stopping
rbd mount operation and restarting OSD's etc.

 Strange is, that it affects only 2 of my OSD's - one is on NVME and has cache
PG's mapped, other one is rotational HDD. IOPS are normal, 
 only CPU and memory usage are unusual.

 I noticed hundred thousands of slow requests (on idle cluster) so I startto debug OSD
operations. You can find OSD 8 log below. OSD 9 
 contains nvme cached PG's and there is nothing unusual in standard log. In the middle
of log I raised debug level to 10.

 It look's to me, that it is trying to do one operation with one snapshotted RBD
object in PG 2.b 
 (rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3) in a loop.

 Is there any way how to stop that? What is causing the problem? Can I somehow flush
OSD's "journals" or "queues" to stop current operation 
 and make cluster usable again? Is there any way how I can access old snapshot data?

 Problem started on Luminous 12.2.12 and upgrade to 12.2.13 didn't help.

 Thank you
 Jan Pekar

 2020-02-03 15:09:22.470277 7f51f6af7700  0 log_channel(cluster) log [WRN] : 30966 slow
requests, 5 included below; oldest blocked for > 
 37.505482 secs
 2020-02-03 15:09:22.470336 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.503404 seconds old, received at 2020-02-03 
 15:08:44.956422: osd_op(osd.9.59923:2670991 2.b 2.162d8f8b (undecoded) 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
 2020-02-03 15:09:22.470360 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.500695 seconds old, received at 2020-02-03 
 15:08:44.959132: osd_op(osd.9.59923:2671023 2.b 2.162d8f8b (undecoded) 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
 2020-02-03 15:09:22.470366 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.498013 seconds old, received at 2020-02-03 
 15:08:44.961814: osd_op(osd.9.59923:2671055 2.b 2.162d8f8b (undecoded) 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
 2020-02-03 15:09:22.470371 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.494836 seconds old, received at 2020-02-03 
 15:08:44.964990: osd_op(osd.9.59923:2671087 2.b 2.162d8f8b (undecoded) 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
 2020-02-03 15:09:22.470376 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request
37.491950 seconds old, received at 2020-02-03 
 15:08:44.967877: osd_op(osd.9.59923:2671119 2.b 2.162d8f8b (undecoded) 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) currently queued_for_pg
 ...
 2020-02-03 15:09:35.537259 7f51ddac5700  0 osd.8 59939 do_command r=0
 2020-02-03 15:09:35.588753 7f51ddac5700  0 osd.8 59939 do_command r=0 Set debug_osd to
10/10
 2020-02-03 15:09:35.588814 7f51ddac5700  0 log_channel(cluster) log [INF] : Set debug_osd
to 10/10
 2020-02-03 15:09:35.597667 7f51f0945700  2 osd.8 59939 ms_handle_reset con 0x5647736ac800
session 0x564785519e00
 2020-02-03 15:09:35.599056 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  dropping
ondisk_read_lock
 2020-02-03 15:09:35.599209 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5800 finish
 2020-02-03 15:09:35.599250 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d59c0 prio 63
cost 29 latency 22.224501 
 osd_op(osd.9.59923:2779946 2.b 2.162d8f8b (undecoded)
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 
 e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227]
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 
 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod
0'0 mlcod 0'0 active+clean]
 2020-02-03 15:09:35.599293 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] _handle_message:
0x5647873d59c0
 2020-02-03 15:09:35.599347 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_op
osd_op(osd.9.59923:2779946 2.b 
 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] snapc
0=[] 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) v8 may_read -> write-ordered flags 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
 2020-02-03 15:09:35.599398 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 
 snapset 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

 map_snapid_to_clone=true
 2020-02-03 15:09:35.599506 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 
 snapset 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

 maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3
 2020-02-03 15:09:35.599606 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: found
obc in cache: 0x564785fb3880
 2020-02-03 15:09:35.613724 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context:
0x564785fb3880 
 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none n=0 w=0) oi: 
 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046
osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 uv 7633387 
 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160 snapset: 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}
 2020-02-03 15:09:35.613881 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 
 snapset 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT
 2020-02-03 15:09:35.614024 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx
0x5647843a3000
 2020-02-03 15:09:35.614053 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 
 8388608] ov 58682'4215046
 2020-02-03 15:09:35.614077 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  taking ondisk_read_lock
 2020-02-03 15:09:35.614095 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608]
 2020-02-03 15:09:35.614112 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max
8388608
 2020-02-03 15:09:35.637123 7f51f72f8700 10 osd.8 59939 tick
 2020-02-03 15:09:35.637149 7f51f72f8700 10 osd.8 59939 do_waiters -- start
 2020-02-03 15:09:35.637152 7f51f72f8700 10 osd.8 59939 do_waiters -- finish
 2020-02-03 15:09:35.645761 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  dropping
ondisk_read_lock
 2020-02-03 15:09:35.645887 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d59c0 finish
 2020-02-03 15:09:35.645924 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5b80 prio 63
cost 29 latency 22.271117 
 osd_op(osd.9.59923:2779947 2.b 2.162d8f8b (undecoded)
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 
 e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227]
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 
 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod
0'0 mlcod 0'0 active+clean]
 2020-02-03 15:09:35.645960 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] _handle_message:
0x5647873d5b80
 2020-02-03 15:09:35.646007 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_op
osd_op(osd.9.59923:2779947 2.b 
 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] snapc
0=[] 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) v8 may_read -> write-ordered flags 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
 2020-02-03 15:09:35.646047 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 
 snapset 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

 map_snapid_to_clone=true
 2020-02-03 15:09:35.646145 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 
 snapset 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

 maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3
 2020-02-03 15:09:35.646239 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: found
obc in cache: 0x564785fb3880
 2020-02-03 15:09:35.646255 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context:
0x564785fb3880 
 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none n=0 w=0) oi: 
 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046
osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 uv 7633387 
 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160 snapset: 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}
 2020-02-03 15:09:35.646359 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 
 snapset 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT
 2020-02-03 15:09:35.646469 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx
0x5647843a3000
 2020-02-03 15:09:35.646494 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 
 8388608] ov 58682'4215046
 2020-02-03 15:09:35.646513 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  taking ondisk_read_lock
 2020-02-03 15:09:35.646530 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608]
 2020-02-03 15:09:35.646547 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max
8388608
 2020-02-03 15:09:35.681653 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  dropping
ondisk_read_lock
 2020-02-03 15:09:35.681780 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5b80 finish
 2020-02-03 15:09:35.681817 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5d40 prio 63
cost 29 latency 22.306952 
 osd_op(osd.9.59923:2779948 2.b 2.162d8f8b (undecoded)
ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 
 e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227]
local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 
 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod
0'0 mlcod 0'0 active+clean]
 2020-02-03 15:09:35.681851 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] _handle_message:
0x5647873d5d40
 2020-02-03 15:09:35.681898 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_op
osd_op(osd.9.59923:2779948 2.b 
 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] snapc
0=[] 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
e59939) v8 may_read -> write-ordered flags 
 ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected
 2020-02-03 15:09:35.681937 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 
 snapset 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

 map_snapid_to_clone=true
 2020-02-03 15:09:35.682035 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 
 snapset 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

 maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3
 2020-02-03 15:09:35.682128 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: found
obc in cache: 0x564785fb3880
 2020-02-03 15:09:35.682144 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context:
0x564785fb3880 
 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none n=0 w=0) oi: 
 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046
osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 uv 7633387 
 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160 snapset: 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}
 2020-02-03 15:09:35.682245 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 
 snapset 

c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]}

 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT
 2020-02-03 15:09:35.682356 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx
0x5647843a3000
 2020-02-03 15:09:35.682385 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 
 8388608] ov 58682'4215046
 2020-02-03 15:09:35.682404 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  taking ondisk_read_lock
 2020-02-03 15:09:35.682420 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op
2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608]
 2020-02-03 15:09:35.682437 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max
8388608
 2020-02-03 15:09:35.714076 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v
59908'4218227 (58731'4216723,59908'4218227] 
 local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237
59932/59932/59932) [8,0] r=0 lpr=59932 
 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  dropping
ondisk_read_lock
 2020-02-03 15:09:35.714196 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5d40 finish

 _______________________________________________
 ceph-users mailing list -- ceph-users(a)ceph.io
 To unsubscribe send an email to ceph-users-leave(a)ceph.io 
-- 
============
Ing. Jan Pekař
jan.pekar(a)imatic.cz | +420603811737
----
Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz
============
--

2024

2023

2022

2021

2020

2019

[ceph-users] Re: Problem with OSD - stuck in CPU loop after rbd snapshot mount