Hi, because load to my OSD's stopped approximately after 12 hours and everything returned to normal, I did another test. I have chosen another rbd image and some old snapshot and I tried to mount it - now without cloning it. rbd-nbd map succeeded, but situation from my first test repeated, only different OSD's (PG) were affected. It looks to me, that it is trying to do some operation on one snapshotted object, that object is in cache so primary OSD that is holding object, and OSD that is holding cached object are affected with CPU high usage (loop). One loop operation with debug=20 is included below. This same operation is executed again and again etc.. and I don't know how to stop that. When I stop affected osd, another one which is holding replica of affected object (affected PG) starts the same bombing of cache and consumes CPU and memory. Whenever affected PG is down, nothing unusual is happening in the cluster. Flushing object from cache didn't help. For this time, object mentioned in logs below is first sector of snapshoted RBD image, which I was trying to map. Any ideas or hints? With regards Jan Pekar 2020-02-05 22:46:53.786394 7ff1c3749700 10 osd.1 60042 dequeue_op 0x561362c80a00 finish 2020-02-05 22:46:53.786434 7ff1c3749700 20 osd.1 op_wq(2) _process 2.c item PGQueueable(0x561362c80bc0 prio 63 cost 29 e60042) queued 2020-02-05 22:46:53.786444 7ff1c3749700 20 osd.1 op_wq(2) _process 2.c item PGQueueable(0x561362c80bc0 prio 63 cost 29 e60042) pg 0x561324afc000 2020-02-05 22:46:53.786450 7ff1c3749700 10 osd.1 60042 dequeue_op 0x561362c80bc0 prio 63 cost 29 latency 19.605707 osd_op(osd.9.60026:836317 2.c 2.4011388c (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e60042) v8 pg pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] 2020-02-05 22:46:53.786474 7ff1c3749700 20 osd.1 60042 share_map osd.9 10.4.4.10:0/1678416 60042 2020-02-05 22:46:53.786482 7ff1c3749700 20 osd.1 60042 should_share_map osd.9 10.4.4.10:0/1678416 60042 2020-02-05 22:46:53.786494 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] _handle_message: 0x561362c80bc0 2020-02-05 22:46:53.786522 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_op: op osd_op(osd.9.60026:836317 2.c 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e60042) v8 2020-02-05 22:46:53.786554 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] op_has_sufficient_caps session=0x561363328580 pool=2 (rbd ) owner=0 need_read_cap=1 need_write_cap=0 classes=[] -> yes 2020-02-05 22:46:53.786588 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_op osd_op(osd.9.60026:836317 2.c 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e60042) v8 may_read -> write-ordered flags ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 2020-02-05 22:46:53.786628 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] find_object_context 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 @7b5 snapset c0f=[c0f,c0c,c09,c06,c03,c00,bfb,bf8,bf5,bf2,bef,bec,be9,be6,be3,be0,bdd,bd8,bd5,bd2,bcf,bcc,bc9,bc6,bc3,bc0,bbd,bba,bb7,bad,ba7,ba1,b9b,b95,b8f,b89,b83,b7d,b77,b71,b6b,b65,b5f,b59,b53,b4d,b47,b41,b3b,b35,b2f,b29,b23,b1d,b17,b0f,b09,b03,afd,af7,af1,aeb,ae5,adf,ad9,ad3,acd,ac7,ac1,abb,ab5,aaf,aa9,aa3,a9d,a97,a91,a8b,a85,a7f,a79,a73,a6d,a67,a61,a5b,a55,a4f,a49,a43,a3d,a37,a31,a2b,a25,a1f,a19,a13,a0d,a07,a01,9fb,9f5,9ef,9e9,9e3,9dd,9d7,9d1,9cb,9c5,9bf,9b9,9b3,9ad,9a7,9a1,99b,995,98f,989,983,97d,977,971,96b,965,95f,959,953,94d,947,941,93b,935,92f,929,923,91d,917,911,90b,905,8ff,8f9,8f3,8ed,8e7,8e1,8db,8d5,8cf,8c9,8c3,8bd,8b7,8b1,8ab,8a5,89f,899,893,88d,887,881,87b,875,86f,869,863,85d,857,851,84b,845,83f,839,833,82d,827,821,81b,815,80f,809,803,7fd,7f7,7f1,7eb,7e5,7df,7d9,7d3,7cd,7c7,7c1,7bb,7b5,7af,7a9,7a3,79d,797,791,78b,785,77f,779,773,76d,767,761,75b,755,74f,749,743,73d,737,731,72b,725,71f,719,713,70d]:{70d=[70d],719=[719,713],71f=[71f],72b=[72b,725],731=[731],73d=[73d,737],743=[743],74f=[74f,749],755=[755],761=[761,75b],767=[767],773=[773,76d],77f=[77f,779],785=[785],78b=[78b],791=[791],79d=[79d,797],7a9=[7a9,7a3],7b5=[7b5,7af],7c1=[7c1,7bb],7cd=[7cd,7c7],7d3=[7d3],7df=[7df,7d9],7eb=[7eb,7e5],7f1=[7f1],7fd=[7fd,7f7],803=[803],80f=[80f,809],81b=[81b,815],827=[827,821],82d=[82d],833=[833],839=[839],83f=[83f],84b=[84b,845],851=[851],857=[857],85d=[85d],863=[863],869=[869],875=[875,86f],87b=[87b],881=[881],88d=[88d,887],899=[899,893],89f=[89f],8a5=[8a5],8ab=[8ab],8b1=[8b1],8bd=[8bd,8b7],8c9=[8c9,8c3],8cf=[8cf],8d5=[8d5],8db=[8db],8e1=[8e1],8ed=[8ed,8e7],8f9=[8f9,8f3],8ff=[8ff],905=[905],911=[911,90b],917=[917],923=[923,91d],92f=[92f,929],935=[935],941=[941,93b],94d=[94d,947],959=[959,953],95f=[95f],96b=[96b,965],971=[971],977=[977],983=[983,97d],989=[989],995=[995,98f],99b=[99b],9a1=[9a1],9ad=[9ad,9a7],9b9=[9b9,9b3],9c5=[9c5,9bf],9cb=[9cb],9d7=[9d7,9d1],9dd=[9dd],9e9=[9e9,9e3],9f5=[9f5,9ef],a01=[a01,9fb],a07=[a07],a0d=[a0d],a19=[a19,a13],a1f=[a1f],a25=[a25],a2b=[a2b],a31=[a31],a37=[a37],a43=[a43,a3d],a49=[a49],a55=[a55,a4f],a5b=[a5b],a61=[a61],a67=[a67],a6d=[a6d],a79=[a79,a73],a7f=[a7f],a8b=[a8b,a85],a91=[a91],a9d=[a9d,a97],aa3=[aa3],aa9=[aa9],aaf=[aaf],abb=[abb,ab5],ac7=[ac7,ac1],acd=[acd],ad9=[ad9,ad3],adf=[adf],ae5=[ae5],af1=[af1,aeb],afd=[afd,af7],b03=[b03],b0f=[b0f,b09],b1d=[b1d,b17],b23=[b23],b2f=[b2f,b29],b35=[b35],b41=[b41,b3b],b47=[b47],b53=[b53,b4d],b59=[b59],b5f=[b5f],b6b=[b6b,b65],b71=[b71],b77=[b77],b7d=[b7d],b83=[b83],b89=[b89],b95=[b95,b8f],b9b=[b9b],ba1=[ba1],ba7=[ba7],bad=[bad],bb7=[bb7],bbd=[bbd,bba],bc3=[bc3,bc0],bc9=[bc9,bc6],bcc=[bcc],bcf=[bcf],bd2=[bd2],bd5=[bd5],bd8=[bd8],be0=[be0,bdd],be6=[be6,be3],bec=[bec,be9],bef=[bef],bf2=[bf2],bf8=[bf8,bf5],c00=[c00,bfb],c03=[c03],c09=[c09,c06],c0c=[c0c],c0f=[c0f]} map_snapid_to_clone=true 2020-02-05 22:46:53.786790 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] find_object_context 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 @7b5 snapset c0f=[c0f,c0c,c09,c06,c03,c00,bfb,bf8,bf5,bf2,bef,bec,be9,be6,be3,be0,bdd,bd8,bd5,bd2,bcf,bcc,bc9,bc6,bc3,bc0,bbd,bba,bb7,bad,ba7,ba1,b9b,b95,b8f,b89,b83,b7d,b77,b71,b6b,b65,b5f,b59,b53,b4d,b47,b41,b3b,b35,b2f,b29,b23,b1d,b17,b0f,b09,b03,afd,af7,af1,aeb,ae5,adf,ad9,ad3,acd,ac7,ac1,abb,ab5,aaf,aa9,aa3,a9d,a97,a91,a8b,a85,a7f,a79,a73,a6d,a67,a61,a5b,a55,a4f,a49,a43,a3d,a37,a31,a2b,a25,a1f,a19,a13,a0d,a07,a01,9fb,9f5,9ef,9e9,9e3,9dd,9d7,9d1,9cb,9c5,9bf,9b9,9b3,9ad,9a7,9a1,99b,995,98f,989,983,97d,977,971,96b,965,95f,959,953,94d,947,941,93b,935,92f,929,923,91d,917,911,90b,905,8ff,8f9,8f3,8ed,8e7,8e1,8db,8d5,8cf,8c9,8c3,8bd,8b7,8b1,8ab,8a5,89f,899,893,88d,887,881,87b,875,86f,869,863,85d,857,851,84b,845,83f,839,833,82d,827,821,81b,815,80f,809,803,7fd,7f7,7f1,7eb,7e5,7df,7d9,7d3,7cd,7c7,7c1,7bb,7b5,7af,7a9,7a3,79d,797,791,78b,785,77f,779,773,76d,767,761,75b,755,74f,749,743,73d,737,731,72b,725,71f,719,713,70d]:{70d=[70d],719=[719,713],71f=[71f],72b=[72b,725],731=[731],73d=[73d,737],743=[743],74f=[74f,749],755=[755],761=[761,75b],767=[767],773=[773,76d],77f=[77f,779],785=[785],78b=[78b],791=[791],79d=[79d,797],7a9=[7a9,7a3],7b5=[7b5,7af],7c1=[7c1,7bb],7cd=[7cd,7c7],7d3=[7d3],7df=[7df,7d9],7eb=[7eb,7e5],7f1=[7f1],7fd=[7fd,7f7],803=[803],80f=[80f,809],81b=[81b,815],827=[827,821],82d=[82d],833=[833],839=[839],83f=[83f],84b=[84b,845],851=[851],857=[857],85d=[85d],863=[863],869=[869],875=[875,86f],87b=[87b],881=[881],88d=[88d,887],899=[899,893],89f=[89f],8a5=[8a5],8ab=[8ab],8b1=[8b1],8bd=[8bd,8b7],8c9=[8c9,8c3],8cf=[8cf],8d5=[8d5],8db=[8db],8e1=[8e1],8ed=[8ed,8e7],8f9=[8f9,8f3],8ff=[8ff],905=[905],911=[911,90b],917=[917],923=[923,91d],92f=[92f,929],935=[935],941=[941,93b],94d=[94d,947],959=[959,953],95f=[95f],96b=[96b,965],971=[971],977=[977],983=[983,97d],989=[989],995=[995,98f],99b=[99b],9a1=[9a1],9ad=[9ad,9a7],9b9=[9b9,9b3],9c5=[9c5,9bf],9cb=[9cb],9d7=[9d7,9d1],9dd=[9dd],9e9=[9e9,9e3],9f5=[9f5,9ef],a01=[a01,9fb],a07=[a07],a0d=[a0d],a19=[a19,a13],a1f=[a1f],a25=[a25],a2b=[a2b],a31=[a31],a37=[a37],a43=[a43,a3d],a49=[a49],a55=[a55,a4f],a5b=[a5b],a61=[a61],a67=[a67],a6d=[a6d],a79=[a79,a73],a7f=[a7f],a8b=[a8b,a85],a91=[a91],a9d=[a9d,a97],aa3=[aa3],aa9=[aa9],aaf=[aaf],abb=[abb,ab5],ac7=[ac7,ac1],acd=[acd],ad9=[ad9,ad3],adf=[adf],ae5=[ae5],af1=[af1,aeb],afd=[afd,af7],b03=[b03],b0f=[b0f,b09],b1d=[b1d,b17],b23=[b23],b2f=[b2f,b29],b35=[b35],b41=[b41,b3b],b47=[b47],b53=[b53,b4d],b59=[b59],b5f=[b5f],b6b=[b6b,b65],b71=[b71],b77=[b77],b7d=[b7d],b83=[b83],b89=[b89],b95=[b95,b8f],b9b=[b9b],ba1=[ba1],ba7=[ba7],bad=[bad],bb7=[bb7],bbd=[bbd,bba],bc3=[bc3,bc0],bc9=[bc9,bc6],bcc=[bcc],bcf=[bcf],bd2=[bd2],bd5=[bd5],bd8=[bd8],be0=[be0,bdd],be6=[be6,be3],bec=[bec,be9],bef=[bef],bf2=[bf2],bf8=[bf8,bf5],c00=[c00,bfb],c03=[c03],c09=[c09,c06],c0c=[c0c],c0f=[c0f]} maps to 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 2020-02-05 22:46:53.786936 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] get_object_context: found obc in cache: 0x56137068c000 2020-02-05 22:46:53.786951 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] get_object_context: 0x56137068c000 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 rwstate(none n=0 w=0) oi: 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5(56424'2944715 osd.1.0:1001912 dirty|data_digest|omap_digest s 4194304 uv 9888802 dd 915b865a od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x56136841dfa0 snapset: c0f=[c0f,c0c,c09,c06,c03,c00,bfb,bf8,bf5,bf2,bef,bec,be9,be6,be3,be0,bdd,bd8,bd5,bd2,bcf,bcc,bc9,bc6,bc3,bc0,bbd,bba,bb7,bad,ba7,ba1,b9b,b95,b8f,b89,b83,b7d,b77,b71,b6b,b65,b5f,b59,b53,b4d,b47,b41,b3b,b35,b2f,b29,b23,b1d,b17,b0f,b09,b03,afd,af7,af1,aeb,ae5,adf,ad9,ad3,acd,ac7,ac1,abb,ab5,aaf,aa9,aa3,a9d,a97,a91,a8b,a85,a7f,a79,a73,a6d,a67,a61,a5b,a55,a4f,a49,a43,a3d,a37,a31,a2b,a25,a1f,a19,a13,a0d,a07,a01,9fb,9f5,9ef,9e9,9e3,9dd,9d7,9d1,9cb,9c5,9bf,9b9,9b3,9ad,9a7,9a1,99b,995,98f,989,983,97d,977,971,96b,965,95f,959,953,94d,947,941,93b,935,92f,929,923,91d,917,911,90b,905,8ff,8f9,8f3,8ed,8e7,8e1,8db,8d5,8cf,8c9,8c3,8bd,8b7,8b1,8ab,8a5,89f,899,893,88d,887,881,87b,875,86f,869,863,85d,857,851,84b,845,83f,839,833,82d,827,821,81b,815,80f,809,803,7fd,7f7,7f1,7eb,7e5,7df,7d9,7d3,7cd,7c7,7c1,7bb,7b5,7af,7a9,7a3,79d,797,791,78b,785,77f,779,773,76d,767,761,75b,755,74f,749,743,73d,737,731,72b,725,71f,719,713,70d]:{70d=[70d],719=[719,713],71f=[71f],72b=[72b,725],731=[731],73d=[73d,737],743=[743],74f=[74f,749],755=[755],761=[761,75b],767=[767],773=[773,76d],77f=[77f,779],785=[785],78b=[78b],791=[791],79d=[79d,797],7a9=[7a9,7a3],7b5=[7b5,7af],7c1=[7c1,7bb],7cd=[7cd,7c7],7d3=[7d3],7df=[7df,7d9],7eb=[7eb,7e5],7f1=[7f1],7fd=[7fd,7f7],803=[803],80f=[80f,809],81b=[81b,815],827=[827,821],82d=[82d],833=[833],839=[839],83f=[83f],84b=[84b,845],851=[851],857=[857],85d=[85d],863=[863],869=[869],875=[875,86f],87b=[87b],881=[881],88d=[88d,887],899=[899,893],89f=[89f],8a5=[8a5],8ab=[8ab],8b1=[8b1],8bd=[8bd,8b7],8c9=[8c9,8c3],8cf=[8cf],8d5=[8d5],8db=[8db],8e1=[8e1],8ed=[8ed,8e7],8f9=[8f9,8f3],8ff=[8ff],905=[905],911=[911,90b],917=[917],923=[923,91d],92f=[92f,929],935=[935],941=[941,93b],94d=[94d,947],959=[959,953],95f=[95f],96b=[96b,965],971=[971],977=[977],983=[983,97d],989=[989],995=[995,98f],99b=[99b],9a1=[9a1],9ad=[9ad,9a7],9b9=[9b9,9b3],9c5=[9c5,9bf],9cb=[9cb],9d7=[9d7,9d1],9dd=[9dd],9e9=[9e9,9e3],9f5=[9f5,9ef],a01=[a01,9fb],a07=[a07],a0d=[a0d],a19=[a19,a13],a1f=[a1f],a25=[a25],a2b=[a2b],a31=[a31],a37=[a37],a43=[a43,a3d],a49=[a49],a55=[a55,a4f],a5b=[a5b],a61=[a61],a67=[a67],a6d=[a6d],a79=[a79,a73],a7f=[a7f],a8b=[a8b,a85],a91=[a91],a9d=[a9d,a97],aa3=[aa3],aa9=[aa9],aaf=[aaf],abb=[abb,ab5],ac7=[ac7,ac1],acd=[acd],ad9=[ad9,ad3],adf=[adf],ae5=[ae5],af1=[af1,aeb],afd=[afd,af7],b03=[b03],b0f=[b0f,b09],b1d=[b1d,b17],b23=[b23],b2f=[b2f,b29],b35=[b35],b41=[b41,b3b],b47=[b47],b53=[b53,b4d],b59=[b59],b5f=[b5f],b6b=[b6b,b65],b71=[b71],b77=[b77],b7d=[b7d],b83=[b83],b89=[b89],b95=[b95,b8f],b9b=[b9b],ba1=[ba1],ba7=[ba7],bad=[bad],bb7=[bb7],bbd=[bbd,bba],bc3=[bc3,bc0],bc9=[bc9,bc6],bcc=[bcc],bcf=[bcf],bd2=[bd2],bd5=[bd5],bd8=[bd8],be0=[be0,bdd],be6=[be6,be3],bec=[bec,be9],bef=[bef],bf2=[bf2],bf8=[bf8,bf5],c00=[c00,bfb],c03=[c03],c09=[c09,c06],c0c=[c0c],c0f=[c0f]} 2020-02-05 22:46:53.787106 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] find_object_context 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 @7b5 snapset c0f=[c0f,c0c,c09,c06,c03,c00,bfb,bf8,bf5,bf2,bef,bec,be9,be6,be3,be0,bdd,bd8,bd5,bd2,bcf,bcc,bc9,bc6,bc3,bc0,bbd,bba,bb7,bad,ba7,ba1,b9b,b95,b8f,b89,b83,b7d,b77,b71,b6b,b65,b5f,b59,b53,b4d,b47,b41,b3b,b35,b2f,b29,b23,b1d,b17,b0f,b09,b03,afd,af7,af1,aeb,ae5,adf,ad9,ad3,acd,ac7,ac1,abb,ab5,aaf,aa9,aa3,a9d,a97,a91,a8b,a85,a7f,a79,a73,a6d,a67,a61,a5b,a55,a4f,a49,a43,a3d,a37,a31,a2b,a25,a1f,a19,a13,a0d,a07,a01,9fb,9f5,9ef,9e9,9e3,9dd,9d7,9d1,9cb,9c5,9bf,9b9,9b3,9ad,9a7,9a1,99b,995,98f,989,983,97d,977,971,96b,965,95f,959,953,94d,947,941,93b,935,92f,929,923,91d,917,911,90b,905,8ff,8f9,8f3,8ed,8e7,8e1,8db,8d5,8cf,8c9,8c3,8bd,8b7,8b1,8ab,8a5,89f,899,893,88d,887,881,87b,875,86f,869,863,85d,857,851,84b,845,83f,839,833,82d,827,821,81b,815,80f,809,803,7fd,7f7,7f1,7eb,7e5,7df,7d9,7d3,7cd,7c7,7c1,7bb,7b5,7af,7a9,7a3,79d,797,791,78b,785,77f,779,773,76d,767,761,75b,755,74f,749,743,73d,737,731,72b,725,71f,719,713,70d]:{70d=[70d],719=[719,713],71f=[71f],72b=[72b,725],731=[731],73d=[73d,737],743=[743],74f=[74f,749],755=[755],761=[761,75b],767=[767],773=[773,76d],77f=[77f,779],785=[785],78b=[78b],791=[791],79d=[79d,797],7a9=[7a9,7a3],7b5=[7b5,7af],7c1=[7c1,7bb],7cd=[7cd,7c7],7d3=[7d3],7df=[7df,7d9],7eb=[7eb,7e5],7f1=[7f1],7fd=[7fd,7f7],803=[803],80f=[80f,809],81b=[81b,815],827=[827,821],82d=[82d],833=[833],839=[839],83f=[83f],84b=[84b,845],851=[851],857=[857],85d=[85d],863=[863],869=[869],875=[875,86f],87b=[87b],881=[881],88d=[88d,887],899=[899,893],89f=[89f],8a5=[8a5],8ab=[8ab],8b1=[8b1],8bd=[8bd,8b7],8c9=[8c9,8c3],8cf=[8cf],8d5=[8d5],8db=[8db],8e1=[8e1],8ed=[8ed,8e7],8f9=[8f9,8f3],8ff=[8ff],905=[905],911=[911,90b],917=[917],923=[923,91d],92f=[92f,929],935=[935],941=[941,93b],94d=[94d,947],959=[959,953],95f=[95f],96b=[96b,965],971=[971],977=[977],983=[983,97d],989=[989],995=[995,98f],99b=[99b],9a1=[9a1],9ad=[9ad,9a7],9b9=[9b9,9b3],9c5=[9c5,9bf],9cb=[9cb],9d7=[9d7,9d1],9dd=[9dd],9e9=[9e9,9e3],9f5=[9f5,9ef],a01=[a01,9fb],a07=[a07],a0d=[a0d],a19=[a19,a13],a1f=[a1f],a25=[a25],a2b=[a2b],a31=[a31],a37=[a37],a43=[a43,a3d],a49=[a49],a55=[a55,a4f],a5b=[a5b],a61=[a61],a67=[a67],a6d=[a6d],a79=[a79,a73],a7f=[a7f],a8b=[a8b,a85],a91=[a91],a9d=[a9d,a97],aa3=[aa3],aa9=[aa9],aaf=[aaf],abb=[abb,ab5],ac7=[ac7,ac1],acd=[acd],ad9=[ad9,ad3],adf=[adf],ae5=[ae5],af1=[af1,aeb],afd=[afd,af7],b03=[b03],b0f=[b0f,b09],b1d=[b1d,b17],b23=[b23],b2f=[b2f,b29],b35=[b35],b41=[b41,b3b],b47=[b47],b53=[b53,b4d],b59=[b59],b5f=[b5f],b6b=[b6b,b65],b71=[b71],b77=[b77],b7d=[b7d],b83=[b83],b89=[b89],b95=[b95,b8f],b9b=[b9b],ba1=[ba1],ba7=[ba7],bad=[bad],bb7=[bb7],bbd=[bbd,bba],bc3=[bc3,bc0],bc9=[bc9,bc6],bcc=[bcc],bcf=[bcf],bd2=[bd2],bd5=[bd5],bd8=[bd8],be0=[be0,bdd],be6=[be6,be3],bec=[bec,be9],bef=[bef],bf2=[bf2],bf8=[bf8,bf5],c00=[c00,bfb],c03=[c03],c09=[c09,c06],c0c=[c0c],c0f=[c0f]} 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 HIT 2020-02-05 22:46:53.787323 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_op obc obc(2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 rwstate(excl n=1 w=0)) 2020-02-05 22:46:53.787347 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] execute_ctx 0x561330e4c000 2020-02-05 22:46:53.787412 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] execute_ctx 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 8388608] ov 56424'2944715 2020-02-05 22:46:53.787432 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean]  taking ondisk_read_lock 2020-02-05 22:46:53.787449 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_osd_op 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 8388608] 2020-02-05 22:46:53.787483 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_osd_op  copy-get max 8388608 2020-02-05 22:46:53.787515 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean]  got attrs 2020-02-05 22:46:53.807853 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean]  got data 2020-02-05 22:46:53.808034 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean]  got reqids 2020-02-05 22:46:53.808051 7ff1c3749700 20 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean]  cursor.is_complete=1 0 attrs 4194304 bytes 0 omap header bytes 0 omap data bytes in 0 keys 0 reqids 2020-02-05 22:46:53.808085 7ff1c3749700 10 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean]  dropping ondisk_read_lock 2020-02-05 22:46:53.808107 7ff1c3749700 15 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] do_osd_op_effects osd.9 con 0x56135d9e5000 2020-02-05 22:46:53.808132 7ff1c3749700 15 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] log_op_stats osd_op(osd.9.60026:836317 2.c 2:311c8802:::rbd_data.09df7e2ae8944a.0000000000000000:7b5 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e60042) v8 inb 0 outb 4194423 lat 19.627387 2020-02-05 22:46:53.808171 7ff1c3749700 15 osd.1 pg_epoch: 60042 pg[2.c( v 60042'3048392 (58743'3046874,60042'3048392] local-lis/les=60041/60042 n=5112 ec=1/1 lis/c 60041/60041 les/c/f 60042/60042/38237 60041/60041/60034) [1,8] r=0 lpr=60041 crt=60042'3048392 lcod 60042'3048390 mlcod 60042'3048390 active+clean] publish_stats_to_osd 60042:3424975 2020-02-05 22:46:53.808292 7ff1c3749700 10 osd.1 60042 dequeue_op 0x561362c80bc0 finish On 03/02/2020 15.48, Jan Pekař - Imatic wrote:

Hi all,

I have small cluster and yesterday I tried to mount older RBD snapshot torecover data. (I have approx. 230 daily snapshots of one RBD image on my small ceph).

After I did mount and ls operation, cluster was stuck and I notice that 2of my OSD's eaten CPU and raise in memory usage (more than 4x more memory than usual). Cluster was unusable after that and it didn't help stopping rbd mount operation and restarting OSD's etc.

Strange is, that it affects only 2 of my OSD's - one is on NVME and has cache PG's mapped, other one is rotational HDD. IOPS are normal, only CPU and memory usage are unusual.

I noticed hundred thousands of slow requests (on idle cluster) so I startto debug OSD operations. You can find OSD 8 log below. OSD 9 contains nvme cached PG's and there is nothing unusual in standard log. In the middle of log I raised debug level to 10.

It look's to me, that it is trying to do one operation with one snapshotted RBD object in PG 2.b (rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3) in a loop.

Is there any way how to stop that? What is causing the problem? Can I somehow flush OSD's "journals" or "queues" to stop current operation and make cluster usable again? Is there any way how I can access old snapshot data?

Problem started on Luminous 12.2.12 and upgrade to 12.2.13 didn't help.

Thank you Jan Pekar

2020-02-03 15:09:22.470277 7f51f6af7700  0 log_channel(cluster) log [WRN] : 30966 slow requests, 5 included below; oldest blocked for > 37.505482 secs 2020-02-03 15:09:22.470336 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request 37.503404 seconds old, received at 2020-02-03 15:08:44.956422: osd_op(osd.9.59923:2670991 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg 2020-02-03 15:09:22.470360 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request 37.500695 seconds old, received at 2020-02-03 15:08:44.959132: osd_op(osd.9.59923:2671023 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg 2020-02-03 15:09:22.470366 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request 37.498013 seconds old, received at 2020-02-03 15:08:44.961814: osd_op(osd.9.59923:2671055 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg 2020-02-03 15:09:22.470371 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request 37.494836 seconds old, received at 2020-02-03 15:08:44.964990: osd_op(osd.9.59923:2671087 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg 2020-02-03 15:09:22.470376 7f51f6af7700  0 log_channel(cluster) log [WRN] : slow request 37.491950 seconds old, received at 2020-02-03 15:08:44.967877: osd_op(osd.9.59923:2671119 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg ... 2020-02-03 15:09:35.537259 7f51ddac5700  0 osd.8 59939 do_command r=0 2020-02-03 15:09:35.588753 7f51ddac5700  0 osd.8 59939 do_command r=0 Set debug_osd to 10/10 2020-02-03 15:09:35.588814 7f51ddac5700  0 log_channel(cluster) log [INF] : Set debug_osd to 10/10 2020-02-03 15:09:35.597667 7f51f0945700  2 osd.8 59939 ms_handle_reset con 0x5647736ac800 session 0x564785519e00 2020-02-03 15:09:35.599056 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  dropping ondisk_read_lock 2020-02-03 15:09:35.599209 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5800 finish 2020-02-03 15:09:35.599250 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d59c0 prio 63 cost 29 latency 22.224501 osd_op(osd.9.59923:2779946 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] 2020-02-03 15:09:35.599293 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] _handle_message: 0x5647873d59c0 2020-02-03 15:09:35.599347 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_op osd_op(osd.9.59923:2779946 2.b 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 may_read -> write-ordered flags ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 2020-02-03 15:09:35.599398 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} map_snapid_to_clone=true 2020-02-03 15:09:35.599506 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 2020-02-03 15:09:35.599606 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: found obc in cache: 0x564785fb3880 2020-02-03 15:09:35.613724 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: 0x564785fb3880 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none n=0 w=0) oi: 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046 osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 uv 7633387 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160 snapset: c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2020-02-03 15:09:35.613881 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT 2020-02-03 15:09:35.614024 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 0x5647843a3000 2020-02-03 15:09:35.614053 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] ov 58682'4215046 2020-02-03 15:09:35.614077 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  taking ondisk_read_lock 2020-02-03 15:09:35.614095 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] 2020-02-03 15:09:35.614112 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max 8388608 2020-02-03 15:09:35.637123 7f51f72f8700 10 osd.8 59939 tick 2020-02-03 15:09:35.637149 7f51f72f8700 10 osd.8 59939 do_waiters -- start 2020-02-03 15:09:35.637152 7f51f72f8700 10 osd.8 59939 do_waiters -- finish 2020-02-03 15:09:35.645761 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  dropping ondisk_read_lock 2020-02-03 15:09:35.645887 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d59c0 finish 2020-02-03 15:09:35.645924 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5b80 prio 63 cost 29 latency 22.271117 osd_op(osd.9.59923:2779947 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] 2020-02-03 15:09:35.645960 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] _handle_message: 0x5647873d5b80 2020-02-03 15:09:35.646007 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_op osd_op(osd.9.59923:2779947 2.b 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 may_read -> write-ordered flags ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 2020-02-03 15:09:35.646047 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} map_snapid_to_clone=true 2020-02-03 15:09:35.646145 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 2020-02-03 15:09:35.646239 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: found obc in cache: 0x564785fb3880 2020-02-03 15:09:35.646255 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: 0x564785fb3880 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none n=0 w=0) oi: 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046 osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 uv 7633387 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160 snapset: c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2020-02-03 15:09:35.646359 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT 2020-02-03 15:09:35.646469 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 0x5647843a3000 2020-02-03 15:09:35.646494 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] ov 58682'4215046 2020-02-03 15:09:35.646513 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  taking ondisk_read_lock 2020-02-03 15:09:35.646530 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] 2020-02-03 15:09:35.646547 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max 8388608 2020-02-03 15:09:35.681653 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  dropping ondisk_read_lock 2020-02-03 15:09:35.681780 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5b80 finish 2020-02-03 15:09:35.681817 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5d40 prio 63 cost 29 latency 22.306952 osd_op(osd.9.59923:2779948 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] 2020-02-03 15:09:35.681851 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] _handle_message: 0x5647873d5d40 2020-02-03 15:09:35.681898 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_op osd_op(osd.9.59923:2779948 2.b 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 may_read -> write-ordered flags ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 2020-02-03 15:09:35.681937 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} map_snapid_to_clone=true 2020-02-03 15:09:35.682035 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 2020-02-03 15:09:35.682128 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: found obc in cache: 0x564785fb3880 2020-02-03 15:09:35.682144 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: 0x564785fb3880 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none n=0 w=0) oi: 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046 osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 uv 7633387 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160 snapset: c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2020-02-03 15:09:35.682245 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT 2020-02-03 15:09:35.682356 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 0x5647843a3000 2020-02-03 15:09:35.682385 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] ov 58682'4215046 2020-02-03 15:09:35.682404 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  taking ondisk_read_lock 2020-02-03 15:09:35.682420 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] 2020-02-03 15:09:35.682437 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max 8388608 2020-02-03 15:09:35.714076 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean]  dropping ondisk_read_lock 2020-02-03 15:09:35.714196 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5d40 finish

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- ============ Ing. Jan Pekař jan.pekar@imatic.cz | +420603811737 ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 http://www.imatic.cz ============ --