singular PG is snaptrimming for ever since pacific -> reef 18.2.6 upgrade
Hi, we have one cluster where one PG that seems to be snaptrimming forever. It sometimes seem to hinder other PGs to on the same main OSD to snaptrim as well. We've synced out the OSD but it still keeps happening. It was gone sometime in between but now it is back again with the same PG. And the constant factor is always PG 3.c1c. Anyone got an idea what I can do? Is it possible to "sync out and recreate" a PG, like I can do with an OSD? Here is some output from the pg list Cheers Boris # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:12:21 PM UTC 2025 3.1b active+clean+snaptrim_wait 3m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 3m [156,117,32]p156 3.65d active+clean+snaptrim_wait 3m [156,263,9]p156 3.721 active+clean+snaptrim_wait 3m [156,23,190]p156 3.849 active+clean+snaptrim_wait 3m [156,135,31]p156 3.abb active+clean+snaptrim_wait 3m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 3m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 3m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 3m [156,266,281]p156 3.103f active+clean+snaptrim_wait 3m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 3m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 3m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 3m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 3m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 3m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 3m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 3m [156,212,134]p156 3.1e74 active+clean+snaptrim 3m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 3m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 3m [156,190,283]p156 # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:47:38 PM UTC 2025 3.1b active+clean+snaptrim_wait 2m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 2m [156,117,32]p156 3.65d active+clean+snaptrim_wait 2m [156,263,9]p156 3.721 active+clean+snaptrim_wait 2m [156,23,190]p156 3.849 active+clean+snaptrim_wait 2m [156,135,31]p156 3.abb active+clean+snaptrim_wait 2m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 2m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 2m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 2m [156,266,281]p156 3.103f active+clean+snaptrim_wait 2m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 2m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 2m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 2m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 2m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 2m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 2m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 2m [156,212,134]p156 3.1e74 active+clean+snaptrim 2m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 2m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 2m [156,190,283]p156 # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 04:24:55 PM UTC 2025 3.1b active+clean+snaptrim_wait 55s [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 55s [156,117,32]p156 3.65d active+clean+snaptrim_wait 55s [156,263,9]p156 3.721 active+clean+snaptrim_wait 55s [156,23,190]p156 3.849 active+clean+snaptrim_wait 55s [156,135,31]p156 3.abb active+clean+snaptrim_wait 55s [156,70,262]p156 3.c1c active+clean+snaptrim 34s [156,111,11]p156 3.f83 active+clean+snaptrim_wait 55s [156,119,97]p156 3.f8a active+clean+snaptrim_wait 55s [156,266,281]p156 3.103f active+clean+snaptrim_wait 55s [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 55s [156,135,280]p156 3.14ea active+clean+snaptrim_wait 55s [156,118,103]p156 3.1688 active+clean+snaptrim_wait 55s [156,221,98]p156 3.1848 active+clean+snaptrim_wait 55s [156,268,68]p156 3.18ca active+clean+snaptrim_wait 55s [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 55s [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 55s [156,212,134]p156 3.1e74 active+clean+snaptrim_wait 34s [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 55s [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 55s [156,190,283]p156 # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Mon May 5 06:32:53 AM UTC 2025 3.c1c active+clean+snaptrim 41h [165,111,25]p165 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
The mentioned PG seems also to take ages to deep scrub. The other PGs seem to take roughly 10min to deep scrub, this PG is scrubbing since over an hour. It also got a lot more omap_bytes and omap_keys root@ceph-rbd-mon1:~# (ceph pg ls | head -n1 | awk '{print $1, $6, $7, $8, $11, $13, $14 }'; ceph pg ls | grep ^3\. | sort -nk8 | tail | awk '{print $1, $6, $7, $8, $11, $13, $14 }')| column -t PG BYTES OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 3.1323 45850562560 2042 40 active+clean 8935596'655909002 8935596:696646107 3.1f8e 46328389632 2408 40 active+clean 8935596'625068094 8935596:699142583 3.b57 43968541474 1143 40 active+clean 8935596'395996789 8935596:548258860 3.532 45439124004 1213 42 active+clean 8935596'535018014 8935596:621720921 3.1837 45275007300 1928 43 active+clean 8935596'369343024 8935596:485257282 3.97d 45998831398 2272 43 active+clean 8935596'341398404 8935596:463788682 3.8f4 45190209204 1405 44 active+clean 8935596'379178010 8935596:509024378 3.1237 45146136576 2097 45 active+clean 8935596'368790387 8935596:876062378 3.101d 45518981120 1296 47 active+clean 8935596'310619462 8935596:489401818 3.c1c 45024581632 263621 9078 active+clean+scrubbing+deep 8935596'371406402 8935596:580117577 root@ceph-rbd-mon1:~# (ceph pg ls | head -n1 | awk '{print $1, $6, $7, $8, $11, $13, $14 }'; ceph pg ls | grep ^3\. | sort -nk7 | tail | awk '{print $1, $6, $7, $8, $11, $13, $14 }')| column -t PG BYTES OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 3.1837 45275007300 1928 43 active+clean 8935596'369343048 8935596:485257403 3.28c 46611274402 1930 29 active+clean 8935596'343110593 8935596:438952996 3.aa 44759281456 1950 34 active+clean 8935596'415139727 8935596:516645874 3.1323 45850562560 2042 40 active+clean 8935596'655909072 8935596:696646714 3.1237 45146136576 2097 45 active+clean 8935596'368790391 8935596:876062474 3.a56 45746631280 2144 30 active+clean 8935596'305003417 8935596:431604881 3.97d 45998610214 2272 43 active+clean 8935596'341398488 8935596:463788934 3.9a4 44538237970 2314 39 active+clean 8935596'385476487 8935596:545205675 3.1f8e 46328389632 2408 40 active+clean 8935596'625068112 8935596:699142847 3.c1c 45024581632 263621 9078 active+clean+scrubbing+deep 8935596'371406432 8935596:580119804 Am Mo., 5. Mai 2025 um 09:27 Uhr schrieb Boris <bb@kervyn.de>:
Hi,
we have one cluster where one PG that seems to be snaptrimming forever. It sometimes seem to hinder other PGs to on the same main OSD to snaptrim as well.
We've synced out the OSD but it still keeps happening. It was gone sometime in between but now it is back again with the same PG.
And the constant factor is always PG 3.c1c.
Anyone got an idea what I can do? Is it possible to "sync out and recreate" a PG, like I can do with an OSD?
Here is some output from the pg list
Cheers Boris
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:12:21 PM UTC 2025 3.1b active+clean+snaptrim_wait 3m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 3m [156,117,32]p156 3.65d active+clean+snaptrim_wait 3m [156,263,9]p156 3.721 active+clean+snaptrim_wait 3m [156,23,190]p156 3.849 active+clean+snaptrim_wait 3m [156,135,31]p156 3.abb active+clean+snaptrim_wait 3m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 3m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 3m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 3m [156,266,281]p156 3.103f active+clean+snaptrim_wait 3m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 3m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 3m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 3m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 3m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 3m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 3m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 3m [156,212,134]p156 3.1e74 active+clean+snaptrim 3m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 3m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 3m [156,190,283]p156
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:47:38 PM UTC 2025 3.1b active+clean+snaptrim_wait 2m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 2m [156,117,32]p156 3.65d active+clean+snaptrim_wait 2m [156,263,9]p156 3.721 active+clean+snaptrim_wait 2m [156,23,190]p156 3.849 active+clean+snaptrim_wait 2m [156,135,31]p156 3.abb active+clean+snaptrim_wait 2m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 2m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 2m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 2m [156,266,281]p156 3.103f active+clean+snaptrim_wait 2m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 2m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 2m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 2m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 2m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 2m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 2m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 2m [156,212,134]p156 3.1e74 active+clean+snaptrim 2m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 2m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 2m [156,190,283]p156
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 04:24:55 PM UTC 2025 3.1b active+clean+snaptrim_wait 55s [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 55s [156,117,32]p156 3.65d active+clean+snaptrim_wait 55s [156,263,9]p156 3.721 active+clean+snaptrim_wait 55s [156,23,190]p156 3.849 active+clean+snaptrim_wait 55s [156,135,31]p156 3.abb active+clean+snaptrim_wait 55s [156,70,262]p156 3.c1c active+clean+snaptrim 34s [156,111,11]p156 3.f83 active+clean+snaptrim_wait 55s [156,119,97]p156 3.f8a active+clean+snaptrim_wait 55s [156,266,281]p156 3.103f active+clean+snaptrim_wait 55s [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 55s [156,135,280]p156 3.14ea active+clean+snaptrim_wait 55s [156,118,103]p156 3.1688 active+clean+snaptrim_wait 55s [156,221,98]p156 3.1848 active+clean+snaptrim_wait 55s [156,268,68]p156 3.18ca active+clean+snaptrim_wait 55s [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 55s [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 55s [156,212,134]p156 3.1e74 active+clean+snaptrim_wait 34s [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 55s [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 55s [156,190,283]p156
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Mon May 5 06:32:53 AM UTC 2025 3.c1c active+clean+snaptrim 41h [165,111,25]p165 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
So I went through some other clusters we have and it always shows basically the same pattern. One PG with disproportional large OMAP data. These are all flash only rbd clusters of different size, which where set up in different initial versions. All of them are Reef 18.2.6. $ (ceph pg ls | head -n1 | awk '{print $1, $7, $8, $11, $13, $14 }'; ceph pg ls-by-pool rbd | sort -nk7 | tail -n3 | awk '{print $1, $7, $8, $11, $13, $14 }')| column -t PG OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 1.1b1 1356 36 active+clean 432394'543016527 432394:691327003 1.ea 1951 57 active+clean 432394'607403423 432394:783494415 1.1c 59094 2044 active+clean 432394'528017978 432394:726223821 $ (ceph pg ls | head -n1 | awk '{print $1, $7, $8, $11, $13, $14 }'; ceph pg ls-by-pool rbd | sort -nk7 | tail -n3 | awk '{print $1, $7, $8, $11, $13, $14 }')| column -t PG OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 2.357 1374 29 active+clean 122007'222637136 122007:294907884 2.291 1517 38 active+clean 122007'196789540 122007:271705492 2.1c 32396 1118 active+clean 122007'201282577 122007:290533282 $ (ceph pg ls | head -n1 | awk '{print $1, $7, $8, $11, $13, $14 }'; ceph pg ls-by-pool rbd | sort -nk7 | tail -n3 | awk '{print $1, $7, $8, $11, $13, $14 }')| column -t PG OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 1.12e 1423 37 active+clean 86947'526495372 86947:597361344 1.304 1490 38 active+clean 86947'356338340 86947:427439006 1.1c 38385 1329 active+clean 86947'449102542 86947:525465853 (ceph pg ls | head -n1 | awk '{print $1, $7, $8, $11, $13, $14 }'; ceph pg ls-by-pool rbd | sort -nk7 | tail -n3 | awk '{print $1, $7, $8, $11, $13, $14 }')| column -t PG OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 2.13f 425 27 active+clean 46780'456499 46828:588064 2.1e2 427 27 active+clean 46588'351645 46828:492723 2.1c 11718 410 active+clean 46828'1366432 46828:4124268 Am Mo., 5. Mai 2025 um 10:00 Uhr schrieb Boris <bb@kervyn.de>:
The mentioned PG seems also to take ages to deep scrub. The other PGs seem to take roughly 10min to deep scrub, this PG is scrubbing since over an hour.
It also got a lot more omap_bytes and omap_keys root@ceph-rbd-mon1:~# (ceph pg ls | head -n1 | awk '{print $1, $6, $7, $8, $11, $13, $14 }'; ceph pg ls | grep ^3\. | sort -nk8 | tail | awk '{print $1, $6, $7, $8, $11, $13, $14 }')| column -t PG BYTES OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 3.1323 45850562560 2042 40 active+clean 8935596'655909002 8935596:696646107 3.1f8e 46328389632 2408 40 active+clean 8935596'625068094 8935596:699142583 3.b57 43968541474 1143 40 active+clean 8935596'395996789 8935596:548258860 3.532 45439124004 1213 42 active+clean 8935596'535018014 8935596:621720921 3.1837 45275007300 1928 43 active+clean 8935596'369343024 8935596:485257282 3.97d 45998831398 2272 43 active+clean 8935596'341398404 8935596:463788682 3.8f4 45190209204 1405 44 active+clean 8935596'379178010 8935596:509024378 3.1237 45146136576 2097 45 active+clean 8935596'368790387 8935596:876062378 3.101d 45518981120 1296 47 active+clean 8935596'310619462 8935596:489401818 3.c1c 45024581632 263621 9078 active+clean+scrubbing+deep 8935596'371406402 8935596:580117577 root@ceph-rbd-mon1:~# (ceph pg ls | head -n1 | awk '{print $1, $6, $7, $8, $11, $13, $14 }'; ceph pg ls | grep ^3\. | sort -nk7 | tail | awk '{print $1, $6, $7, $8, $11, $13, $14 }')| column -t PG BYTES OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 3.1837 45275007300 1928 43 active+clean 8935596'369343048 8935596:485257403 3.28c 46611274402 1930 29 active+clean 8935596'343110593 8935596:438952996 3.aa 44759281456 1950 34 active+clean 8935596'415139727 8935596:516645874 3.1323 45850562560 2042 40 active+clean 8935596'655909072 8935596:696646714 3.1237 45146136576 2097 45 active+clean 8935596'368790391 8935596:876062474 3.a56 45746631280 2144 30 active+clean 8935596'305003417 8935596:431604881 3.97d 45998610214 2272 43 active+clean 8935596'341398488 8935596:463788934 3.9a4 44538237970 2314 39 active+clean 8935596'385476487 8935596:545205675 3.1f8e 46328389632 2408 40 active+clean 8935596'625068112 8935596:699142847 3.c1c 45024581632 263621 9078 active+clean+scrubbing+deep 8935596'371406432 8935596:580119804
Am Mo., 5. Mai 2025 um 09:27 Uhr schrieb Boris <bb@kervyn.de>:
Hi,
we have one cluster where one PG that seems to be snaptrimming forever. It sometimes seem to hinder other PGs to on the same main OSD to snaptrim as well.
We've synced out the OSD but it still keeps happening. It was gone sometime in between but now it is back again with the same PG.
And the constant factor is always PG 3.c1c.
Anyone got an idea what I can do? Is it possible to "sync out and recreate" a PG, like I can do with an OSD?
Here is some output from the pg list
Cheers Boris
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:12:21 PM UTC 2025 3.1b active+clean+snaptrim_wait 3m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 3m [156,117,32]p156 3.65d active+clean+snaptrim_wait 3m [156,263,9]p156 3.721 active+clean+snaptrim_wait 3m [156,23,190]p156 3.849 active+clean+snaptrim_wait 3m [156,135,31]p156 3.abb active+clean+snaptrim_wait 3m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 3m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 3m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 3m [156,266,281]p156 3.103f active+clean+snaptrim_wait 3m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 3m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 3m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 3m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 3m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 3m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 3m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 3m [156,212,134]p156 3.1e74 active+clean+snaptrim 3m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 3m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 3m [156,190,283]p156
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:47:38 PM UTC 2025 3.1b active+clean+snaptrim_wait 2m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 2m [156,117,32]p156 3.65d active+clean+snaptrim_wait 2m [156,263,9]p156 3.721 active+clean+snaptrim_wait 2m [156,23,190]p156 3.849 active+clean+snaptrim_wait 2m [156,135,31]p156 3.abb active+clean+snaptrim_wait 2m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 2m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 2m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 2m [156,266,281]p156 3.103f active+clean+snaptrim_wait 2m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 2m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 2m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 2m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 2m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 2m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 2m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 2m [156,212,134]p156 3.1e74 active+clean+snaptrim 2m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 2m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 2m [156,190,283]p156
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 04:24:55 PM UTC 2025 3.1b active+clean+snaptrim_wait 55s [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 55s [156,117,32]p156 3.65d active+clean+snaptrim_wait 55s [156,263,9]p156 3.721 active+clean+snaptrim_wait 55s [156,23,190]p156 3.849 active+clean+snaptrim_wait 55s [156,135,31]p156 3.abb active+clean+snaptrim_wait 55s [156,70,262]p156 3.c1c active+clean+snaptrim 34s [156,111,11]p156 3.f83 active+clean+snaptrim_wait 55s [156,119,97]p156 3.f8a active+clean+snaptrim_wait 55s [156,266,281]p156 3.103f active+clean+snaptrim_wait 55s [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 55s [156,135,280]p156 3.14ea active+clean+snaptrim_wait 55s [156,118,103]p156 3.1688 active+clean+snaptrim_wait 55s [156,221,98]p156 3.1848 active+clean+snaptrim_wait 55s [156,268,68]p156 3.18ca active+clean+snaptrim_wait 55s [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 55s [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 55s [156,212,134]p156 3.1e74 active+clean+snaptrim_wait 34s [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 55s [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 55s [156,190,283]p156
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Mon May 5 06:32:53 AM UTC 2025 3.c1c active+clean+snaptrim 41h [165,111,25]p165 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
The OMAPs seem to be a red herring. It is just the `rbd_directory` which is just a list of rbd names referencing to an ID and back. I've now offline compacted the OSDs. The deep-scrub still takes around 100minutes, compared to the other PGs. I hope this solves the snaptrim problem, because the during the long running snap trim the latency of the cluster slowly degrades. Am Mo., 5. Mai 2025 um 13:50 Uhr schrieb Boris <bb@kervyn.de>:
So I went through some other clusters we have and it always shows basically the same pattern.
One PG with disproportional large OMAP data. These are all flash only rbd clusters of different size, which where set up in different initial versions. All of them are Reef 18.2.6.
$ (ceph pg ls | head -n1 | awk '{print $1, $7, $8, $11, $13, $14 }'; ceph pg ls-by-pool rbd | sort -nk7 | tail -n3 | awk '{print $1, $7, $8, $11, $13, $14 }')| column -t PG OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 1.1b1 1356 36 active+clean 432394'543016527 432394:691327003 1.ea 1951 57 active+clean 432394'607403423 432394:783494415 1.1c 59094 2044 active+clean 432394'528017978 432394:726223821
$ (ceph pg ls | head -n1 | awk '{print $1, $7, $8, $11, $13, $14 }'; ceph pg ls-by-pool rbd | sort -nk7 | tail -n3 | awk '{print $1, $7, $8, $11, $13, $14 }')| column -t PG OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 2.357 1374 29 active+clean 122007'222637136 122007:294907884 2.291 1517 38 active+clean 122007'196789540 122007:271705492 2.1c 32396 1118 active+clean 122007'201282577 122007:290533282
$ (ceph pg ls | head -n1 | awk '{print $1, $7, $8, $11, $13, $14 }'; ceph pg ls-by-pool rbd | sort -nk7 | tail -n3 | awk '{print $1, $7, $8, $11, $13, $14 }')| column -t PG OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 1.12e 1423 37 active+clean 86947'526495372 86947:597361344 1.304 1490 38 active+clean 86947'356338340 86947:427439006 1.1c 38385 1329 active+clean 86947'449102542 86947:525465853
(ceph pg ls | head -n1 | awk '{print $1, $7, $8, $11, $13, $14 }'; ceph pg ls-by-pool rbd | sort -nk7 | tail -n3 | awk '{print $1, $7, $8, $11, $13, $14 }')| column -t PG OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 2.13f 425 27 active+clean 46780'456499 46828:588064 2.1e2 427 27 active+clean 46588'351645 46828:492723 2.1c 11718 410 active+clean 46828'1366432 46828:4124268
Am Mo., 5. Mai 2025 um 10:00 Uhr schrieb Boris <bb@kervyn.de>:
The mentioned PG seems also to take ages to deep scrub. The other PGs seem to take roughly 10min to deep scrub, this PG is scrubbing since over an hour.
It also got a lot more omap_bytes and omap_keys root@ceph-rbd-mon1:~# (ceph pg ls | head -n1 | awk '{print $1, $6, $7, $8, $11, $13, $14 }'; ceph pg ls | grep ^3\. | sort -nk8 | tail | awk '{print $1, $6, $7, $8, $11, $13, $14 }')| column -t PG BYTES OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 3.1323 45850562560 2042 40 active+clean 8935596'655909002 8935596:696646107 3.1f8e 46328389632 2408 40 active+clean 8935596'625068094 8935596:699142583 3.b57 43968541474 1143 40 active+clean 8935596'395996789 8935596:548258860 3.532 45439124004 1213 42 active+clean 8935596'535018014 8935596:621720921 3.1837 45275007300 1928 43 active+clean 8935596'369343024 8935596:485257282 3.97d 45998831398 2272 43 active+clean 8935596'341398404 8935596:463788682 3.8f4 45190209204 1405 44 active+clean 8935596'379178010 8935596:509024378 3.1237 45146136576 2097 45 active+clean 8935596'368790387 8935596:876062378 3.101d 45518981120 1296 47 active+clean 8935596'310619462 8935596:489401818 3.c1c 45024581632 263621 9078 active+clean+scrubbing+deep 8935596'371406402 8935596:580117577 root@ceph-rbd-mon1:~# (ceph pg ls | head -n1 | awk '{print $1, $6, $7, $8, $11, $13, $14 }'; ceph pg ls | grep ^3\. | sort -nk7 | tail | awk '{print $1, $6, $7, $8, $11, $13, $14 }')| column -t PG BYTES OMAP_BYTES* OMAP_KEYS* STATE VERSION REPORTED 3.1837 45275007300 1928 43 active+clean 8935596'369343048 8935596:485257403 3.28c 46611274402 1930 29 active+clean 8935596'343110593 8935596:438952996 3.aa 44759281456 1950 34 active+clean 8935596'415139727 8935596:516645874 3.1323 45850562560 2042 40 active+clean 8935596'655909072 8935596:696646714 3.1237 45146136576 2097 45 active+clean 8935596'368790391 8935596:876062474 3.a56 45746631280 2144 30 active+clean 8935596'305003417 8935596:431604881 3.97d 45998610214 2272 43 active+clean 8935596'341398488 8935596:463788934 3.9a4 44538237970 2314 39 active+clean 8935596'385476487 8935596:545205675 3.1f8e 46328389632 2408 40 active+clean 8935596'625068112 8935596:699142847 3.c1c 45024581632 263621 9078 active+clean+scrubbing+deep 8935596'371406432 8935596:580119804
Am Mo., 5. Mai 2025 um 09:27 Uhr schrieb Boris <bb@kervyn.de>:
Hi,
we have one cluster where one PG that seems to be snaptrimming forever. It sometimes seem to hinder other PGs to on the same main OSD to snaptrim as well.
We've synced out the OSD but it still keeps happening. It was gone sometime in between but now it is back again with the same PG.
And the constant factor is always PG 3.c1c.
Anyone got an idea what I can do? Is it possible to "sync out and recreate" a PG, like I can do with an OSD?
Here is some output from the pg list
Cheers Boris
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:12:21 PM UTC 2025 3.1b active+clean+snaptrim_wait 3m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 3m [156,117,32]p156 3.65d active+clean+snaptrim_wait 3m [156,263,9]p156 3.721 active+clean+snaptrim_wait 3m [156,23,190]p156 3.849 active+clean+snaptrim_wait 3m [156,135,31]p156 3.abb active+clean+snaptrim_wait 3m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 3m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 3m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 3m [156,266,281]p156 3.103f active+clean+snaptrim_wait 3m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 3m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 3m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 3m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 3m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 3m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 3m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 3m [156,212,134]p156 3.1e74 active+clean+snaptrim 3m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 3m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 3m [156,190,283]p156
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:47:38 PM UTC 2025 3.1b active+clean+snaptrim_wait 2m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 2m [156,117,32]p156 3.65d active+clean+snaptrim_wait 2m [156,263,9]p156 3.721 active+clean+snaptrim_wait 2m [156,23,190]p156 3.849 active+clean+snaptrim_wait 2m [156,135,31]p156 3.abb active+clean+snaptrim_wait 2m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 2m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 2m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 2m [156,266,281]p156 3.103f active+clean+snaptrim_wait 2m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 2m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 2m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 2m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 2m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 2m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 2m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 2m [156,212,134]p156 3.1e74 active+clean+snaptrim 2m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 2m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 2m [156,190,283]p156
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 04:24:55 PM UTC 2025 3.1b active+clean+snaptrim_wait 55s [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 55s [156,117,32]p156 3.65d active+clean+snaptrim_wait 55s [156,263,9]p156 3.721 active+clean+snaptrim_wait 55s [156,23,190]p156 3.849 active+clean+snaptrim_wait 55s [156,135,31]p156 3.abb active+clean+snaptrim_wait 55s [156,70,262]p156 3.c1c active+clean+snaptrim 34s [156,111,11]p156 3.f83 active+clean+snaptrim_wait 55s [156,119,97]p156 3.f8a active+clean+snaptrim_wait 55s [156,266,281]p156 3.103f active+clean+snaptrim_wait 55s [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 55s [156,135,280]p156 3.14ea active+clean+snaptrim_wait 55s [156,118,103]p156 3.1688 active+clean+snaptrim_wait 55s [156,221,98]p156 3.1848 active+clean+snaptrim_wait 55s [156,268,68]p156 3.18ca active+clean+snaptrim_wait 55s [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 55s [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 55s [156,212,134]p156 3.1e74 active+clean+snaptrim_wait 34s [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 55s [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 55s [156,190,283]p156
# date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Mon May 5 06:32:53 AM UTC 2025 3.c1c active+clean+snaptrim 41h [165,111,25]p165 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
participants (1)
-
Boris