BlueStore fragmentation woes
Hi, I've been seeing relatively large fragmentation numbers on all my OSDs: ceph daemon osd.13 bluestore allocator score block { "fragmentation_rating": 0.77251526920454427 } These aren't that old, as I recreated them all around July last year. They mostly hold CephFS data with erasure coding, with a mix of large and small files. The OSDs are at around 80%-85% utilization right now. Most of the data was written sequentially when the OSDs were created (I rsynced everything from a remote backup). Since then more data has been added, but not particularly quickly. At some point I noticed pathologically slow writes, and I couldn't figure out what was wrong. Eventually I did some block tracing and noticed the I/Os were very small, even though CephFS-side I was just writing one large file sequentially, and that's when I stumbled upon the free space fragmentation problem. Indeed, deleting some large files opened up some larger free extents and resolved the problem, but only until those get filled up and I'm back to fragmented tiny extents. So effectively I'm stuck at the current utilization, as trying to fill them up any more just slows down to an absolute crawl. I'm adding a few more OSDs and plan on doing the dance of removing one OSD at a time and replacing it with another one to hopefully improve the situation, but obviously this is going to take forever. Is there any plan for offering a defrag tool of some sort for bluestore? - Hector
Yep, bluestore fragmentation is an issue. It's sort of a natural result of using copy-on-write and never implementing any kind of defragmentation scheme. Adam and I have been talking about doing it now, probably piggybacking on scrub or other operations that already area reading all of the extents for an object anyway. I wrote a very simply prototype for clone to speed up the rbd mirror use case here: https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5d... Adam ended up going the extra mile and completely changed how shared blobs works which probably eliminates the need to do defrag on clone anymore from an rbd-mirror perspective, but I think we still need to identify any times we are doing full object reads of fragmented objects and consider defragmenting at that time. It might be clone, or scrub, or other things, but the point is that if we are already doing most of the work (seeks on HDD especially!) the extra cost of a large write to clean it up isn't that bad, especially if we are doing it over the course of months or years and can help keep freespace less fragmented. Mark On 5/24/23 07:17, Hector Martin wrote:
Hi,
I've been seeing relatively large fragmentation numbers on all my OSDs:
ceph daemon osd.13 bluestore allocator score block { "fragmentation_rating": 0.77251526920454427 }
These aren't that old, as I recreated them all around July last year. They mostly hold CephFS data with erasure coding, with a mix of large and small files. The OSDs are at around 80%-85% utilization right now. Most of the data was written sequentially when the OSDs were created (I rsynced everything from a remote backup). Since then more data has been added, but not particularly quickly.
At some point I noticed pathologically slow writes, and I couldn't figure out what was wrong. Eventually I did some block tracing and noticed the I/Os were very small, even though CephFS-side I was just writing one large file sequentially, and that's when I stumbled upon the free space fragmentation problem. Indeed, deleting some large files opened up some larger free extents and resolved the problem, but only until those get filled up and I'm back to fragmented tiny extents. So effectively I'm stuck at the current utilization, as trying to fill them up any more just slows down to an absolute crawl.
I'm adding a few more OSDs and plan on doing the dance of removing one OSD at a time and replacing it with another one to hopefully improve the situation, but obviously this is going to take forever.
Is there any plan for offering a defrag tool of some sort for bluestore?
- Hector _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
On 24/05/2023 22.07, Mark Nelson wrote:
Yep, bluestore fragmentation is an issue. It's sort of a natural result of using copy-on-write and never implementing any kind of defragmentation scheme. Adam and I have been talking about doing it now, probably piggybacking on scrub or other operations that already area reading all of the extents for an object anyway.
I wrote a very simply prototype for clone to speed up the rbd mirror use case here:
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5d...
Adam ended up going the extra mile and completely changed how shared blobs works which probably eliminates the need to do defrag on clone anymore from an rbd-mirror perspective, but I think we still need to identify any times we are doing full object reads of fragmented objects and consider defragmenting at that time. It might be clone, or scrub, or other things, but the point is that if we are already doing most of the work (seeks on HDD especially!) the extra cost of a large write to clean it up isn't that bad, especially if we are doing it over the course of months or years and can help keep freespace less fragmented.
Note that my particular issue seemed to specifically be free space fragmentation. I don't use RBD mirror and I would not *expect* most of my cephfs use cases to lead to any weird cow/fragmentation issues with objects other than those forced by the free space becoming fragmented (unless there is some weird pathological use case I'm hitting). Most of my write workloads are just copying files in bulk and incrementally writing out files. Would simply defragging objects during scrub/etc help with free space fragmentation itself? Those seem like two somewhat unrelated issues... note that if free space is already fragmented, you wouldn't even have a place to put down a defragmented object. Are there any stats I can look at to figure out how bad object and free space fragmentation is? It would be nice to have some clearer data beyond my hunch/deduction after seeing the I/O patterns and the sole fragmentation number :). Also would be interesting to get some kind of trace of the bluestore ops the OSD is doing, so I can find out whether it's doing something pathological that causes more fragmentation for some reason.
Mark
On 5/24/23 07:17, Hector Martin wrote:
Hi,
I've been seeing relatively large fragmentation numbers on all my OSDs:
ceph daemon osd.13 bluestore allocator score block { "fragmentation_rating": 0.77251526920454427 }
These aren't that old, as I recreated them all around July last year. They mostly hold CephFS data with erasure coding, with a mix of large and small files. The OSDs are at around 80%-85% utilization right now. Most of the data was written sequentially when the OSDs were created (I rsynced everything from a remote backup). Since then more data has been added, but not particularly quickly.
At some point I noticed pathologically slow writes, and I couldn't figure out what was wrong. Eventually I did some block tracing and noticed the I/Os were very small, even though CephFS-side I was just writing one large file sequentially, and that's when I stumbled upon the free space fragmentation problem. Indeed, deleting some large files opened up some larger free extents and resolved the problem, but only until those get filled up and I'm back to fragmented tiny extents. So effectively I'm stuck at the current utilization, as trying to fill them up any more just slows down to an absolute crawl.
I'm adding a few more OSDs and plan on doing the dance of removing one OSD at a time and replacing it with another one to hopefully improve the situation, but obviously this is going to take forever.
Is there any plan for offering a defrag tool of some sort for bluestore?
- Hector _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
- Hector
On 5/24/23 09:18, Hector Martin wrote:
> On 24/05/2023 22.07, Mark Nelson wrote:
>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>> of using copy-on-write and never implementing any kind of
>> defragmentation scheme. Adam and I have been talking about doing it
>> now, probably piggybacking on scrub or other operations that already
>> area reading all of the extents for an object anyway.
>>
>>
>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>> case here:
>>
>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>
>>
>> Adam ended up going the extra mile and completely changed how shared
>> blobs works which probably eliminates the need to do defrag on clone
>> anymore from an rbd-mirror perspective, but I think we still need to
>> identify any times we are doing full object reads of fragmented objects
>> and consider defragmenting at that time. It might be clone, or scrub,
>> or other things, but the point is that if we are already doing most of
>> the work (seeks on HDD especially!) the extra cost of a large write to
>> clean it up isn't that bad, especially if we are doing it over the
>> course of months or years and can help keep freespace less fragmented.
> Note that my particular issue seemed to specifically be free space
> fragmentation. I don't use RBD mirror and I would not *expect* most of
> my cephfs use cases to lead to any weird cow/fragmentation issues with
> objects other than those forced by the free space becoming fragmented
> (unless there is some weird pathological use case I'm hitting). Most of
> my write workloads are just copying files in bulk and incrementally
> writing out files.
>
> Would simply defragging objects during scrub/etc help with free space
> fragmentation itself? Those seem like two somewhat unrelated issues...
> note that if free space is already fragmented, you wouldn't even have a
> place to put down a defragmented object.
That is indeed one of the big issues. If you've already fragmented
freespace, it becomes much harder. The approach I've been advocating
for this is that when we scrub and encounter a heavily fragmented
object, we do a quick search to see if we can easily find contiguous
freespace for the whole object and if we can, we do that. If we can't,
we look and see if any of extents have space next to them that would
allow an adjacent range to fit in that would allow us to improve the
amount of contiguous space used (ie eliminating extents is a win,
splitting an extent to add to another extent is a judgement call). The
idea here is that we'd try to repair the holes that get left behind when
we do COW lazily after the fact. When we punch a hole we'd mark it as a
range that a new extent might easily fit back into to make things
contiguous again.
Long term I want to restructure how bluestore works. I think we should
write overwrite extents to the fast device unless they are quite large
and once we have space pressure on the fast device we write large blobs
or perhaps whole objects to the slow device and try to keep everything
neatly aligned (perhaps segment the disk for different object sizes, or
even put some small objects on the fast device). We take out the
current blob level compression, leave small extents uncompressed on the
fast device, and only compress when we write large extents or whole
objects to the slow device. The goal would be to keep fragmentation on
the slow device low, improve behavior on HDD and QLC flash, and make
better use of the DB/WAL devices that people put in their systems.
Mark
>
> Are there any stats I can look at to figure out how bad object and free
> space fragmentation is? It would be nice to have some clearer data
> beyond my hunch/deduction after seeing the I/O patterns and the sole
> fragmentation number :). Also would be interesting to get some kind of
> trace of the bluestore ops the OSD is doing, so I can find out whether
> it's doing something pathological that causes more fragmentation for
> some reason.
>
>> Mark
>>
>>
>> On 5/24/23 07:17, Hector Martin wrote:
>>> Hi,
>>>
>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>
>>> ceph daemon osd.13 bluestore allocator score block
>>> {
>>> "fragmentation_rating": 0.77251526920454427
>>> }
>>>
>>> These aren't that old, as I recreated them all around July last year.
>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>> Most of the data was written sequentially when the OSDs were created (I
>>> rsynced everything from a remote backup). Since then more data has been
>>> added, but not particularly quickly.
>>>
>>> At some point I noticed pathologically slow writes, and I couldn't
>>> figure out what was wrong. Eventually I did some block tracing and
>>> noticed the I/Os were very small, even though CephFS-side I was just
>>> writing one large file sequentially, and that's when I stumbled upon the
>>> free space fragmentation problem. Indeed, deleting some large files
>>> opened up some larger free extents and resolved the problem, but only
>>> until those get filled up and I'm back to fragmented tiny extents. So
>>> effectively I'm stuck at the current utilization, as trying to fill them
>>> up any more just slows down to an absolute crawl.
>>>
>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>> OSD at a time and replacing it with another one to hopefully improve the
>>> situation, but obviously this is going to take forever.
>>>
>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
> - Hector
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io
--
Best Regards,
Mark Nelson
Head of R&D (USA)
Clyso GmbH
p: +49 89 21552391 12
a: Loristraße 8 | 80335 München | Germany
w: https://clyso.com | e: mark.nelson@clyso.com
We are hiring: https://www.clyso.com/jobs/
Hi Hector,
I can advise two tools for further fragmentation analysis:
1) One might want to use ceph-bluestore-tool's free-dump command to get
a list of free chunks for an OSD and try to analyze whether it's really
highly fragmented and lacks long enough extents. free-dump just returns
a list of extents in json format, I can take a look to the output if
shared...
2) You might want to look for allocation probs in OSD logs and see how
fragmentation in allocated chunks has evolved.
E.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
probe -1: 35168547, 46401246, 1199516209152
probe -3: 27275094, 35681802, 200121712640
probe -5: 34847167, 52539758, 271272230912
probe -9: 44291522, 60025613, 523997483008
probe -17: 10646313, 10646313, 155178434560
The first probe refers to the last day while others match days (or
rather probes) -1, -3, -5, -9, -17
'cnt' column represents the amount of allocations performed in the
previous 24 hours and 'frags' one shows amount of fragments in the
resulted allocations. So significant mismatch between frags and cnt
might indicate some issues with high fragmentation indeed.
Apart from retrospective analysis you might also want how OSD behavior
changes after reboot - e.g. wouldn't rebooted OSD produce less
fragmentation... Which in turn might indicate some issues with BlueStore
allocator..
Just FYI: allocation probe printing interval is controlled by
bluestore_alloc_stats_dump_interval parameter.
Thanks,
Igor
On 24/05/2023 17:18, Hector Martin wrote:
> On 24/05/2023 22.07, Mark Nelson wrote:
>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>> of using copy-on-write and never implementing any kind of
>> defragmentation scheme. Adam and I have been talking about doing it
>> now, probably piggybacking on scrub or other operations that already
>> area reading all of the extents for an object anyway.
>>
>>
>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>> case here:
>>
>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>
>>
>> Adam ended up going the extra mile and completely changed how shared
>> blobs works which probably eliminates the need to do defrag on clone
>> anymore from an rbd-mirror perspective, but I think we still need to
>> identify any times we are doing full object reads of fragmented objects
>> and consider defragmenting at that time. It might be clone, or scrub,
>> or other things, but the point is that if we are already doing most of
>> the work (seeks on HDD especially!) the extra cost of a large write to
>> clean it up isn't that bad, especially if we are doing it over the
>> course of months or years and can help keep freespace less fragmented.
> Note that my particular issue seemed to specifically be free space
> fragmentation. I don't use RBD mirror and I would not *expect* most of
> my cephfs use cases to lead to any weird cow/fragmentation issues with
> objects other than those forced by the free space becoming fragmented
> (unless there is some weird pathological use case I'm hitting). Most of
> my write workloads are just copying files in bulk and incrementally
> writing out files.
>
> Would simply defragging objects during scrub/etc help with free space
> fragmentation itself? Those seem like two somewhat unrelated issues...
> note that if free space is already fragmented, you wouldn't even have a
> place to put down a defragmented object.
>
> Are there any stats I can look at to figure out how bad object and free
> space fragmentation is? It would be nice to have some clearer data
> beyond my hunch/deduction after seeing the I/O patterns and the sole
> fragmentation number :). Also would be interesting to get some kind of
> trace of the bluestore ops the OSD is doing, so I can find out whether
> it's doing something pathological that causes more fragmentation for
> some reason.
>
>> Mark
>>
>>
>> On 5/24/23 07:17, Hector Martin wrote:
>>> Hi,
>>>
>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>
>>> ceph daemon osd.13 bluestore allocator score block
>>> {
>>> "fragmentation_rating": 0.77251526920454427
>>> }
>>>
>>> These aren't that old, as I recreated them all around July last year.
>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>> Most of the data was written sequentially when the OSDs were created (I
>>> rsynced everything from a remote backup). Since then more data has been
>>> added, but not particularly quickly.
>>>
>>> At some point I noticed pathologically slow writes, and I couldn't
>>> figure out what was wrong. Eventually I did some block tracing and
>>> noticed the I/Os were very small, even though CephFS-side I was just
>>> writing one large file sequentially, and that's when I stumbled upon the
>>> free space fragmentation problem. Indeed, deleting some large files
>>> opened up some larger free extents and resolved the problem, but only
>>> until those get filled up and I'm back to fragmented tiny extents. So
>>> effectively I'm stuck at the current utilization, as trying to fill them
>>> up any more just slows down to an absolute crawl.
>>>
>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>> OSD at a time and replacing it with another one to hopefully improve the
>>> situation, but obviously this is going to take forever.
>>>
>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
> - Hector
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io
Is this related to https://tracker.ceph.com/issues/58022 ?
We still see run away osds at times, somewhat randomly, that causes runaway fragmentation issues.
Thanks,
Kevin
________________________________________
From: Igor Fedotov <igor.fedotov@croit.io>
Sent: Thursday, May 25, 2023 8:29 AM
To: Hector Martin; ceph-users@ceph.io
Subject: [ceph-users] Re: BlueStore fragmentation woes
Check twice before you click! This email originated from outside PNNL.
Hi Hector,
I can advise two tools for further fragmentation analysis:
1) One might want to use ceph-bluestore-tool's free-dump command to get
a list of free chunks for an OSD and try to analyze whether it's really
highly fragmented and lacks long enough extents. free-dump just returns
a list of extents in json format, I can take a look to the output if
shared...
2) You might want to look for allocation probs in OSD logs and see how
fragmentation in allocated chunks has evolved.
E.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
probe -1: 35168547, 46401246, 1199516209152
probe -3: 27275094, 35681802, 200121712640
probe -5: 34847167, 52539758, 271272230912
probe -9: 44291522, 60025613, 523997483008
probe -17: 10646313, 10646313, 155178434560
The first probe refers to the last day while others match days (or
rather probes) -1, -3, -5, -9, -17
'cnt' column represents the amount of allocations performed in the
previous 24 hours and 'frags' one shows amount of fragments in the
resulted allocations. So significant mismatch between frags and cnt
might indicate some issues with high fragmentation indeed.
Apart from retrospective analysis you might also want how OSD behavior
changes after reboot - e.g. wouldn't rebooted OSD produce less
fragmentation... Which in turn might indicate some issues with BlueStore
allocator..
Just FYI: allocation probe printing interval is controlled by
bluestore_alloc_stats_dump_interval parameter.
Thanks,
Igor
On 24/05/2023 17:18, Hector Martin wrote:
> On 24/05/2023 22.07, Mark Nelson wrote:
>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>> of using copy-on-write and never implementing any kind of
>> defragmentation scheme. Adam and I have been talking about doing it
>> now, probably piggybacking on scrub or other operations that already
>> area reading all of the extents for an object anyway.
>>
>>
>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>> case here:
>>
>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>
>>
>> Adam ended up going the extra mile and completely changed how shared
>> blobs works which probably eliminates the need to do defrag on clone
>> anymore from an rbd-mirror perspective, but I think we still need to
>> identify any times we are doing full object reads of fragmented objects
>> and consider defragmenting at that time. It might be clone, or scrub,
>> or other things, but the point is that if we are already doing most of
>> the work (seeks on HDD especially!) the extra cost of a large write to
>> clean it up isn't that bad, especially if we are doing it over the
>> course of months or years and can help keep freespace less fragmented.
> Note that my particular issue seemed to specifically be free space
> fragmentation. I don't use RBD mirror and I would not *expect* most of
> my cephfs use cases to lead to any weird cow/fragmentation issues with
> objects other than those forced by the free space becoming fragmented
> (unless there is some weird pathological use case I'm hitting). Most of
> my write workloads are just copying files in bulk and incrementally
> writing out files.
>
> Would simply defragging objects during scrub/etc help with free space
> fragmentation itself? Those seem like two somewhat unrelated issues...
> note that if free space is already fragmented, you wouldn't even have a
> place to put down a defragmented object.
>
> Are there any stats I can look at to figure out how bad object and free
> space fragmentation is? It would be nice to have some clearer data
> beyond my hunch/deduction after seeing the I/O patterns and the sole
> fragmentation number :). Also would be interesting to get some kind of
> trace of the bluestore ops the OSD is doing, so I can find out whether
> it's doing something pathological that causes more fragmentation for
> some reason.
>
>> Mark
>>
>>
>> On 5/24/23 07:17, Hector Martin wrote:
>>> Hi,
>>>
>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>
>>> ceph daemon osd.13 bluestore allocator score block
>>> {
>>> "fragmentation_rating": 0.77251526920454427
>>> }
>>>
>>> These aren't that old, as I recreated them all around July last year.
>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>> Most of the data was written sequentially when the OSDs were created (I
>>> rsynced everything from a remote backup). Since then more data has been
>>> added, but not particularly quickly.
>>>
>>> At some point I noticed pathologically slow writes, and I couldn't
>>> figure out what was wrong. Eventually I did some block tracing and
>>> noticed the I/Os were very small, even though CephFS-side I was just
>>> writing one large file sequentially, and that's when I stumbled upon the
>>> free space fragmentation problem. Indeed, deleting some large files
>>> opened up some larger free extents and resolved the problem, but only
>>> until those get filled up and I'm back to fragmented tiny extents. So
>>> effectively I'm stuck at the current utilization, as trying to fill them
>>> up any more just slows down to an absolute crawl.
>>>
>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>> OSD at a time and replacing it with another one to hopefully improve the
>>> situation, but obviously this is going to take forever.
>>>
>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
> - Hector
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO
it's mostly like a very imprecise first turn marker to alert that
something might be wrong. But not a real quantitative high-quality estimate.
So in fact I'd like to see a series of allocation probes showing
eventual degradation without OSD restart and immediate severe
improvement after the restart.
Can you try to collect something like that? Would the same behavior
persist with an alternative allocator?
Thanks,
Igor
On 25/05/2023 18:41, Fox, Kevin M wrote:
> Is this related to https://tracker.ceph.com/issues/58022 ?
>
> We still see run away osds at times, somewhat randomly, that causes runaway fragmentation issues.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov@croit.io>
> Sent: Thursday, May 25, 2023 8:29 AM
> To: Hector Martin; ceph-users@ceph.io
> Subject: [ceph-users] Re: BlueStore fragmentation woes
>
> Check twice before you click! This email originated from outside PNNL.
>
>
> Hi Hector,
>
> I can advise two tools for further fragmentation analysis:
>
> 1) One might want to use ceph-bluestore-tool's free-dump command to get
> a list of free chunks for an OSD and try to analyze whether it's really
> highly fragmented and lacks long enough extents. free-dump just returns
> a list of extents in json format, I can take a look to the output if
> shared...
>
> 2) You might want to look for allocation probs in OSD logs and see how
> fragmentation in allocated chunks has evolved.
>
> E.g.
>
> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
> probe -1: 35168547, 46401246, 1199516209152
> probe -3: 27275094, 35681802, 200121712640
> probe -5: 34847167, 52539758, 271272230912
> probe -9: 44291522, 60025613, 523997483008
> probe -17: 10646313, 10646313, 155178434560
>
> The first probe refers to the last day while others match days (or
> rather probes) -1, -3, -5, -9, -17
>
> 'cnt' column represents the amount of allocations performed in the
> previous 24 hours and 'frags' one shows amount of fragments in the
> resulted allocations. So significant mismatch between frags and cnt
> might indicate some issues with high fragmentation indeed.
>
> Apart from retrospective analysis you might also want how OSD behavior
> changes after reboot - e.g. wouldn't rebooted OSD produce less
> fragmentation... Which in turn might indicate some issues with BlueStore
> allocator..
>
> Just FYI: allocation probe printing interval is controlled by
> bluestore_alloc_stats_dump_interval parameter.
>
>
> Thanks,
>
> Igor
>
>
>
> On 24/05/2023 17:18, Hector Martin wrote:
>> On 24/05/2023 22.07, Mark Nelson wrote:
>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>> of using copy-on-write and never implementing any kind of
>>> defragmentation scheme. Adam and I have been talking about doing it
>>> now, probably piggybacking on scrub or other operations that already
>>> area reading all of the extents for an object anyway.
>>>
>>>
>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>> case here:
>>>
>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>
>>>
>>> Adam ended up going the extra mile and completely changed how shared
>>> blobs works which probably eliminates the need to do defrag on clone
>>> anymore from an rbd-mirror perspective, but I think we still need to
>>> identify any times we are doing full object reads of fragmented objects
>>> and consider defragmenting at that time. It might be clone, or scrub,
>>> or other things, but the point is that if we are already doing most of
>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>> clean it up isn't that bad, especially if we are doing it over the
>>> course of months or years and can help keep freespace less fragmented.
>> Note that my particular issue seemed to specifically be free space
>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>> objects other than those forced by the free space becoming fragmented
>> (unless there is some weird pathological use case I'm hitting). Most of
>> my write workloads are just copying files in bulk and incrementally
>> writing out files.
>>
>> Would simply defragging objects during scrub/etc help with free space
>> fragmentation itself? Those seem like two somewhat unrelated issues...
>> note that if free space is already fragmented, you wouldn't even have a
>> place to put down a defragmented object.
>>
>> Are there any stats I can look at to figure out how bad object and free
>> space fragmentation is? It would be nice to have some clearer data
>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>> fragmentation number :). Also would be interesting to get some kind of
>> trace of the bluestore ops the OSD is doing, so I can find out whether
>> it's doing something pathological that causes more fragmentation for
>> some reason.
>>
>>> Mark
>>>
>>>
>>> On 5/24/23 07:17, Hector Martin wrote:
>>>> Hi,
>>>>
>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>
>>>> ceph daemon osd.13 bluestore allocator score block
>>>> {
>>>> "fragmentation_rating": 0.77251526920454427
>>>> }
>>>>
>>>> These aren't that old, as I recreated them all around July last year.
>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>> Most of the data was written sequentially when the OSDs were created (I
>>>> rsynced everything from a remote backup). Since then more data has been
>>>> added, but not particularly quickly.
>>>>
>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>> figure out what was wrong. Eventually I did some block tracing and
>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>> free space fragmentation problem. Indeed, deleting some large files
>>>> opened up some larger free extents and resolved the problem, but only
>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>> up any more just slows down to an absolute crawl.
>>>>
>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>> situation, but obviously this is going to take forever.
>>>>
>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>> - Hector
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-leave@ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io
If you can give me instructions on what you want me to gather before the restart and after restart I can do it. I have some running away right now.
Thanks,
Kevin
________________________________________
From: Igor Fedotov <igor.fedotov@croit.io>
Sent: Thursday, May 25, 2023 9:17 AM
To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO
it's mostly like a very imprecise first turn marker to alert that
something might be wrong. But not a real quantitative high-quality estimate.
So in fact I'd like to see a series of allocation probes showing
eventual degradation without OSD restart and immediate severe
improvement after the restart.
Can you try to collect something like that? Would the same behavior
persist with an alternative allocator?
Thanks,
Igor
On 25/05/2023 18:41, Fox, Kevin M wrote:
> Is this related to https://tracker.ceph.com/issues/58022 ?
>
> We still see run away osds at times, somewhat randomly, that causes runaway fragmentation issues.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov@croit.io>
> Sent: Thursday, May 25, 2023 8:29 AM
> To: Hector Martin; ceph-users@ceph.io
> Subject: [ceph-users] Re: BlueStore fragmentation woes
>
> Check twice before you click! This email originated from outside PNNL.
>
>
> Hi Hector,
>
> I can advise two tools for further fragmentation analysis:
>
> 1) One might want to use ceph-bluestore-tool's free-dump command to get
> a list of free chunks for an OSD and try to analyze whether it's really
> highly fragmented and lacks long enough extents. free-dump just returns
> a list of extents in json format, I can take a look to the output if
> shared...
>
> 2) You might want to look for allocation probs in OSD logs and see how
> fragmentation in allocated chunks has evolved.
>
> E.g.
>
> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
> probe -1: 35168547, 46401246, 1199516209152
> probe -3: 27275094, 35681802, 200121712640
> probe -5: 34847167, 52539758, 271272230912
> probe -9: 44291522, 60025613, 523997483008
> probe -17: 10646313, 10646313, 155178434560
>
> The first probe refers to the last day while others match days (or
> rather probes) -1, -3, -5, -9, -17
>
> 'cnt' column represents the amount of allocations performed in the
> previous 24 hours and 'frags' one shows amount of fragments in the
> resulted allocations. So significant mismatch between frags and cnt
> might indicate some issues with high fragmentation indeed.
>
> Apart from retrospective analysis you might also want how OSD behavior
> changes after reboot - e.g. wouldn't rebooted OSD produce less
> fragmentation... Which in turn might indicate some issues with BlueStore
> allocator..
>
> Just FYI: allocation probe printing interval is controlled by
> bluestore_alloc_stats_dump_interval parameter.
>
>
> Thanks,
>
> Igor
>
>
>
> On 24/05/2023 17:18, Hector Martin wrote:
>> On 24/05/2023 22.07, Mark Nelson wrote:
>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>> of using copy-on-write and never implementing any kind of
>>> defragmentation scheme. Adam and I have been talking about doing it
>>> now, probably piggybacking on scrub or other operations that already
>>> area reading all of the extents for an object anyway.
>>>
>>>
>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>> case here:
>>>
>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>
>>>
>>> Adam ended up going the extra mile and completely changed how shared
>>> blobs works which probably eliminates the need to do defrag on clone
>>> anymore from an rbd-mirror perspective, but I think we still need to
>>> identify any times we are doing full object reads of fragmented objects
>>> and consider defragmenting at that time. It might be clone, or scrub,
>>> or other things, but the point is that if we are already doing most of
>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>> clean it up isn't that bad, especially if we are doing it over the
>>> course of months or years and can help keep freespace less fragmented.
>> Note that my particular issue seemed to specifically be free space
>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>> objects other than those forced by the free space becoming fragmented
>> (unless there is some weird pathological use case I'm hitting). Most of
>> my write workloads are just copying files in bulk and incrementally
>> writing out files.
>>
>> Would simply defragging objects during scrub/etc help with free space
>> fragmentation itself? Those seem like two somewhat unrelated issues...
>> note that if free space is already fragmented, you wouldn't even have a
>> place to put down a defragmented object.
>>
>> Are there any stats I can look at to figure out how bad object and free
>> space fragmentation is? It would be nice to have some clearer data
>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>> fragmentation number :). Also would be interesting to get some kind of
>> trace of the bluestore ops the OSD is doing, so I can find out whether
>> it's doing something pathological that causes more fragmentation for
>> some reason.
>>
>>> Mark
>>>
>>>
>>> On 5/24/23 07:17, Hector Martin wrote:
>>>> Hi,
>>>>
>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>
>>>> ceph daemon osd.13 bluestore allocator score block
>>>> {
>>>> "fragmentation_rating": 0.77251526920454427
>>>> }
>>>>
>>>> These aren't that old, as I recreated them all around July last year.
>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>> Most of the data was written sequentially when the OSDs were created (I
>>>> rsynced everything from a remote backup). Since then more data has been
>>>> added, but not particularly quickly.
>>>>
>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>> figure out what was wrong. Eventually I did some block tracing and
>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>> free space fragmentation problem. Indeed, deleting some large files
>>>> opened up some larger free extents and resolved the problem, but only
>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>> up any more just slows down to an absolute crawl.
>>>>
>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>> situation, but obviously this is going to take forever.
>>>>
>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>> - Hector
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-leave@ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io
Just run through available logs for a specific OSD (which you suspect
suffer from high fragmentation) and collect all allocation stats probes
you can find ("allocation stats probe" string is a perfect grep pattern,
please append lines with historic probes following day-0 line as well.
Given this is printed once per day there wouldn't be too many).
Then do OSD restart and wait a couple more days. Would allocation stats
show much better disparity between cnt and frags columns?
Is the similar pattern (eventual degradation in stats prior to restart
and severe improvement afterwards) be observed for other OSDs?
On 25/05/2023 19:20, Fox, Kevin M wrote:
> If you can give me instructions on what you want me to gather before the restart and after restart I can do it. I have some running away right now.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov@croit.io>
> Sent: Thursday, May 25, 2023 9:17 AM
> To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Perhaps...
>
> I don't like the idea to use fragmentation score as a real index. IMO
> it's mostly like a very imprecise first turn marker to alert that
> something might be wrong. But not a real quantitative high-quality estimate.
>
> So in fact I'd like to see a series of allocation probes showing
> eventual degradation without OSD restart and immediate severe
> improvement after the restart.
>
> Can you try to collect something like that? Would the same behavior
> persist with an alternative allocator?
>
>
> Thanks,
>
> Igor
>
>
> On 25/05/2023 18:41, Fox, Kevin M wrote:
>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>
>> We still see run away osds at times, somewhat randomly, that causes runaway fragmentation issues.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov@croit.io>
>> Sent: Thursday, May 25, 2023 8:29 AM
>> To: Hector Martin; ceph-users@ceph.io
>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Check twice before you click! This email originated from outside PNNL.
>>
>>
>> Hi Hector,
>>
>> I can advise two tools for further fragmentation analysis:
>>
>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>> a list of free chunks for an OSD and try to analyze whether it's really
>> highly fragmented and lacks long enough extents. free-dump just returns
>> a list of extents in json format, I can take a look to the output if
>> shared...
>>
>> 2) You might want to look for allocation probs in OSD logs and see how
>> fragmentation in allocated chunks has evolved.
>>
>> E.g.
>>
>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>> probe -1: 35168547, 46401246, 1199516209152
>> probe -3: 27275094, 35681802, 200121712640
>> probe -5: 34847167, 52539758, 271272230912
>> probe -9: 44291522, 60025613, 523997483008
>> probe -17: 10646313, 10646313, 155178434560
>>
>> The first probe refers to the last day while others match days (or
>> rather probes) -1, -3, -5, -9, -17
>>
>> 'cnt' column represents the amount of allocations performed in the
>> previous 24 hours and 'frags' one shows amount of fragments in the
>> resulted allocations. So significant mismatch between frags and cnt
>> might indicate some issues with high fragmentation indeed.
>>
>> Apart from retrospective analysis you might also want how OSD behavior
>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>> fragmentation... Which in turn might indicate some issues with BlueStore
>> allocator..
>>
>> Just FYI: allocation probe printing interval is controlled by
>> bluestore_alloc_stats_dump_interval parameter.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>> On 24/05/2023 17:18, Hector Martin wrote:
>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>>> of using copy-on-write and never implementing any kind of
>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>> now, probably piggybacking on scrub or other operations that already
>>>> area reading all of the extents for an object anyway.
>>>>
>>>>
>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>> case here:
>>>>
>>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>>
>>>>
>>>> Adam ended up going the extra mile and completely changed how shared
>>>> blobs works which probably eliminates the need to do defrag on clone
>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>> identify any times we are doing full object reads of fragmented objects
>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>> or other things, but the point is that if we are already doing most of
>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>> clean it up isn't that bad, especially if we are doing it over the
>>>> course of months or years and can help keep freespace less fragmented.
>>> Note that my particular issue seemed to specifically be free space
>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>> objects other than those forced by the free space becoming fragmented
>>> (unless there is some weird pathological use case I'm hitting). Most of
>>> my write workloads are just copying files in bulk and incrementally
>>> writing out files.
>>>
>>> Would simply defragging objects during scrub/etc help with free space
>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>> note that if free space is already fragmented, you wouldn't even have a
>>> place to put down a defragmented object.
>>>
>>> Are there any stats I can look at to figure out how bad object and free
>>> space fragmentation is? It would be nice to have some clearer data
>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>> fragmentation number :). Also would be interesting to get some kind of
>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>> it's doing something pathological that causes more fragmentation for
>>> some reason.
>>>
>>>> Mark
>>>>
>>>>
>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>> Hi,
>>>>>
>>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>>
>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>> {
>>>>> "fragmentation_rating": 0.77251526920454427
>>>>> }
>>>>>
>>>>> These aren't that old, as I recreated them all around July last year.
>>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>>> Most of the data was written sequentially when the OSDs were created (I
>>>>> rsynced everything from a remote backup). Since then more data has been
>>>>> added, but not particularly quickly.
>>>>>
>>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>> opened up some larger free extents and resolved the problem, but only
>>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>>> up any more just slows down to an absolute crawl.
>>>>>
>>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>>> situation, but obviously this is going to take forever.
>>>>>
>>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>>
>>>>> - Hector
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-leave@ceph.io
Ok, I'm gathering the "allocation stats probe" stuff. Not sure I follow what you mean by the historic probes. just:
| egrep "allocation stats probe|probe" ?
That gets something like:
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 24503, 24503, 58141900800
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 24594, 24594, 56951898112
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -6: 19737, 19737, 37299027968
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -14: 20373, 20373, 35302801408
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -30: 19072, 19072, 33645854720
if that is the right query, then I'll gather the metrics, restart and gather some more after and let you know.
Thanks,
Kevin
________________________________________
From: Igor Fedotov <igor.fedotov@croit.io>
Sent: Thursday, May 25, 2023 9:29 AM
To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
Just run through available logs for a specific OSD (which you suspect
suffer from high fragmentation) and collect all allocation stats probes
you can find ("allocation stats probe" string is a perfect grep pattern,
please append lines with historic probes following day-0 line as well.
Given this is printed once per day there wouldn't be too many).
Then do OSD restart and wait a couple more days. Would allocation stats
show much better disparity between cnt and frags columns?
Is the similar pattern (eventual degradation in stats prior to restart
and severe improvement afterwards) be observed for other OSDs?
On 25/05/2023 19:20, Fox, Kevin M wrote:
> If you can give me instructions on what you want me to gather before the restart and after restart I can do it. I have some running away right now.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov@croit.io>
> Sent: Thursday, May 25, 2023 9:17 AM
> To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Perhaps...
>
> I don't like the idea to use fragmentation score as a real index. IMO
> it's mostly like a very imprecise first turn marker to alert that
> something might be wrong. But not a real quantitative high-quality estimate.
>
> So in fact I'd like to see a series of allocation probes showing
> eventual degradation without OSD restart and immediate severe
> improvement after the restart.
>
> Can you try to collect something like that? Would the same behavior
> persist with an alternative allocator?
>
>
> Thanks,
>
> Igor
>
>
> On 25/05/2023 18:41, Fox, Kevin M wrote:
>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>
>> We still see run away osds at times, somewhat randomly, that causes runaway fragmentation issues.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov@croit.io>
>> Sent: Thursday, May 25, 2023 8:29 AM
>> To: Hector Martin; ceph-users@ceph.io
>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Check twice before you click! This email originated from outside PNNL.
>>
>>
>> Hi Hector,
>>
>> I can advise two tools for further fragmentation analysis:
>>
>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>> a list of free chunks for an OSD and try to analyze whether it's really
>> highly fragmented and lacks long enough extents. free-dump just returns
>> a list of extents in json format, I can take a look to the output if
>> shared...
>>
>> 2) You might want to look for allocation probs in OSD logs and see how
>> fragmentation in allocated chunks has evolved.
>>
>> E.g.
>>
>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>> probe -1: 35168547, 46401246, 1199516209152
>> probe -3: 27275094, 35681802, 200121712640
>> probe -5: 34847167, 52539758, 271272230912
>> probe -9: 44291522, 60025613, 523997483008
>> probe -17: 10646313, 10646313, 155178434560
>>
>> The first probe refers to the last day while others match days (or
>> rather probes) -1, -3, -5, -9, -17
>>
>> 'cnt' column represents the amount of allocations performed in the
>> previous 24 hours and 'frags' one shows amount of fragments in the
>> resulted allocations. So significant mismatch between frags and cnt
>> might indicate some issues with high fragmentation indeed.
>>
>> Apart from retrospective analysis you might also want how OSD behavior
>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>> fragmentation... Which in turn might indicate some issues with BlueStore
>> allocator..
>>
>> Just FYI: allocation probe printing interval is controlled by
>> bluestore_alloc_stats_dump_interval parameter.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>> On 24/05/2023 17:18, Hector Martin wrote:
>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>>> of using copy-on-write and never implementing any kind of
>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>> now, probably piggybacking on scrub or other operations that already
>>>> area reading all of the extents for an object anyway.
>>>>
>>>>
>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>> case here:
>>>>
>>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>>
>>>>
>>>> Adam ended up going the extra mile and completely changed how shared
>>>> blobs works which probably eliminates the need to do defrag on clone
>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>> identify any times we are doing full object reads of fragmented objects
>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>> or other things, but the point is that if we are already doing most of
>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>> clean it up isn't that bad, especially if we are doing it over the
>>>> course of months or years and can help keep freespace less fragmented.
>>> Note that my particular issue seemed to specifically be free space
>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>> objects other than those forced by the free space becoming fragmented
>>> (unless there is some weird pathological use case I'm hitting). Most of
>>> my write workloads are just copying files in bulk and incrementally
>>> writing out files.
>>>
>>> Would simply defragging objects during scrub/etc help with free space
>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>> note that if free space is already fragmented, you wouldn't even have a
>>> place to put down a defragmented object.
>>>
>>> Are there any stats I can look at to figure out how bad object and free
>>> space fragmentation is? It would be nice to have some clearer data
>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>> fragmentation number :). Also would be interesting to get some kind of
>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>> it's doing something pathological that causes more fragmentation for
>>> some reason.
>>>
>>>> Mark
>>>>
>>>>
>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>> Hi,
>>>>>
>>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>>
>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>> {
>>>>> "fragmentation_rating": 0.77251526920454427
>>>>> }
>>>>>
>>>>> These aren't that old, as I recreated them all around July last year.
>>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>>> Most of the data was written sequentially when the OSDs were created (I
>>>>> rsynced everything from a remote backup). Since then more data has been
>>>>> added, but not particularly quickly.
>>>>>
>>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>> opened up some larger free extents and resolved the problem, but only
>>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>>> up any more just slows down to an absolute crawl.
>>>>>
>>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>>> situation, but obviously this is going to take forever.
>>>>>
>>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>>
>>>>> - Hector
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-leave@ceph.io
Yeah this looks fine. Please collect all of them for a given OSD.
Then restart OSD, wait more to come (1-2 days) and collect them too.
A side note - in the attached probe I can't see any fragmentation at all
- amount of allocations is equal to amount of fragments, e.g.
cnt: 27637 frags: 27637
And the average requested chunk is 63777406976 / 27637 = ~2308 bytes.
I.e. in average one needed less than a single alloc unit. Which would
tell us nothing about the fragmentation...
Thanks,
Igor
On 25/05/2023 19:36, Fox, Kevin M wrote:
> Ok, I'm gathering the "allocation stats probe" stuff. Not sure I follow what you mean by the historic probes. just:
> | egrep "allocation stats probe|probe" ?
>
> That gets something like:
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 24503, 24503, 58141900800
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 24594, 24594, 56951898112
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -6: 19737, 19737, 37299027968
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -14: 20373, 20373, 35302801408
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -30: 19072, 19072, 33645854720
>
> if that is the right query, then I'll gather the metrics, restart and gather some more after and let you know.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov@croit.io>
> Sent: Thursday, May 25, 2023 9:29 AM
> To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Just run through available logs for a specific OSD (which you suspect
> suffer from high fragmentation) and collect all allocation stats probes
> you can find ("allocation stats probe" string is a perfect grep pattern,
> please append lines with historic probes following day-0 line as well.
> Given this is printed once per day there wouldn't be too many).
>
> Then do OSD restart and wait a couple more days. Would allocation stats
> show much better disparity between cnt and frags columns?
>
> Is the similar pattern (eventual degradation in stats prior to restart
> and severe improvement afterwards) be observed for other OSDs?
>
>
> On 25/05/2023 19:20, Fox, Kevin M wrote:
>> If you can give me instructions on what you want me to gather before the restart and after restart I can do it. I have some running away right now.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov@croit.io>
>> Sent: Thursday, May 25, 2023 9:17 AM
>> To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
>> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Perhaps...
>>
>> I don't like the idea to use fragmentation score as a real index. IMO
>> it's mostly like a very imprecise first turn marker to alert that
>> something might be wrong. But not a real quantitative high-quality estimate.
>>
>> So in fact I'd like to see a series of allocation probes showing
>> eventual degradation without OSD restart and immediate severe
>> improvement after the restart.
>>
>> Can you try to collect something like that? Would the same behavior
>> persist with an alternative allocator?
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>> On 25/05/2023 18:41, Fox, Kevin M wrote:
>>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>>
>>> We still see run away osds at times, somewhat randomly, that causes runaway fragmentation issues.
>>>
>>> Thanks,
>>> Kevin
>>>
>>> ________________________________________
>>> From: Igor Fedotov <igor.fedotov@croit.io>
>>> Sent: Thursday, May 25, 2023 8:29 AM
>>> To: Hector Martin; ceph-users@ceph.io
>>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>>
>>> Check twice before you click! This email originated from outside PNNL.
>>>
>>>
>>> Hi Hector,
>>>
>>> I can advise two tools for further fragmentation analysis:
>>>
>>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>>> a list of free chunks for an OSD and try to analyze whether it's really
>>> highly fragmented and lacks long enough extents. free-dump just returns
>>> a list of extents in json format, I can take a look to the output if
>>> shared...
>>>
>>> 2) You might want to look for allocation probs in OSD logs and see how
>>> fragmentation in allocated chunks has evolved.
>>>
>>> E.g.
>>>
>>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>>> probe -1: 35168547, 46401246, 1199516209152
>>> probe -3: 27275094, 35681802, 200121712640
>>> probe -5: 34847167, 52539758, 271272230912
>>> probe -9: 44291522, 60025613, 523997483008
>>> probe -17: 10646313, 10646313, 155178434560
>>>
>>> The first probe refers to the last day while others match days (or
>>> rather probes) -1, -3, -5, -9, -17
>>>
>>> 'cnt' column represents the amount of allocations performed in the
>>> previous 24 hours and 'frags' one shows amount of fragments in the
>>> resulted allocations. So significant mismatch between frags and cnt
>>> might indicate some issues with high fragmentation indeed.
>>>
>>> Apart from retrospective analysis you might also want how OSD behavior
>>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>>> fragmentation... Which in turn might indicate some issues with BlueStore
>>> allocator..
>>>
>>> Just FYI: allocation probe printing interval is controlled by
>>> bluestore_alloc_stats_dump_interval parameter.
>>>
>>>
>>> Thanks,
>>>
>>> Igor
>>>
>>>
>>>
>>> On 24/05/2023 17:18, Hector Martin wrote:
>>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>>>> of using copy-on-write and never implementing any kind of
>>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>>> now, probably piggybacking on scrub or other operations that already
>>>>> area reading all of the extents for an object anyway.
>>>>>
>>>>>
>>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>>> case here:
>>>>>
>>>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>>>
>>>>>
>>>>> Adam ended up going the extra mile and completely changed how shared
>>>>> blobs works which probably eliminates the need to do defrag on clone
>>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>>> identify any times we are doing full object reads of fragmented objects
>>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>>> or other things, but the point is that if we are already doing most of
>>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>>> clean it up isn't that bad, especially if we are doing it over the
>>>>> course of months or years and can help keep freespace less fragmented.
>>>> Note that my particular issue seemed to specifically be free space
>>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>>> objects other than those forced by the free space becoming fragmented
>>>> (unless there is some weird pathological use case I'm hitting). Most of
>>>> my write workloads are just copying files in bulk and incrementally
>>>> writing out files.
>>>>
>>>> Would simply defragging objects during scrub/etc help with free space
>>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>>> note that if free space is already fragmented, you wouldn't even have a
>>>> place to put down a defragmented object.
>>>>
>>>> Are there any stats I can look at to figure out how bad object and free
>>>> space fragmentation is? It would be nice to have some clearer data
>>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>>> fragmentation number :). Also would be interesting to get some kind of
>>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>>> it's doing something pathological that causes more fragmentation for
>>>> some reason.
>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>>>
>>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>>> {
>>>>>> "fragmentation_rating": 0.77251526920454427
>>>>>> }
>>>>>>
>>>>>> These aren't that old, as I recreated them all around July last year.
>>>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>>>> Most of the data was written sequentially when the OSDs were created (I
>>>>>> rsynced everything from a remote backup). Since then more data has been
>>>>>> added, but not particularly quickly.
>>>>>>
>>>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>>> opened up some larger free extents and resolved the problem, but only
>>>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>>>> up any more just slows down to an absolute crawl.
>>>>>>
>>>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>>>> situation, but obviously this is going to take forever.
>>>>>>
>>>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>>>
>>>>>> - Hector
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
Ok, I restarted it May 25th, ~11:30, let it run over the long weekend and just checked on it. Data attached.
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 107: cnt: 17991 fr
ags: 17991 size: 32016760832
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 20267, 20267, 39482425344
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 19737, 19737, 37299027968
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -7: 18498, 18498, 32395558912
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -11: 20373, 20373, 35302801408
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -27: 19072, 19072, 33645854720
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 108: cnt: 24594 fr
ags: 24594 size: 56951898112
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 17991, 17991, 32016760832
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 20267, 20267, 39482425344
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -4: 19737, 19737, 37299027968
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -12: 20373, 20373, 35302801408
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -28: 19072, 19072, 33645854720
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 109: cnt: 24503 frags: 24503 size: 58141900800
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 24594, 24594, 56951898112
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 20267, 20267, 39482425344
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -5: 19737, 19737, 37299027968
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -13: 20373, 20373, 35302801408
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -29: 19072, 19072, 33645854720
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 24503, 24503, 58141900800
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 24594, 24594, 56951898112
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -6: 19737, 19737, 37299027968
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -14: 20373, 20373, 35302801408
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -30: 19072, 19072, 33645854720
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 111: cnt: 22136 frags: 22136 size: 48656023552
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 27637, 27637, 63777406976
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 24594, 24594, 56951898112
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -7: 19737, 19737, 37299027968
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -15: 20373, 20373, 35302801408
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -31: 19072, 19072, 33645854720
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 0: cnt: 21986 frags: 21986 size: 47407562752
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 0, 0, 0
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 0, 0, 0
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -4: 0, 0, 0
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -8: 0, 0, 0
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -16: 0, 0, 0
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 1: cnt: 21145 frags: 21145 size: 44858146816
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 21986, 21986, 47407562752
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 0, 0, 0
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -5: 0, 0, 0
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -9: 0, 0, 0
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -17: 0, 0, 0
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 2: cnt: 17987 frags: 17987 size: 32446676992
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 21145, 21145, 44858146816
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 21986, 21986, 47407562752
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -6: 0, 0, 0
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -10: 0, 0, 0
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -18: 0, 0, 0
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 3: cnt: 17509 frags: 17509 size: 31015436288
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 17987, 17987, 32446676992
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 21986, 21986, 47407562752
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -7: 0, 0, 0
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -11: 0, 0, 0
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -19: 0, 0, 0
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 4: cnt: 21016 frags: 21016 size: 45432438784
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 17509, 17509, 31015436288
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 17987, 17987, 32446676992
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -4: 21986, 21986, 47407562752
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -12: 0, 0, 0
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -20: 0, 0, 0
Thanks,
Kevin
________________________________________
From: Fox, Kevin M <Kevin.Fox@pnnl.gov>
Sent: Thursday, May 25, 2023 9:36 AM
To: Igor Fedotov; Hector Martin; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
Ok, I'm gathering the "allocation stats probe" stuff. Not sure I follow what you mean by the historic probes. just:
| egrep "allocation stats probe|probe" ?
That gets something like:
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 24503, 24503, 58141900800
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 24594, 24594, 56951898112
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -6: 19737, 19737, 37299027968
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -14: 20373, 20373, 35302801408
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -30: 19072, 19072, 33645854720
if that is the right query, then I'll gather the metrics, restart and gather some more after and let you know.
Thanks,
Kevin
________________________________________
From: Igor Fedotov <igor.fedotov@croit.io>
Sent: Thursday, May 25, 2023 9:29 AM
To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
Just run through available logs for a specific OSD (which you suspect
suffer from high fragmentation) and collect all allocation stats probes
you can find ("allocation stats probe" string is a perfect grep pattern,
please append lines with historic probes following day-0 line as well.
Given this is printed once per day there wouldn't be too many).
Then do OSD restart and wait a couple more days. Would allocation stats
show much better disparity between cnt and frags columns?
Is the similar pattern (eventual degradation in stats prior to restart
and severe improvement afterwards) be observed for other OSDs?
On 25/05/2023 19:20, Fox, Kevin M wrote:
> If you can give me instructions on what you want me to gather before the restart and after restart I can do it. I have some running away right now.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov@croit.io>
> Sent: Thursday, May 25, 2023 9:17 AM
> To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Perhaps...
>
> I don't like the idea to use fragmentation score as a real index. IMO
> it's mostly like a very imprecise first turn marker to alert that
> something might be wrong. But not a real quantitative high-quality estimate.
>
> So in fact I'd like to see a series of allocation probes showing
> eventual degradation without OSD restart and immediate severe
> improvement after the restart.
>
> Can you try to collect something like that? Would the same behavior
> persist with an alternative allocator?
>
>
> Thanks,
>
> Igor
>
>
> On 25/05/2023 18:41, Fox, Kevin M wrote:
>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>
>> We still see run away osds at times, somewhat randomly, that causes runaway fragmentation issues.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov@croit.io>
>> Sent: Thursday, May 25, 2023 8:29 AM
>> To: Hector Martin; ceph-users@ceph.io
>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Check twice before you click! This email originated from outside PNNL.
>>
>>
>> Hi Hector,
>>
>> I can advise two tools for further fragmentation analysis:
>>
>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>> a list of free chunks for an OSD and try to analyze whether it's really
>> highly fragmented and lacks long enough extents. free-dump just returns
>> a list of extents in json format, I can take a look to the output if
>> shared...
>>
>> 2) You might want to look for allocation probs in OSD logs and see how
>> fragmentation in allocated chunks has evolved.
>>
>> E.g.
>>
>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>> probe -1: 35168547, 46401246, 1199516209152
>> probe -3: 27275094, 35681802, 200121712640
>> probe -5: 34847167, 52539758, 271272230912
>> probe -9: 44291522, 60025613, 523997483008
>> probe -17: 10646313, 10646313, 155178434560
>>
>> The first probe refers to the last day while others match days (or
>> rather probes) -1, -3, -5, -9, -17
>>
>> 'cnt' column represents the amount of allocations performed in the
>> previous 24 hours and 'frags' one shows amount of fragments in the
>> resulted allocations. So significant mismatch between frags and cnt
>> might indicate some issues with high fragmentation indeed.
>>
>> Apart from retrospective analysis you might also want how OSD behavior
>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>> fragmentation... Which in turn might indicate some issues with BlueStore
>> allocator..
>>
>> Just FYI: allocation probe printing interval is controlled by
>> bluestore_alloc_stats_dump_interval parameter.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>> On 24/05/2023 17:18, Hector Martin wrote:
>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>>> of using copy-on-write and never implementing any kind of
>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>> now, probably piggybacking on scrub or other operations that already
>>>> area reading all of the extents for an object anyway.
>>>>
>>>>
>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>> case here:
>>>>
>>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>>
>>>>
>>>> Adam ended up going the extra mile and completely changed how shared
>>>> blobs works which probably eliminates the need to do defrag on clone
>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>> identify any times we are doing full object reads of fragmented objects
>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>> or other things, but the point is that if we are already doing most of
>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>> clean it up isn't that bad, especially if we are doing it over the
>>>> course of months or years and can help keep freespace less fragmented.
>>> Note that my particular issue seemed to specifically be free space
>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>> objects other than those forced by the free space becoming fragmented
>>> (unless there is some weird pathological use case I'm hitting). Most of
>>> my write workloads are just copying files in bulk and incrementally
>>> writing out files.
>>>
>>> Would simply defragging objects during scrub/etc help with free space
>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>> note that if free space is already fragmented, you wouldn't even have a
>>> place to put down a defragmented object.
>>>
>>> Are there any stats I can look at to figure out how bad object and free
>>> space fragmentation is? It would be nice to have some clearer data
>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>> fragmentation number :). Also would be interesting to get some kind of
>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>> it's doing something pathological that causes more fragmentation for
>>> some reason.
>>>
>>>> Mark
>>>>
>>>>
>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>> Hi,
>>>>>
>>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>>
>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>> {
>>>>> "fragmentation_rating": 0.77251526920454427
>>>>> }
>>>>>
>>>>> These aren't that old, as I recreated them all around July last year.
>>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>>> Most of the data was written sequentially when the OSDs were created (I
>>>>> rsynced everything from a remote backup). Since then more data has been
>>>>> added, but not particularly quickly.
>>>>>
>>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>> opened up some larger free extents and resolved the problem, but only
>>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>>> up any more just slows down to an absolute crawl.
>>>>>
>>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>>> situation, but obviously this is going to take forever.
>>>>>
>>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>>
>>>>> - Hector
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Kevin,
according to the shared probes there were no fragmented allocations -
cnt = frags for all the probes. And average allocation request is
pretty large - more than 1.5 MB for the probes I checked.
So to me it looks like your disk fragmentation (at least for new
allocations) is of little significance at the moment - it doesn't affect
write requests.
As I mentioned before for further analysis you might want to run through
the output from 'ceph tell osd.N bluestore allocator dump block' command.
This is my recent commit to build free space histogram from it:
https://github.com/ceph/ceph/pull/51820
One can use this as an example and create a script to do the same (just
to avoid all the tricks with building/upgrading Ceph binaries) or
backport and build custom Ceph image.
Thanks,
Igor
On 31/05/2023 01:11, Fox, Kevin M wrote:
> Ok, I restarted it May 25th, ~11:30, let it run over the long weekend and just checked on it. Data attached.
>
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 107: cnt: 17991 fr
> ags: 17991 size: 32016760832
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 20267, 20267, 39482425344
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 19737, 19737, 37299027968
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -7: 18498, 18498, 32395558912
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -11: 20373, 20373, 35302801408
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -27: 19072, 19072, 33645854720
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 108: cnt: 24594 fr
> ags: 24594 size: 56951898112
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 17991, 17991, 32016760832
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 20267, 20267, 39482425344
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -4: 19737, 19737, 37299027968
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -12: 20373, 20373, 35302801408
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -28: 19072, 19072, 33645854720
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 109: cnt: 24503 frags: 24503 size: 58141900800
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 24594, 24594, 56951898112
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 20267, 20267, 39482425344
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -5: 19737, 19737, 37299027968
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -13: 20373, 20373, 35302801408
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -29: 19072, 19072, 33645854720
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 24503, 24503, 58141900800
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 24594, 24594, 56951898112
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -6: 19737, 19737, 37299027968
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -14: 20373, 20373, 35302801408
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -30: 19072, 19072, 33645854720
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 111: cnt: 22136 frags: 22136 size: 48656023552
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 27637, 27637, 63777406976
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 24594, 24594, 56951898112
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -7: 19737, 19737, 37299027968
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -15: 20373, 20373, 35302801408
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -31: 19072, 19072, 33645854720
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 0: cnt: 21986 frags: 21986 size: 47407562752
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 0, 0, 0
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 0, 0, 0
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -4: 0, 0, 0
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -8: 0, 0, 0
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -16: 0, 0, 0
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 1: cnt: 21145 frags: 21145 size: 44858146816
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 21986, 21986, 47407562752
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 0, 0, 0
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -5: 0, 0, 0
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -9: 0, 0, 0
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -17: 0, 0, 0
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 2: cnt: 17987 frags: 17987 size: 32446676992
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 21145, 21145, 44858146816
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 21986, 21986, 47407562752
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -6: 0, 0, 0
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -10: 0, 0, 0
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -18: 0, 0, 0
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 3: cnt: 17509 frags: 17509 size: 31015436288
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 17987, 17987, 32446676992
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -3: 21986, 21986, 47407562752
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -7: 0, 0, 0
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -11: 0, 0, 0
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -19: 0, 0, 0
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 4: cnt: 21016 frags: 21016 size: 45432438784
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 17509, 17509, 31015436288
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 17987, 17987, 32446676992
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -4: 21986, 21986, 47407562752
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -12: 0, 0, 0
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug 2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -20: 0, 0, 0
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Fox, Kevin M <Kevin.Fox@pnnl.gov>
> Sent: Thursday, May 25, 2023 9:36 AM
> To: Igor Fedotov; Hector Martin; ceph-users@ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Ok, I'm gathering the "allocation stats probe" stuff. Not sure I follow what you mean by the historic probes. just:
> | egrep "allocation stats probe|probe" ?
>
> That gets something like:
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -1: 24503, 24503, 58141900800
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -2: 24594, 24594, 56951898112
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -6: 19737, 19737, 37299027968
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -14: 20373, 20373, 35302801408
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug 2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -30: 19072, 19072, 33645854720
>
> if that is the right query, then I'll gather the metrics, restart and gather some more after and let you know.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov@croit.io>
> Sent: Thursday, May 25, 2023 9:29 AM
> To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Just run through available logs for a specific OSD (which you suspect
> suffer from high fragmentation) and collect all allocation stats probes
> you can find ("allocation stats probe" string is a perfect grep pattern,
> please append lines with historic probes following day-0 line as well.
> Given this is printed once per day there wouldn't be too many).
>
> Then do OSD restart and wait a couple more days. Would allocation stats
> show much better disparity between cnt and frags columns?
>
> Is the similar pattern (eventual degradation in stats prior to restart
> and severe improvement afterwards) be observed for other OSDs?
>
>
> On 25/05/2023 19:20, Fox, Kevin M wrote:
>> If you can give me instructions on what you want me to gather before the restart and after restart I can do it. I have some running away right now.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov@croit.io>
>> Sent: Thursday, May 25, 2023 9:17 AM
>> To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io
>> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Perhaps...
>>
>> I don't like the idea to use fragmentation score as a real index. IMO
>> it's mostly like a very imprecise first turn marker to alert that
>> something might be wrong. But not a real quantitative high-quality estimate.
>>
>> So in fact I'd like to see a series of allocation probes showing
>> eventual degradation without OSD restart and immediate severe
>> improvement after the restart.
>>
>> Can you try to collect something like that? Would the same behavior
>> persist with an alternative allocator?
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>> On 25/05/2023 18:41, Fox, Kevin M wrote:
>>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>>
>>> We still see run away osds at times, somewhat randomly, that causes runaway fragmentation issues.
>>>
>>> Thanks,
>>> Kevin
>>>
>>> ________________________________________
>>> From: Igor Fedotov <igor.fedotov@croit.io>
>>> Sent: Thursday, May 25, 2023 8:29 AM
>>> To: Hector Martin; ceph-users@ceph.io
>>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>>
>>> Check twice before you click! This email originated from outside PNNL.
>>>
>>>
>>> Hi Hector,
>>>
>>> I can advise two tools for further fragmentation analysis:
>>>
>>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>>> a list of free chunks for an OSD and try to analyze whether it's really
>>> highly fragmented and lacks long enough extents. free-dump just returns
>>> a list of extents in json format, I can take a look to the output if
>>> shared...
>>>
>>> 2) You might want to look for allocation probs in OSD logs and see how
>>> fragmentation in allocated chunks has evolved.
>>>
>>> E.g.
>>>
>>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>>> probe -1: 35168547, 46401246, 1199516209152
>>> probe -3: 27275094, 35681802, 200121712640
>>> probe -5: 34847167, 52539758, 271272230912
>>> probe -9: 44291522, 60025613, 523997483008
>>> probe -17: 10646313, 10646313, 155178434560
>>>
>>> The first probe refers to the last day while others match days (or
>>> rather probes) -1, -3, -5, -9, -17
>>>
>>> 'cnt' column represents the amount of allocations performed in the
>>> previous 24 hours and 'frags' one shows amount of fragments in the
>>> resulted allocations. So significant mismatch between frags and cnt
>>> might indicate some issues with high fragmentation indeed.
>>>
>>> Apart from retrospective analysis you might also want how OSD behavior
>>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>>> fragmentation... Which in turn might indicate some issues with BlueStore
>>> allocator..
>>>
>>> Just FYI: allocation probe printing interval is controlled by
>>> bluestore_alloc_stats_dump_interval parameter.
>>>
>>>
>>> Thanks,
>>>
>>> Igor
>>>
>>>
>>>
>>> On 24/05/2023 17:18, Hector Martin wrote:
>>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>>>> of using copy-on-write and never implementing any kind of
>>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>>> now, probably piggybacking on scrub or other operations that already
>>>>> area reading all of the extents for an object anyway.
>>>>>
>>>>>
>>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>>> case here:
>>>>>
>>>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>>>
>>>>>
>>>>> Adam ended up going the extra mile and completely changed how shared
>>>>> blobs works which probably eliminates the need to do defrag on clone
>>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>>> identify any times we are doing full object reads of fragmented objects
>>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>>> or other things, but the point is that if we are already doing most of
>>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>>> clean it up isn't that bad, especially if we are doing it over the
>>>>> course of months or years and can help keep freespace less fragmented.
>>>> Note that my particular issue seemed to specifically be free space
>>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>>> objects other than those forced by the free space becoming fragmented
>>>> (unless there is some weird pathological use case I'm hitting). Most of
>>>> my write workloads are just copying files in bulk and incrementally
>>>> writing out files.
>>>>
>>>> Would simply defragging objects during scrub/etc help with free space
>>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>>> note that if free space is already fragmented, you wouldn't even have a
>>>> place to put down a defragmented object.
>>>>
>>>> Are there any stats I can look at to figure out how bad object and free
>>>> space fragmentation is? It would be nice to have some clearer data
>>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>>> fragmentation number :). Also would be interesting to get some kind of
>>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>>> it's doing something pathological that causes more fragmentation for
>>>> some reason.
>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>>>
>>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>>> {
>>>>>> "fragmentation_rating": 0.77251526920454427
>>>>>> }
>>>>>>
>>>>>> These aren't that old, as I recreated them all around July last year.
>>>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>>>> Most of the data was written sequentially when the OSDs were created (I
>>>>>> rsynced everything from a remote backup). Since then more data has been
>>>>>> added, but not particularly quickly.
>>>>>>
>>>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>>> opened up some larger free extents and resolved the problem, but only
>>>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>>>> up any more just slows down to an absolute crawl.
>>>>>>
>>>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>>>> situation, but obviously this is going to take forever.
>>>>>>
>>>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>>>
>>>>>> - Hector
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
On 5/25/23 18:17, Igor Fedotov wrote:
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO it's mostly like a very imprecise first turn marker to alert that something might be wrong. But not a real quantitative high-quality estimate.
Chiming in on the high fragmentation issue. We started collecting "fragmentation_rating" of each OSD this afternoon. All OSDs that have been provisioned a year ago have a fragmentation rating of ~ 0.9. Not sure for how long they are on this level. And after reading your mails it might not be a problem at all. But we will start collecting this information in the coming weeks. We will be re-provisioning all our OSDs, so that might be a good time to look at the behavior and development of "cnt versus frags" ratio. After we completely emptied a host, even after having the OSDs run idle for a couple of hours, the fragmentation ratio would not drop lower than 0.27 for some OSDs, and up to 0.62 for others. Is it expected that this will not go to ~ zero? We have not noticed any performance degradation whatsoever. We are running with bitmap for current OSDs, hybrid allocator for the new provisioned OSDs (4k min alloc size). Gr. Stefan
On 25/05/2023 20:36, Stefan Kooman wrote:
On 5/25/23 18:17, Igor Fedotov wrote:
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO it's mostly like a very imprecise first turn marker to alert that something might be wrong. But not a real quantitative high-quality estimate.
Chiming in on the high fragmentation issue. We started collecting "fragmentation_rating" of each OSD this afternoon. All OSDs that have been provisioned a year ago have a fragmentation rating of ~ 0.9. Not sure for how long they are on this level.
Could you please collect allocation probes from existing OSD logs? Just a few samples from different OSDs...
And after reading your mails it might not be a problem at all. But we will start collecting this information in the coming weeks.
We will be re-provisioning all our OSDs, so that might be a good time to look at the behavior and development of "cnt versus frags" ratio.
After we completely emptied a host, even after having the OSDs run idle for a couple of hours, the fragmentation ratio would not drop lower than 0.27 for some OSDs, and up to 0.62 for others. Is it expected that this will not go to ~ zero?
You might be facing the issue fixed by https://github.com/ceph/ceph/pull/49885
We have not noticed any performance degradation whatsoever. We are running with bitmap for current OSDs, hybrid allocator for the new provisioned OSDs (4k min alloc size).
Gr. Stefan
Hi Igor, Should we backpot this to the p,q and reef release's? Thanks, k Sent from my iPhone
On 25 May 2023, at 23:13, Igor Fedotov <igor.fedotov@croit.io> wrote:
You might be facing the issue fixed by https://github.com/ceph/ceph/pull/49885
yeah, definitely this makes sense On 26/05/2023 09:39, Konstantin Shalygin wrote:
Hi Igor,
Should we backpot this to the p,q and reef release's?
Thanks, k Sent from my iPhone
On 25 May 2023, at 23:13, Igor Fedotov <igor.fedotov@croit.io> wrote:
You might be facing the issue fixed by https://github.com/ceph/ceph/pull/49885
On 5/25/23 22:12, Igor Fedotov wrote:
On 25/05/2023 20:36, Stefan Kooman wrote:
On 5/25/23 18:17, Igor Fedotov wrote:
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO it's mostly like a very imprecise first turn marker to alert that something might be wrong. But not a real quantitative high-quality estimate.
Chiming in on the high fragmentation issue. We started collecting "fragmentation_rating" of each OSD this afternoon. All OSDs that have been provisioned a year ago have a fragmentation rating of ~ 0.9. Not sure for how long they are on this level.
Could you please collect allocation probes from existing OSD logs? Just a few samples from different OSDs...
10 OSDs from one host, but I have checked other nodes and they are similar: CNT FRAG Size Ratio Avg Frag size 21350923 37146899 317040259072 1.73982637659271 8534.77053554322 20951932 38122769 317841477632 1.8195347808498 8337.31352599283 21188454 37298950 278389411840 1.76034315670223 7463.73321072041 21605451 39369462 270427185152 1.82220042525379 6868.95810646333 19215230 36063713 290967818240 1.87682962941375 8068.16032059705 19293599 35464928 269238423552 1.83817068033807 7591.68109835159 19963538 36088151 315796836352 1.80770317365589 8750.70702159277 18030613 31753098 297826177024 1.76106591606176 9379.43683554909 17889602 31718012 299550142464 1.77298589426417 9444.16511551859 18475332 33264944 266053271552 1.80050588536109 7998.0074985847 18618154 31914219 254801883136 1.71414518324427 7983.96110323113 16437108 29421873 275350355968 1.78996651965784 9358.69568766067 17164338 28605353 249404649472 1.66655731202683 8718.81040838755 17895480 29658102 309047177216 1.65729569701399 10420.3288941416 19546560 34588509 301368737792 1.76954456436324 8712.97279081905 18525784 34806856 314875801600 1.87883309014075 9046.37297893266 18550989 35236438 273069948928 1.89943716747393 7749.64679823767 19085807 34605572 255512043520 1.81315738967705 7383.55209155335 17203820 31205542 277097357312 1.81387284916954 8879.74826112618 18003801 33723670 269696761856 1.87314167713807 7997.25420916525 18655425 33227176 306511810560 1.78109992133655 9224.7325069094 26380965 45627920 335281111040 1.72957736762093 7348.15680925188 24923956 44721109 328790982656 1.79430219664968 7352.03106559813 25312482 43035393 287792226304 1.70016488308021 6687.33817079351 25841471 46276699 288168476672 1.79079197929561 6227.07502693742 25618384 43785917 321591488512 1.70915999229303 7344.63294469772 26006097 45056206 298747666432 1.73252472295247 6630.55532088077 26684805 45196730 351100243968 1.69372532420604 7768.26650883814 24025872 42450135 353265467392 1.76685095966548 8321.89267223768 24080466 45510525 371726323712 1.88993539410741 8167.91991988666 23195936 45095051 326473826304 1.94409274969546 7239.68193990955 23653302 43312705 307549573120 1.83114835298683 7100.67803707942 21589455 40034670 322982109184 1.85436223378497 8067.56017182107 22469039 42042723 314323701760 1.87114023879704 7476.29266924504 23647633 43486098 370003841024 1.83891969230071 8508.55464254346 23750561 37387139 320471453696 1.57415814304344 8571.70305799542 23142315 38640274 329341046784 1.66968058294946 8523.25857689312 23539469 39573256 292528910336 1.68114480407353 7392.08596674481 23810938 37968499 277270380544 1.59458224619291 7302.64266027477 19361754 33610252 286391676928 1.73590946357443 8520.96190555191 20331818 34119736 256076865536 1.67814486633709 7505.24170339419 21017537 35862221 318755282944 1.70629988661374 8888.33078531305 21660731 42648077 329217507328 1.96891217567865 7719.39863380007 20708620 42285124 344562262016 2.04190931119505 8148.54562129225 21371937 43158447 312754188288 2.01939800777066 7246.65065654471 21447150 40034134 283613331456 1.86664120873869 7084.28790931259 18906469 36598724 302526169088 1.93577785465916 8266.03050663734 20086704 36824872 280208515072 1.83329589563325 7609.21898308296 20912511 40116356 340019290112 1.91829455582833 8475.82691987278 17728197 30717152 270751887360 1.73267208165613 8814.35516417668 16778676 30875765 267493560320 1.84017886751017 8663.54437922429 17700395 31528725 239652761600 1.78124414737637 7601.09270514428 17727766 31338207 232399462400 1.76774710361136 7415.85063880649 15488369 27225173 246367821824 1.75778179096844 9049.26561252705 16332731 29287976 227973730304 1.7932075168568 7783.86769724204 17043318 31659676 274151649280 1.85760049774346 8659.33211950748 21627836 34504152 279215091712 1.59535850003671 8092.2171833697 21244729 35619286 303324131328 1.67661757417569 8515.72744405938 22132156 38534232 281272401920 1.74109707160929 7299.28656473548 22035014 34627308 246920048640 1.57146748352418 7130.78962534425 20277457 33126067 265162657792 1.63364010585746 8004.65258347754 20669142 34587911 254815776768 1.67340816566067 7367.1918714027 21648239 34364823 292156514304 1.58741886580243 8501.61557078295 21117643 34737044 292367892480 1.64492997632359 8416.60253186771 20531946 37038043 292538568704 1.8039226773731 7898.32682855301 21393711 35682241 257189515264 1.66788459468299 7207.77361668512 21738966 34753281 252140285952 1.59866301828707 7255.1505554828 19197606 32922066 269381632000 1.71490476468785 8182.40361950553 20044574 33864896 245486792704 1.68947945713389 7249.00477190304 20601681 35851902 305202065408 1.74024158514055 8512.85561943129 1.76995040322111 8014.69622126768 So average fragment size is around 8 KiB, and the ratio between requests / fragments a bit lower than two.
And after reading your mails it might not be a problem at all. But we will start collecting this information in the coming weeks.
We will be re-provisioning all our OSDs, so that might be a good time to look at the behavior and development of "cnt versus frags" ratio.
After we completely emptied a host, even after having the OSDs run idle for a couple of hours, the fragmentation ratio would not drop lower than 0.27 for some OSDs, and up to 0.62 for others. Is it expected that this will not go to ~ zero?
You might be facing the issue fixed by https://github.com/ceph/ceph/pull/49885
Possibly. I have read some tracker tickets that got mentioned in PRs [1,2]. The problem seems to reveal itself in Pacific release. I wonder if this has something to do with the change in default allocator: bitmap -> hybrid in Pacific. BlueFS 4K allocation unit will not be backported to Pacific [3]. Would it make sense to skip re-provisiong OSDs in Pacific altogether and do re-provisioning in Quincy release with BlueFS 4K alloc size support [4]? Gr. Stefan [1]: https://tracker.ceph.com/issues/58022 [2]: https://tracker.ceph.com/issues/57672 [3]: https://tracker.ceph.com/issues/58589 [4]: https://tracker.ceph.com/issues/58588
Hi Stefan, given that allocation probes include every allocation (including short 4K ones) your stats look pretty high indeed. Although you omitted historic probes so it's hard to tell if there is negative trend in it.. As I mentioned in my reply to Hector one might want to make further investigation by e.g. building a histogram (chunk-size, num chanks) using the output from 'ceph tell osd.N bluestore allocator dump block' command and monitoring how it evolves over time. Script to build such a histogram still to be written. ;) As for Pacific release being a culprit - likely it is. But there were two major updates which could have the impact. Both came in the same PR (https://github.com/ceph/ceph/pull/34588): 1. 4K allocation unit for spinners 2. Switch to avl/hybrid allocator. Honestly I'd rather bet on 1.
BlueFS 4K allocation unit will not be backported to Pacific [3]. Would it make sense to skip re-provisiong OSDs in Pacific altogether and do re-provisioning in Quincy release with BlueFS 4K alloc size support [4]?
IIRC this feature doesn't require OSD redeployment - new superblock format is applied on-the-fly and 4K allocations are enabled immediately. So there is no specific requirement to re-provision OSD at Quincy+. Hence you're free to go with Pacific and enable 4K for BlueFS later in Quincy. Thanks, Igor On 26/05/2023 16:03, Stefan Kooman wrote:
On 5/25/23 22:12, Igor Fedotov wrote:
On 25/05/2023 20:36, Stefan Kooman wrote:
On 5/25/23 18:17, Igor Fedotov wrote:
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO it's mostly like a very imprecise first turn marker to alert that something might be wrong. But not a real quantitative high-quality estimate.
Chiming in on the high fragmentation issue. We started collecting "fragmentation_rating" of each OSD this afternoon. All OSDs that have been provisioned a year ago have a fragmentation rating of ~ 0.9. Not sure for how long they are on this level.
Could you please collect allocation probes from existing OSD logs? Just a few samples from different OSDs...
10 OSDs from one host, but I have checked other nodes and they are similar:
CNT FRAG Size Ratio Avg Frag size 21350923 37146899 317040259072 1.73982637659271 8534.77053554322 20951932 38122769 317841477632 1.8195347808498 8337.31352599283 21188454 37298950 278389411840 1.76034315670223 7463.73321072041 21605451 39369462 270427185152 1.82220042525379 6868.95810646333 19215230 36063713 290967818240 1.87682962941375 8068.16032059705 19293599 35464928 269238423552 1.83817068033807 7591.68109835159 19963538 36088151 315796836352 1.80770317365589 8750.70702159277 18030613 31753098 297826177024 1.76106591606176 9379.43683554909 17889602 31718012 299550142464 1.77298589426417 9444.16511551859 18475332 33264944 266053271552 1.80050588536109 7998.0074985847 18618154 31914219 254801883136 1.71414518324427 7983.96110323113 16437108 29421873 275350355968 1.78996651965784 9358.69568766067 17164338 28605353 249404649472 1.66655731202683 8718.81040838755 17895480 29658102 309047177216 1.65729569701399 10420.3288941416 19546560 34588509 301368737792 1.76954456436324 8712.97279081905 18525784 34806856 314875801600 1.87883309014075 9046.37297893266 18550989 35236438 273069948928 1.89943716747393 7749.64679823767 19085807 34605572 255512043520 1.81315738967705 7383.55209155335 17203820 31205542 277097357312 1.81387284916954 8879.74826112618 18003801 33723670 269696761856 1.87314167713807 7997.25420916525 18655425 33227176 306511810560 1.78109992133655 9224.7325069094 26380965 45627920 335281111040 1.72957736762093 7348.15680925188 24923956 44721109 328790982656 1.79430219664968 7352.03106559813 25312482 43035393 287792226304 1.70016488308021 6687.33817079351 25841471 46276699 288168476672 1.79079197929561 6227.07502693742 25618384 43785917 321591488512 1.70915999229303 7344.63294469772 26006097 45056206 298747666432 1.73252472295247 6630.55532088077 26684805 45196730 351100243968 1.69372532420604 7768.26650883814 24025872 42450135 353265467392 1.76685095966548 8321.89267223768 24080466 45510525 371726323712 1.88993539410741 8167.91991988666 23195936 45095051 326473826304 1.94409274969546 7239.68193990955 23653302 43312705 307549573120 1.83114835298683 7100.67803707942 21589455 40034670 322982109184 1.85436223378497 8067.56017182107 22469039 42042723 314323701760 1.87114023879704 7476.29266924504 23647633 43486098 370003841024 1.83891969230071 8508.55464254346 23750561 37387139 320471453696 1.57415814304344 8571.70305799542 23142315 38640274 329341046784 1.66968058294946 8523.25857689312 23539469 39573256 292528910336 1.68114480407353 7392.08596674481 23810938 37968499 277270380544 1.59458224619291 7302.64266027477 19361754 33610252 286391676928 1.73590946357443 8520.96190555191 20331818 34119736 256076865536 1.67814486633709 7505.24170339419 21017537 35862221 318755282944 1.70629988661374 8888.33078531305 21660731 42648077 329217507328 1.96891217567865 7719.39863380007 20708620 42285124 344562262016 2.04190931119505 8148.54562129225 21371937 43158447 312754188288 2.01939800777066 7246.65065654471 21447150 40034134 283613331456 1.86664120873869 7084.28790931259 18906469 36598724 302526169088 1.93577785465916 8266.03050663734 20086704 36824872 280208515072 1.83329589563325 7609.21898308296 20912511 40116356 340019290112 1.91829455582833 8475.82691987278 17728197 30717152 270751887360 1.73267208165613 8814.35516417668 16778676 30875765 267493560320 1.84017886751017 8663.54437922429 17700395 31528725 239652761600 1.78124414737637 7601.09270514428 17727766 31338207 232399462400 1.76774710361136 7415.85063880649 15488369 27225173 246367821824 1.75778179096844 9049.26561252705 16332731 29287976 227973730304 1.7932075168568 7783.86769724204 17043318 31659676 274151649280 1.85760049774346 8659.33211950748 21627836 34504152 279215091712 1.59535850003671 8092.2171833697 21244729 35619286 303324131328 1.67661757417569 8515.72744405938 22132156 38534232 281272401920 1.74109707160929 7299.28656473548 22035014 34627308 246920048640 1.57146748352418 7130.78962534425 20277457 33126067 265162657792 1.63364010585746 8004.65258347754 20669142 34587911 254815776768 1.67340816566067 7367.1918714027 21648239 34364823 292156514304 1.58741886580243 8501.61557078295 21117643 34737044 292367892480 1.64492997632359 8416.60253186771 20531946 37038043 292538568704 1.8039226773731 7898.32682855301 21393711 35682241 257189515264 1.66788459468299 7207.77361668512 21738966 34753281 252140285952 1.59866301828707 7255.1505554828 19197606 32922066 269381632000 1.71490476468785 8182.40361950553 20044574 33864896 245486792704 1.68947945713389 7249.00477190304 20601681 35851902 305202065408 1.74024158514055 8512.85561943129 1.76995040322111 8014.69622126768
So average fragment size is around 8 KiB, and the ratio between requests / fragments a bit lower than two.
And after reading your mails it might not be a problem at all. But we will start collecting this information in the coming weeks.
We will be re-provisioning all our OSDs, so that might be a good time to look at the behavior and development of "cnt versus frags" ratio.
After we completely emptied a host, even after having the OSDs run idle for a couple of hours, the fragmentation ratio would not drop lower than 0.27 for some OSDs, and up to 0.62 for others. Is it expected that this will not go to ~ zero?
You might be facing the issue fixed by https://github.com/ceph/ceph/pull/49885
Possibly.
I have read some tracker tickets that got mentioned in PRs [1,2]. The problem seems to reveal itself in Pacific release. I wonder if this has something to do with the change in default allocator: bitmap -> hybrid in Pacific.
BlueFS 4K allocation unit will not be backported to Pacific [3]. Would it make sense to skip re-provisiong OSDs in Pacific altogether and do re-provisioning in Quincy release with BlueFS 4K alloc size support [4]?
Gr. Stefan
[1]: https://tracker.ceph.com/issues/58022 [2]: https://tracker.ceph.com/issues/57672 [3]: https://tracker.ceph.com/issues/58589 [4]: https://tracker.ceph.com/issues/58588
On 5/29/23 15:52, Igor Fedotov wrote:
Hi Stefan,
given that allocation probes include every allocation (including short 4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there is negative trend in it..
I did not omit them. We (currently) don't store logs for longer than 7 days. I will increase the interval in which the probes get created (every hour).
As I mentioned in my reply to Hector one might want to make further investigation by e.g. building a histogram (chunk-size, num chanks) using the output from 'ceph tell osd.N bluestore allocator dump block' command and monitoring how it evolves over time. Script to build such a histogram still to be written. ;).
We started to investigate such a script. But when we issue a "ceph tell osd.N bluestore allocator dump block" on OSDs that are primary for three or more CephFS metadata PGs, that will cause a massive amount of slow ops (thousands), osd op tp threads will time out (2023-05-31T11:52:35.454+0200 7fee13285700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fedf6fa5700' had timed out after 15.000000954s) and the OSD will reboot itself. This is true for SSD as well as NVMe OSDs. So it seems that the whole OSD is just busy processing this data, and production IO (client / rep ops) are just starved. Ideally this call would be asynchronous,processed in batches, and not hinder IO in any way. Should I open a tracker for this? So for us this is not a suitable way of obtaining this data. The offline way of doing this, ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-$id/ --allocator block free-dump > /root/osd.$id_free_dump did work and resulted in a 2.7 GiB file of JSON data. So that's quite a bit of data to process ...
As for Pacific release being a culprit - likely it is. But there were two major updates which could have the impact. Both came in the same PR (https://github.com/ceph/ceph/pull/34588):
1. 4K allocation unit for spinners
@Kevin: what drive types do you use in the clusters that are suffering from this problem? Did only HDD suffer from this after upgrading to Pacific?
2. Switch to avl/hybrid allocator.
Honestly I'd rather bet on 1.
We have no spinners. We have 4K alloc size since Luminous, bitmap since Luminous (12.2.13?). Not sure if we are suffering (more or less) on the 3 nodes that got provisioned / filled with hybrid allocator in use. We plan to do some experiments though: fill an OSD with PGs with bitmap allocator. At certain amount of PGs dump the free extents, until all PGs are present. Repeat this process with the same PGs on an OSD with hybrid allocator. My bet is on # 2 ;-).
BlueFS 4K allocation unit will not be backported to Pacific [3]. Would it make sense to skip re-provisiong OSDs in Pacific altogether and do re-provisioning in Quincy release with BlueFS 4K alloc size support [4]?
IIRC this feature doesn't require OSD redeployment - new superblock format is applied on-the-fly and 4K allocations are enabled immediately. So there is no specific requirement to re-provision OSD at Quincy+. Hence you're free to go with Pacific and enable 4K for BlueFS later in Quincy.
Ah, that's good to know. Gr. Stefan
On 31/05/2023 15:26, Stefan Kooman wrote:
On 5/29/23 15:52, Igor Fedotov wrote:
Hi Stefan,
given that allocation probes include every allocation (including short 4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there is negative trend in it..
I did not omit them. We (currently) don't store logs for longer than 7 days. I will increase the interval in which the probes get created (every hour).
Allocation probe contains historic data on its own, e.g. allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508> probe -1: 35168547, 46401246, 1199516209152 probe -3: 27275094, 35681802, 200121712640 probe -5: 34847167, 52539758, 271272230912 probe -9: 44291522, 60025613, 523997483008 probe -17: 10646313, 10646313, 155178434560 in the snippet above probes -1 through -17 are historic data 1 through 17 days (or more correctly probe attempts) back. The major idea behind this representation is to try to visualize how allocation fragmentation evolved without the need for grep through all the logs. From the info you shared it's unclear which records were for the current day and which were historic ones if any. Hence no way to estimate the degradation over time. Please note that probes are collected since OSD restart. Hence some historic records might be void if restart occurred not long ago.
As I mentioned in my reply to Hector one might want to make further investigation by e.g. building a histogram (chunk-size, num chanks) using the output from 'ceph tell osd.N bluestore allocator dump block' command and monitoring how it evolves over time. Script to build such a histogram still to be written. ;).
We started to investigate such a script. But when we issue a "ceph tell osd.N bluestore allocator dump block" on OSDs that are primary for three or more CephFS metadata PGs, that will cause a massive amount of slow ops (thousands), osd op tp threads will time out (2023-05-31T11:52:35.454+0200 7fee13285700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fedf6fa5700' had timed out after 15.000000954s) and the OSD will reboot itself. This is true for SSD as well as NVMe OSDs. So it seems that the whole OSD is just busy processing this data, and production IO (client / rep ops) are just starved. Ideally this call would be asynchronous,processed in batches, and not hinder IO in any way. Should I open a tracker for this?
ah... this makes sense, good to know.. I knew that this dump might be huge but never heard it causes that drastic impact.. Perhaps it's really big this time or you're writing it to slow device.. Unfortunately there is no simple enough way to process that in batches since we should collect a complete consistent snapshot made at a given point in time. Processing in batches would create potentially inconsistent chunks since allocation map is permanently updated by OSD which is processing regular user ops..
So for us this is not a suitable way of obtaining this data. The offline way of doing this, ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-$id/ --allocator block free-dump > /root/osd.$id_free_dump did work and resulted in a 2.7 GiB file of JSON data. So that's quite a bit of data to process ...
Yeah, offline method is fine too. In fact Ceph codebase has a way to convert this JSON file to a binary format which might drastically improve processing time and save disk space. The tool name is ceph_test_alloc_replay, it's primarily intended for dev purposes hence it's not very user-friendly. And I'm not sure it's included in regular ceph packages, perhaps you'll need to run it yourself.
As for Pacific release being a culprit - likely it is. But there were two major updates which could have the impact. Both came in the same PR (https://github.com/ceph/ceph/pull/34588):
1. 4K allocation unit for spinners
@Kevin: what drive types do you use in the clusters that are suffering from this problem? Did only HDD suffer from this after upgrading to Pacific?
2. Switch to avl/hybrid allocator.
Honestly I'd rather bet on 1.
We have no spinners. We have 4K alloc size since Luminous, bitmap since Luminous (12.2.13?). Not sure if we are suffering (more or less) on the 3 nodes that got provisioned / filled with hybrid allocator in use. We plan to do some experiments though: fill an OSD with PGs with bitmap allocator. At certain amount of PGs dump the free extents, until all PGs are present. Repeat this process with the same PGs on an OSD with hybrid allocator. My bet is on # 2 ;-).
Looking forward for the results... ;) Knowing internal design for both bitmap and hybrid allocator I'd be very surprised the latter one is worse in this regard...
>BlueFS 4K allocation unit will not be backported to Pacific [3]. Would it make sense to skip re-provisiong OSDs in Pacific altogether and do re-provisioning in Quincy release with BlueFS 4K alloc size support [4]?
IIRC this feature doesn't require OSD redeployment - new superblock format is applied on-the-fly and 4K allocations are enabled immediately. So there is no specific requirement to re-provision OSD at Quincy+. Hence you're free to go with Pacific and enable 4K for BlueFS later in Quincy.
Ah, that's good to know.
Gr. Stefan
On 5/31/23 09:15, Igor Fedotov wrote:
On 31/05/2023 15:26, Stefan Kooman wrote:
On 5/29/23 15:52, Igor Fedotov wrote:
Hi Stefan,
given that allocation probes include every allocation (including short 4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there is negative trend in it..
I did not omit them. We (currently) don't store logs for longer than 7 days. I will increase the interval in which the probes get created (every hour).
Allocation probe contains historic data on its own, e.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508> probe -1: 35168547, 46401246, 1199516209152 probe -3: 27275094, 35681802, 200121712640 probe -5: 34847167, 52539758, 271272230912 probe -9: 44291522, 60025613, 523997483008 probe -17: 10646313, 10646313, 155178434560
in the snippet above probes -1 through -17 are historic data 1 through 17 days (or more correctly probe attempts) back.
The major idea behind this representation is to try to visualize how allocation fragmentation evolved without the need for grep through all the logs.
From the info you shared it's unclear which records were for the current day and which were historic ones if any.
Hence no way to estimate the degradation over time.
Please note that probes are collected since OSD restart. Hence some historic records might be void if restart occurred not long ago.
As I mentioned in my reply to Hector one might want to make further investigation by e.g. building a histogram (chunk-size, num chanks) using the output from 'ceph tell osd.N bluestore allocator dump block' command and monitoring how it evolves over time. Script to build such a histogram still to be written. ;).
We started to investigate such a script. But when we issue a "ceph tell osd.N bluestore allocator dump block" on OSDs that are primary for three or more CephFS metadata PGs, that will cause a massive amount of slow ops (thousands), osd op tp threads will time out (2023-05-31T11:52:35.454+0200 7fee13285700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fedf6fa5700' had timed out after 15.000000954s) and the OSD will reboot itself. This is true for SSD as well as NVMe OSDs. So it seems that the whole OSD is just busy processing this data, and production IO (client / rep ops) are just starved. Ideally this call would be asynchronous,processed in batches, and not hinder IO in any way. Should I open a tracker for this?
ah... this makes sense, good to know.. I knew that this dump might be huge but never heard it causes that drastic impact.. Perhaps it's really big this time or you're writing it to slow device..
Unfortunately there is no simple enough way to process that in batches since we should collect a complete consistent snapshot made at a given point in time. Processing in batches would create potentially inconsistent chunks since allocation map is permanently updated by OSD which is processing regular user ops..
So for us this is not a suitable way of obtaining this data. The offline way of doing this, ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-$id/ --allocator block free-dump > /root/osd.$id_free_dump did work and resulted in a 2.7 GiB file of JSON data. So that's quite a bit of data to process ...
Yeah, offline method is fine too. In fact Ceph codebase has a way to convert this JSON file to a binary format which might drastically improve processing time and save disk space.
The tool name is ceph_test_alloc_replay, it's primarily intended for dev purposes hence it's not very user-friendly. And I'm not sure it's included in regular ceph packages, perhaps you'll need to run it yourself.
As for Pacific release being a culprit - likely it is. But there were two major updates which could have the impact. Both came in the same PR (https://github.com/ceph/ceph/pull/34588):
1. 4K allocation unit for spinners
@Kevin: what drive types do you use in the clusters that are suffering from this problem? Did only HDD suffer from this after upgrading to Pacific?
2. Switch to avl/hybrid allocator.
Honestly I'd rather bet on 1.
We have no spinners. We have 4K alloc size since Luminous, bitmap since Luminous (12.2.13?). Not sure if we are suffering (more or less) on the 3 nodes that got provisioned / filled with hybrid allocator in use. We plan to do some experiments though: fill an OSD with PGs with bitmap allocator. At certain amount of PGs dump the free extents, until all PGs are present. Repeat this process with the same PGs on an OSD with hybrid allocator. My bet is on # 2 ;-).
Looking forward for the results... ;) Knowing internal design for both bitmap and hybrid allocator I'd be very surprised the latter one is worse in this regard...
Related to this, I was a little surprised to learn how the hybrid allocator works. I figured we would do something like have a coarse grained implementation of one data structure and implement the other at the leaves. Adam was explaining that this isn't what we do though, we just switch over at some point?
>BlueFS 4K allocation unit will not be backported to Pacific [3]. Would it make sense to skip re-provisiong OSDs in Pacific altogether and do re-provisioning in Quincy release with BlueFS 4K alloc size support [4]?
IIRC this feature doesn't require OSD redeployment - new superblock format is applied on-the-fly and 4K allocations are enabled immediately. So there is no specific requirement to re-provision OSD at Quincy+. Hence you're free to go with Pacific and enable 4K for BlueFS later in Quincy.
Ah, that's good to know.
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
On 5/31/23 16:15, Igor Fedotov wrote:
On 31/05/2023 15:26, Stefan Kooman wrote:
On 5/29/23 15:52, Igor Fedotov wrote:
Hi Stefan,
given that allocation probes include every allocation (including short 4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there is negative trend in it..
I did not omit them. We (currently) don't store logs for longer than 7 days. I will increase the interval in which the probes get created (every hour).
Allocation probe contains historic data on its own, e.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508> probe -1: 35168547, 46401246, 1199516209152 probe -3: 27275094, 35681802, 200121712640 probe -5: 34847167, 52539758, 271272230912 probe -9: 44291522, 60025613, 523997483008 probe -17: 10646313, 10646313, 155178434560
in the snippet above probes -1 through -17 are historic data 1 through 17 days (or more correctly probe attempts) back.
Ah, I just blindly applied your suggested grep keywords, but then you don't get the historic probes, here we go: ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0 bluestore(/var/lib/ceph/osd/ceph-55) allocation stats probe 15: cnt: 19940282 frags: 37474984 size: 331549941760 ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0 bluestore(/var/lib/ceph/osd/ceph-55) probe -1: 19269005, 35807252, 265695309824 ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0 bluestore(/var/lib/ceph/osd/ceph-55) probe -3: 18281841, 33452239, 321271795712 ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0 bluestore(/var/lib/ceph/osd/ceph-55) probe -7: 21188454, 37298950, 278389411840 ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0 bluestore(/var/lib/ceph/osd/ceph-55) probe -15: 20079763, 34770424, 306357551104 ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0 bluestore(/var/lib/ceph/osd/ceph-55) probe -31: 0, 0, 0 ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0 bluestore(/var/lib/ceph/osd/ceph-66) allocation stats probe 15: cnt: 18130163 frags: 32547753 size: 286495289344 ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0 bluestore(/var/lib/ceph/osd/ceph-66) probe -1: 16671297, 28781657, 219492544512 ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0 bluestore(/var/lib/ceph/osd/ceph-66) probe -3: 16781500, 31159184, 306530078720 ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0 bluestore(/var/lib/ceph/osd/ceph-66) probe -7: 18475332, 33264944, 266053271552 ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0 bluestore(/var/lib/ceph/osd/ceph-66) probe -15: 18799693, 32644833, 270106509312 ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0 bluestore(/var/lib/ceph/osd/ceph-66) probe -31: 0, 0, 0 ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0 bluestore(/var/lib/ceph/osd/ceph-67) allocation stats probe 15: cnt: 19073940 frags: 39349763 size: 350442409984 ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0 bluestore(/var/lib/ceph/osd/ceph-67) probe -1: 19346357, 40814157, 291156762624 ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0 bluestore(/var/lib/ceph/osd/ceph-67) probe -3: 18880370, 33120073, 329544183808 ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0 bluestore(/var/lib/ceph/osd/ceph-67) probe -7: 18550989, 35236438, 273069948928 ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0 bluestore(/var/lib/ceph/osd/ceph-67) probe -15: 18664247, 36208229, 327013150720 ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0 bluestore(/var/lib/ceph/osd/ceph-67) probe -31: 0, 0, 0 ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0 bluestore(/var/lib/ceph/osd/ceph-68) allocation stats probe 15: cnt: 24318407 frags: 42340467 size: 324867026944 ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0 bluestore(/var/lib/ceph/osd/ceph-68) probe -1: 24202548, 41934313, 263141662720 ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0 bluestore(/var/lib/ceph/osd/ceph-68) probe -3: 24258584, 43985640, 348764803072 ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0 bluestore(/var/lib/ceph/osd/ceph-68) probe -7: 25312482, 43035393, 287792226304 ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0 bluestore(/var/lib/ceph/osd/ceph-68) probe -15: 25280732, 40994669, 298337083392 ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0 bluestore(/var/lib/ceph/osd/ceph-68) probe -31: 0, 0, 0 ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0 bluestore(/var/lib/ceph/osd/ceph-75) allocation stats probe 15: cnt: 21755123 frags: 42279513 size: 364872880128 ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0 bluestore(/var/lib/ceph/osd/ceph-75) probe -1: 20263259, 38412587, 291021299712 ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0 bluestore(/var/lib/ceph/osd/ceph-75) probe -3: 19733485, 38065426, 353624813568 ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0 bluestore(/var/lib/ceph/osd/ceph-75) probe -7: 23195936, 45095051, 326473826304 ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0 bluestore(/var/lib/ceph/osd/ceph-75) probe -15: 24094232, 45266621, 367246442496 ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0 bluestore(/var/lib/ceph/osd/ceph-75) probe -31: 0, 0, 0 ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0 bluestore(/var/lib/ceph/osd/ceph-76) allocation stats probe 15: cnt: 21035351 frags: 34478758 size: 296729862144 ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0 bluestore(/var/lib/ceph/osd/ceph-76) probe -1: 20215704, 32981561, 229284646912 ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0 bluestore(/var/lib/ceph/osd/ceph-76) probe -3: 19533561, 31158296, 311621177344 ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0 bluestore(/var/lib/ceph/osd/ceph-76) probe -7: 23539469, 39573256, 292528910336 ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0 bluestore(/var/lib/ceph/osd/ceph-76) probe -15: 21308764, 34003659, 272235884544 ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0 bluestore(/var/lib/ceph/osd/ceph-76) probe -31: 0, 0, 0 ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0 bluestore(/var/lib/ceph/osd/ceph-77) allocation stats probe 15: cnt: 21619552 frags: 38792793 size: 319570493440 ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0 bluestore(/var/lib/ceph/osd/ceph-77) probe -1: 19698381, 33288562, 245733462016 ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0 bluestore(/var/lib/ceph/osd/ceph-77) probe -3: 19345976, 35673735, 327406686208 ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0 bluestore(/var/lib/ceph/osd/ceph-77) probe -7: 21371937, 43158447, 312754188288 ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0 bluestore(/var/lib/ceph/osd/ceph-77) probe -15: 23144793, 41159338, 309643296768 ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0 bluestore(/var/lib/ceph/osd/ceph-77) probe -31: 0, 0, 0 ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0 bluestore(/var/lib/ceph/osd/ceph-84) allocation stats probe 15: cnt: 17348416 frags: 32011874 size: 257342664704 ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0 bluestore(/var/lib/ceph/osd/ceph-84) probe -1: 16688730, 29763231, 204166496256 ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0 bluestore(/var/lib/ceph/osd/ceph-84) probe -3: 15778722, 28658535, 269868609536 ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0 bluestore(/var/lib/ceph/osd/ceph-84) probe -7: 17700395, 31528725, 239652761600 ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0 bluestore(/var/lib/ceph/osd/ceph-84) probe -15: 17331958, 29820789, 239322513408 ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0 bluestore(/var/lib/ceph/osd/ceph-84) probe -31: 0, 0, 0 ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0 bluestore(/var/lib/ceph/osd/ceph-85) allocation stats probe 15: cnt: 21817405 frags: 35609259 size: 304181972992 ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0 bluestore(/var/lib/ceph/osd/ceph-85) probe -1: 21402843, 35001032, 247483416576 ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0 bluestore(/var/lib/ceph/osd/ceph-85) probe -3: 20555960, 33268576, 297290342400 ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0 bluestore(/var/lib/ceph/osd/ceph-85) probe -7: 22132156, 38534232, 281272401920 ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0 bluestore(/var/lib/ceph/osd/ceph-85) probe -15: 22188633, 37241809, 290561724416 ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0 bluestore(/var/lib/ceph/osd/ceph-85) probe -31: 0, 0, 0 ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0 bluestore(/var/lib/ceph/osd/ceph-86) allocation stats probe 15: cnt: 20080223 frags: 33329658 size: 278508113920 ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0 bluestore(/var/lib/ceph/osd/ceph-86) probe -1: 20682155, 32852611, 219409920000 ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0 bluestore(/var/lib/ceph/osd/ceph-86) probe -3: 19216200, 34658404, 296857645056 ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0 bluestore(/var/lib/ceph/osd/ceph-86) probe -7: 21393711, 35682241, 257189515264 ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0 bluestore(/var/lib/ceph/osd/ceph-86) probe -15: 20233752, 31906022, 256953790464 ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0 bluestore(/var/lib/ceph/osd/ceph-86) probe -31: 0, 0, 0 Thanks, Gr. Stefan
So chiming in, I think something is definitely wrong with at *least* the
frag score.
Here's what happened so far:
1. I had 8 OSDs (all 8T HDDs)
2. I added 2 more (osd.0,1) , with Quincy defaults
3. I marked 2 old ones out (the ones that seemed to be struggling the
most with IOPS)
4. I added 2 more (osd.2,3), but this time I had previously set
bluestore_min_alloc_size_hdd to 16K as an experiment
This has all happened in the space of a ~week. That means there was data
movement into the first 2 new OSDs, then before that completed I added 2
new OSDs. So I would expect some data thashing on the first 2, but
nothing extreme.
The fragmentation scores for the 4 new OSDs are, respectively:
0.746, 0.835, 0.160, 0.067
That seems ridiculous for the first two, it's only been a week. The
newest two seem in better shape, though those mostly would've seen only
data moving in, not out. The rebalance isn't done yet, but it's almost
done and all 4 OSDs have a similar fullness level at this time.
Looking at alloc stats:
ceph-0) allocation stats probe 6: cnt: 2219302 frags: 2328003 size:
1238454677504
ceph-0) probe -1: 1848577, 1970325, 1022324588544
ceph-0) probe -2: 848301, 862622, 505329963008
ceph-0) probe -6: 2187448, 2187448, 1055241568256
ceph-0) probe -14: 0, 0, 0
ceph-0) probe -22: 0, 0, 0
ceph-1) allocation stats probe 6: cnt: 1882396 frags: 1947321 size:
1054829641728
ceph-1) probe -1: 2212293, 2345923, 1215418728448
ceph-1) probe -2: 1471623, 1525498, 826984652800
ceph-1) probe -6: 2095298, 2095298, 1000065933312
ceph-1) probe -14: 0, 0, 0
ceph-1) probe -22: 0, 0, 0
ceph-2) allocation stats probe 3: cnt: 2760200 frags: 2760200 size:
1554513903616
ceph-2) probe -1: 2584046, 2584046, 1498140393472
ceph-2) probe -3: 1696921, 1696921, 869424496640
ceph-2) probe -7: 0, 0, 0
ceph-2) probe -11: 0, 0, 0
ceph-2) probe -19: 0, 0, 0
ceph-3) allocation stats probe 3: cnt: 2544818 frags: 2544818 size:
1432225021952
ceph-3) probe -1: 2688015, 2688015, 1515260739584
ceph-3) probe -3: 1086875, 1086875, 622025424896
ceph-3) probe -7: 0, 0, 0
ceph-3) probe -11: 0, 0, 0
ceph-3) probe -19: 0, 0, 0
So OSDs 2 and 3 (the latest ones to be added, note that these 4 new OSDs
are 0-3 since those IDs were free) are in good shape, but 0 and 1 are
already suffering from at least some fragmentation of objects, which is
a bit worrying when they are only ~70% full right now and only a week old.
I did delete a couple million small objects during the rebalance to try
to reduce load (I had some nasty directories), but that was cumulatively
only about 60GB of data. So while that could explain a high frag score
if there are now a million little holes in the free space map of the
OSDs (how is it calculated?), it should not actually cause new data
moving in to end up fragmented since there should be plenty of
unfragmented free space going around still.
I am now restarting OSDs 0 and 1 to see whether that makes the frag
score go down over time. I will do further analysis later with the raw
bluestore free space map, since I still have a bunch of rebalancing and
moving data around planned (I'm moving my cluster to new machines).
On 26/05/2023 00.29, Igor Fedotov wrote:
> Hi Hector,
>
> I can advise two tools for further fragmentation analysis:
>
> 1) One might want to use ceph-bluestore-tool's free-dump command to get
> a list of free chunks for an OSD and try to analyze whether it's really
> highly fragmented and lacks long enough extents. free-dump just returns
> a list of extents in json format, I can take a look to the output if
> shared...
>
> 2) You might want to look for allocation probs in OSD logs and see how
> fragmentation in allocated chunks has evolved.
>
> E.g.
>
> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
> probe -1: 35168547, 46401246, 1199516209152
> probe -3: 27275094, 35681802, 200121712640
> probe -5: 34847167, 52539758, 271272230912
> probe -9: 44291522, 60025613, 523997483008
> probe -17: 10646313, 10646313, 155178434560
>
> The first probe refers to the last day while others match days (or
> rather probes) -1, -3, -5, -9, -17
>
> 'cnt' column represents the amount of allocations performed in the
> previous 24 hours and 'frags' one shows amount of fragments in the
> resulted allocations. So significant mismatch between frags and cnt
> might indicate some issues with high fragmentation indeed.
>
> Apart from retrospective analysis you might also want how OSD behavior
> changes after reboot - e.g. wouldn't rebooted OSD produce less
> fragmentation... Which in turn might indicate some issues with BlueStore
> allocator..
>
> Just FYI: allocation probe printing interval is controlled by
> bluestore_alloc_stats_dump_interval parameter.
>
>
> Thanks,
>
> Igor
>
>
>
> On 24/05/2023 17:18, Hector Martin wrote:
>> On 24/05/2023 22.07, Mark Nelson wrote:
>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>> of using copy-on-write and never implementing any kind of
>>> defragmentation scheme. Adam and I have been talking about doing it
>>> now, probably piggybacking on scrub or other operations that already
>>> area reading all of the extents for an object anyway.
>>>
>>>
>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>> case here:
>>>
>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>
>>>
>>> Adam ended up going the extra mile and completely changed how shared
>>> blobs works which probably eliminates the need to do defrag on clone
>>> anymore from an rbd-mirror perspective, but I think we still need to
>>> identify any times we are doing full object reads of fragmented objects
>>> and consider defragmenting at that time. It might be clone, or scrub,
>>> or other things, but the point is that if we are already doing most of
>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>> clean it up isn't that bad, especially if we are doing it over the
>>> course of months or years and can help keep freespace less fragmented.
>> Note that my particular issue seemed to specifically be free space
>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>> objects other than those forced by the free space becoming fragmented
>> (unless there is some weird pathological use case I'm hitting). Most of
>> my write workloads are just copying files in bulk and incrementally
>> writing out files.
>>
>> Would simply defragging objects during scrub/etc help with free space
>> fragmentation itself? Those seem like two somewhat unrelated issues...
>> note that if free space is already fragmented, you wouldn't even have a
>> place to put down a defragmented object.
>>
>> Are there any stats I can look at to figure out how bad object and free
>> space fragmentation is? It would be nice to have some clearer data
>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>> fragmentation number :). Also would be interesting to get some kind of
>> trace of the bluestore ops the OSD is doing, so I can find out whether
>> it's doing something pathological that causes more fragmentation for
>> some reason.
>>
>>> Mark
>>>
>>>
>>> On 5/24/23 07:17, Hector Martin wrote:
>>>> Hi,
>>>>
>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>
>>>> ceph daemon osd.13 bluestore allocator score block
>>>> {
>>>> "fragmentation_rating": 0.77251526920454427
>>>> }
>>>>
>>>> These aren't that old, as I recreated them all around July last year.
>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>> Most of the data was written sequentially when the OSDs were created (I
>>>> rsynced everything from a remote backup). Since then more data has been
>>>> added, but not particularly quickly.
>>>>
>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>> figure out what was wrong. Eventually I did some block tracing and
>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>> free space fragmentation problem. Indeed, deleting some large files
>>>> opened up some larger free extents and resolved the problem, but only
>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>> up any more just slows down to an absolute crawl.
>>>>
>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>> situation, but obviously this is going to take forever.
>>>>
>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>> - Hector
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-leave@ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io
- Hector
So fragmentation score calculation was improved recently indeed,
seehttps://github.com/ceph/ceph/pull/49885
And yeah one can see some fragmentation in allocations for the first two
OSDs. Doesn't look that dramatic as fragmentation scores tell though.
Additionally you might want to collect free extents dump using 'ceph
tell osd.N ceph bluestore allocator dump block' command and do more
analysis on these data.
E.g. I'd recommend to build something like a histogram showing amount of
chunks for specific size range:
[1-4K]: N1 chunks
(4K-16]: N2 chunks
(16K-64K): N3
...
[16M-inf) : Nn chunks
This should be even more informative about fragmentation state -
particularly if observed in evolution.
Looking for volunteers to write a script for building such a histogram... ;)
Thanks,
Igor
On 28/05/2023 08:31, Hector Martin wrote:
> So chiming in, I think something is definitely wrong with at *least* the
> frag score.
>
> Here's what happened so far:
>
> 1. I had 8 OSDs (all 8T HDDs)
> 2. I added 2 more (osd.0,1) , with Quincy defaults
> 3. I marked 2 old ones out (the ones that seemed to be struggling the
> most with IOPS)
> 4. I added 2 more (osd.2,3), but this time I had previously set
> bluestore_min_alloc_size_hdd to 16K as an experiment
>
> This has all happened in the space of a ~week. That means there was data
> movement into the first 2 new OSDs, then before that completed I added 2
> new OSDs. So I would expect some data thashing on the first 2, but
> nothing extreme.
>
> The fragmentation scores for the 4 new OSDs are, respectively:
>
> 0.746, 0.835, 0.160, 0.067
>
> That seems ridiculous for the first two, it's only been a week. The
> newest two seem in better shape, though those mostly would've seen only
> data moving in, not out. The rebalance isn't done yet, but it's almost
> done and all 4 OSDs have a similar fullness level at this time.
>
> Looking at alloc stats:
>
> ceph-0) allocation stats probe 6: cnt: 2219302 frags: 2328003 size:
> 1238454677504
> ceph-0) probe -1: 1848577, 1970325, 1022324588544
> ceph-0) probe -2: 848301, 862622, 505329963008
> ceph-0) probe -6: 2187448, 2187448, 1055241568256
> ceph-0) probe -14: 0, 0, 0
> ceph-0) probe -22: 0, 0, 0
>
> ceph-1) allocation stats probe 6: cnt: 1882396 frags: 1947321 size:
> 1054829641728
> ceph-1) probe -1: 2212293, 2345923, 1215418728448
> ceph-1) probe -2: 1471623, 1525498, 826984652800
> ceph-1) probe -6: 2095298, 2095298, 1000065933312
> ceph-1) probe -14: 0, 0, 0
> ceph-1) probe -22: 0, 0, 0
>
> ceph-2) allocation stats probe 3: cnt: 2760200 frags: 2760200 size:
> 1554513903616
> ceph-2) probe -1: 2584046, 2584046, 1498140393472
> ceph-2) probe -3: 1696921, 1696921, 869424496640
> ceph-2) probe -7: 0, 0, 0
> ceph-2) probe -11: 0, 0, 0
> ceph-2) probe -19: 0, 0, 0
>
> ceph-3) allocation stats probe 3: cnt: 2544818 frags: 2544818 size:
> 1432225021952
> ceph-3) probe -1: 2688015, 2688015, 1515260739584
> ceph-3) probe -3: 1086875, 1086875, 622025424896
> ceph-3) probe -7: 0, 0, 0
> ceph-3) probe -11: 0, 0, 0
> ceph-3) probe -19: 0, 0, 0
>
> So OSDs 2 and 3 (the latest ones to be added, note that these 4 new OSDs
> are 0-3 since those IDs were free) are in good shape, but 0 and 1 are
> already suffering from at least some fragmentation of objects, which is
> a bit worrying when they are only ~70% full right now and only a week old.
>
> I did delete a couple million small objects during the rebalance to try
> to reduce load (I had some nasty directories), but that was cumulatively
> only about 60GB of data. So while that could explain a high frag score
> if there are now a million little holes in the free space map of the
> OSDs (how is it calculated?), it should not actually cause new data
> moving in to end up fragmented since there should be plenty of
> unfragmented free space going around still.
>
> I am now restarting OSDs 0 and 1 to see whether that makes the frag
> score go down over time. I will do further analysis later with the raw
> bluestore free space map, since I still have a bunch of rebalancing and
> moving data around planned (I'm moving my cluster to new machines).
>
> On 26/05/2023 00.29, Igor Fedotov wrote:
>> Hi Hector,
>>
>> I can advise two tools for further fragmentation analysis:
>>
>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>> a list of free chunks for an OSD and try to analyze whether it's really
>> highly fragmented and lacks long enough extents. free-dump just returns
>> a list of extents in json format, I can take a look to the output if
>> shared...
>>
>> 2) You might want to look for allocation probs in OSD logs and see how
>> fragmentation in allocated chunks has evolved.
>>
>> E.g.
>>
>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>> probe -1: 35168547, 46401246, 1199516209152
>> probe -3: 27275094, 35681802, 200121712640
>> probe -5: 34847167, 52539758, 271272230912
>> probe -9: 44291522, 60025613, 523997483008
>> probe -17: 10646313, 10646313, 155178434560
>>
>> The first probe refers to the last day while others match days (or
>> rather probes) -1, -3, -5, -9, -17
>>
>> 'cnt' column represents the amount of allocations performed in the
>> previous 24 hours and 'frags' one shows amount of fragments in the
>> resulted allocations. So significant mismatch between frags and cnt
>> might indicate some issues with high fragmentation indeed.
>>
>> Apart from retrospective analysis you might also want how OSD behavior
>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>> fragmentation... Which in turn might indicate some issues with BlueStore
>> allocator..
>>
>> Just FYI: allocation probe printing interval is controlled by
>> bluestore_alloc_stats_dump_interval parameter.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>> On 24/05/2023 17:18, Hector Martin wrote:
>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>>> of using copy-on-write and never implementing any kind of
>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>> now, probably piggybacking on scrub or other operations that already
>>>> area reading all of the extents for an object anyway.
>>>>
>>>>
>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>> case here:
>>>>
>>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>>
>>>>
>>>> Adam ended up going the extra mile and completely changed how shared
>>>> blobs works which probably eliminates the need to do defrag on clone
>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>> identify any times we are doing full object reads of fragmented objects
>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>> or other things, but the point is that if we are already doing most of
>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>> clean it up isn't that bad, especially if we are doing it over the
>>>> course of months or years and can help keep freespace less fragmented.
>>> Note that my particular issue seemed to specifically be free space
>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>> objects other than those forced by the free space becoming fragmented
>>> (unless there is some weird pathological use case I'm hitting). Most of
>>> my write workloads are just copying files in bulk and incrementally
>>> writing out files.
>>>
>>> Would simply defragging objects during scrub/etc help with free space
>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>> note that if free space is already fragmented, you wouldn't even have a
>>> place to put down a defragmented object.
>>>
>>> Are there any stats I can look at to figure out how bad object and free
>>> space fragmentation is? It would be nice to have some clearer data
>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>> fragmentation number :). Also would be interesting to get some kind of
>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>> it's doing something pathological that causes more fragmentation for
>>> some reason.
>>>
>>>> Mark
>>>>
>>>>
>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>> Hi,
>>>>>
>>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>>
>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>> {
>>>>> "fragmentation_rating": 0.77251526920454427
>>>>> }
>>>>>
>>>>> These aren't that old, as I recreated them all around July last year.
>>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>>> Most of the data was written sequentially when the OSDs were created (I
>>>>> rsynced everything from a remote backup). Since then more data has been
>>>>> added, but not particularly quickly.
>>>>>
>>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>> opened up some larger free extents and resolved the problem, but only
>>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>>> up any more just slows down to an absolute crawl.
>>>>>
>>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>>> situation, but obviously this is going to take forever.
>>>>>
>>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>>
>>>>> - Hector
>>>>> _______________________________________________
>>>>> ceph-users mailing list --ceph-users@ceph.io
>>>>> To unsubscribe send an email toceph-users-leave@ceph.io
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list --ceph-users@ceph.io
>>> To unsubscribe send an email toceph-users-leave@ceph.io
>> _______________________________________________
>> ceph-users mailing list --ceph-users@ceph.io
>> To unsubscribe send an email toceph-users-leave@ceph.io
> - Hector
>
On 29/05/2023 22.26, Igor Fedotov wrote:
> So fragmentation score calculation was improved recently indeed, see
> https://github.com/ceph/ceph/pull/49885
>
>
> And yeah one can see some fragmentation in allocations for the first two
> OSDs. Doesn't look that dramatic as fragmentation scores tell though.
>
>
> Additionally you might want to collect free extents dump using 'ceph
> tell osd.N ceph bluestore allocator dump block' command and do more
> analysis on these data.
>
> E.g. I'd recommend to build something like a histogram showing amount of
> chunks for specific size range:
>
> [1-4K]: N1 chunks
>
> (4K-16]: N2 chunks
>
> (16K-64K): N3
>
> ...
>
> [16M-inf) : Nn chunks
>
>
> This should be even more informative about fragmentation state -
> particularly if observed in evolution.
>
> Looking for volunteers to write a script for building such a histogram... ;)
I'm up for that, once I get through some other cluster maintenance I
need to deal with first :)
Backfill is almost done and I was finally able to destroy two OSDs, will
be doing a bunch of restructuring in the coming weeks. I can probably
get the script done partway through doing this, so I can see how the
distributions evolve over a bunch of data movement.
>
>
> Thanks,
>
> Igor
>
>
> On 28/05/2023 08:31, Hector Martin wrote:
>> So chiming in, I think something is definitely wrong with at *least* the
>> frag score.
>>
>> Here's what happened so far:
>>
>> 1. I had 8 OSDs (all 8T HDDs)
>> 2. I added 2 more (osd.0,1) , with Quincy defaults
>> 3. I marked 2 old ones out (the ones that seemed to be struggling the
>> most with IOPS)
>> 4. I added 2 more (osd.2,3), but this time I had previously set
>> bluestore_min_alloc_size_hdd to 16K as an experiment
>>
>> This has all happened in the space of a ~week. That means there was data
>> movement into the first 2 new OSDs, then before that completed I added 2
>> new OSDs. So I would expect some data thashing on the first 2, but
>> nothing extreme.
>>
>> The fragmentation scores for the 4 new OSDs are, respectively:
>>
>> 0.746, 0.835, 0.160, 0.067
>>
>> That seems ridiculous for the first two, it's only been a week. The
>> newest two seem in better shape, though those mostly would've seen only
>> data moving in, not out. The rebalance isn't done yet, but it's almost
>> done and all 4 OSDs have a similar fullness level at this time.
>>
>> Looking at alloc stats:
>>
>> ceph-0) allocation stats probe 6: cnt: 2219302 frags: 2328003 size:
>> 1238454677504
>> ceph-0) probe -1: 1848577, 1970325, 1022324588544
>> ceph-0) probe -2: 848301, 862622, 505329963008
>> ceph-0) probe -6: 2187448, 2187448, 1055241568256
>> ceph-0) probe -14: 0, 0, 0
>> ceph-0) probe -22: 0, 0, 0
>>
>> ceph-1) allocation stats probe 6: cnt: 1882396 frags: 1947321 size:
>> 1054829641728
>> ceph-1) probe -1: 2212293, 2345923, 1215418728448
>> ceph-1) probe -2: 1471623, 1525498, 826984652800
>> ceph-1) probe -6: 2095298, 2095298, 1000065933312
>> ceph-1) probe -14: 0, 0, 0
>> ceph-1) probe -22: 0, 0, 0
>>
>> ceph-2) allocation stats probe 3: cnt: 2760200 frags: 2760200 size:
>> 1554513903616
>> ceph-2) probe -1: 2584046, 2584046, 1498140393472
>> ceph-2) probe -3: 1696921, 1696921, 869424496640
>> ceph-2) probe -7: 0, 0, 0
>> ceph-2) probe -11: 0, 0, 0
>> ceph-2) probe -19: 0, 0, 0
>>
>> ceph-3) allocation stats probe 3: cnt: 2544818 frags: 2544818 size:
>> 1432225021952
>> ceph-3) probe -1: 2688015, 2688015, 1515260739584
>> ceph-3) probe -3: 1086875, 1086875, 622025424896
>> ceph-3) probe -7: 0, 0, 0
>> ceph-3) probe -11: 0, 0, 0
>> ceph-3) probe -19: 0, 0, 0
>>
>> So OSDs 2 and 3 (the latest ones to be added, note that these 4 new OSDs
>> are 0-3 since those IDs were free) are in good shape, but 0 and 1 are
>> already suffering from at least some fragmentation of objects, which is
>> a bit worrying when they are only ~70% full right now and only a week old.
>>
>> I did delete a couple million small objects during the rebalance to try
>> to reduce load (I had some nasty directories), but that was cumulatively
>> only about 60GB of data. So while that could explain a high frag score
>> if there are now a million little holes in the free space map of the
>> OSDs (how is it calculated?), it should not actually cause new data
>> moving in to end up fragmented since there should be plenty of
>> unfragmented free space going around still.
>>
>> I am now restarting OSDs 0 and 1 to see whether that makes the frag
>> score go down over time. I will do further analysis later with the raw
>> bluestore free space map, since I still have a bunch of rebalancing and
>> moving data around planned (I'm moving my cluster to new machines).
>>
>> On 26/05/2023 00.29, Igor Fedotov wrote:
>>> Hi Hector,
>>>
>>> I can advise two tools for further fragmentation analysis:
>>>
>>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>>> a list of free chunks for an OSD and try to analyze whether it's really
>>> highly fragmented and lacks long enough extents. free-dump just returns
>>> a list of extents in json format, I can take a look to the output if
>>> shared...
>>>
>>> 2) You might want to look for allocation probs in OSD logs and see how
>>> fragmentation in allocated chunks has evolved.
>>>
>>> E.g.
>>>
>>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>>> probe -1: 35168547, 46401246, 1199516209152
>>> probe -3: 27275094, 35681802, 200121712640
>>> probe -5: 34847167, 52539758, 271272230912
>>> probe -9: 44291522, 60025613, 523997483008
>>> probe -17: 10646313, 10646313, 155178434560
>>>
>>> The first probe refers to the last day while others match days (or
>>> rather probes) -1, -3, -5, -9, -17
>>>
>>> 'cnt' column represents the amount of allocations performed in the
>>> previous 24 hours and 'frags' one shows amount of fragments in the
>>> resulted allocations. So significant mismatch between frags and cnt
>>> might indicate some issues with high fragmentation indeed.
>>>
>>> Apart from retrospective analysis you might also want how OSD behavior
>>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>>> fragmentation... Which in turn might indicate some issues with BlueStore
>>> allocator..
>>>
>>> Just FYI: allocation probe printing interval is controlled by
>>> bluestore_alloc_stats_dump_interval parameter.
>>>
>>>
>>> Thanks,
>>>
>>> Igor
>>>
>>>
>>>
>>> On 24/05/2023 17:18, Hector Martin wrote:
>>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>>>> of using copy-on-write and never implementing any kind of
>>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>>> now, probably piggybacking on scrub or other operations that already
>>>>> area reading all of the extents for an object anyway.
>>>>>
>>>>>
>>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>>> case here:
>>>>>
>>>>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5dfdf
>>>>>
>>>>>
>>>>> Adam ended up going the extra mile and completely changed how shared
>>>>> blobs works which probably eliminates the need to do defrag on clone
>>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>>> identify any times we are doing full object reads of fragmented objects
>>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>>> or other things, but the point is that if we are already doing most of
>>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>>> clean it up isn't that bad, especially if we are doing it over the
>>>>> course of months or years and can help keep freespace less fragmented.
>>>> Note that my particular issue seemed to specifically be free space
>>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>>> objects other than those forced by the free space becoming fragmented
>>>> (unless there is some weird pathological use case I'm hitting). Most of
>>>> my write workloads are just copying files in bulk and incrementally
>>>> writing out files.
>>>>
>>>> Would simply defragging objects during scrub/etc help with free space
>>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>>> note that if free space is already fragmented, you wouldn't even have a
>>>> place to put down a defragmented object.
>>>>
>>>> Are there any stats I can look at to figure out how bad object and free
>>>> space fragmentation is? It would be nice to have some clearer data
>>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>>> fragmentation number :). Also would be interesting to get some kind of
>>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>>> it's doing something pathological that causes more fragmentation for
>>>> some reason.
>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>>>>
>>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>>> {
>>>>>> "fragmentation_rating": 0.77251526920454427
>>>>>> }
>>>>>>
>>>>>> These aren't that old, as I recreated them all around July last year.
>>>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>>>> Most of the data was written sequentially when the OSDs were created (I
>>>>>> rsynced everything from a remote backup). Since then more data has been
>>>>>> added, but not particularly quickly.
>>>>>>
>>>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>>> writing one large file sequentially, and that's when I stumbled upon the
>>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>>> opened up some larger free extents and resolved the problem, but only
>>>>>> until those get filled up and I'm back to fragmented tiny extents. So
>>>>>> effectively I'm stuck at the current utilization, as trying to fill them
>>>>>> up any more just slows down to an absolute crawl.
>>>>>>
>>>>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>>>> situation, but obviously this is going to take forever.
>>>>>>
>>>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>>>
>>>>>> - Hector
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>> - Hector
>>
- Hector
Hi Hector, Not related to fragmentation. But I see you mentioned CephFS, and your OSDs are at high utilization. Is your pool NEAR FULL? CephFS write performance is severely degraded if the pool is NEAR FULL. Buffered write will be disabled, and every single write() system call needs to wait for reply from OSD. If this is the case, use “ceph osd set-nearfull-ratio” to get normal performance. Weiwen Hu
在 2023年5月24日，20:19，Hector Martin <marcan@marcan.st> 写道：
Hi,
I've been seeing relatively large fragmentation numbers on all my OSDs:
ceph daemon osd.13 bluestore allocator score block { "fragmentation_rating": 0.77251526920454427 }
These aren't that old, as I recreated them all around July last year. They mostly hold CephFS data with erasure coding, with a mix of large and small files. The OSDs are at around 80%-85% utilization right now. Most of the data was written sequentially when the OSDs were created (I rsynced everything from a remote backup). Since then more data has been added, but not particularly quickly.
At some point I noticed pathologically slow writes, and I couldn't figure out what was wrong. Eventually I did some block tracing and noticed the I/Os were very small, even though CephFS-side I was just writing one large file sequentially, and that's when I stumbled upon the free space fragmentation problem. Indeed, deleting some large files opened up some larger free extents and resolved the problem, but only until those get filled up and I'm back to fragmented tiny extents. So effectively I'm stuck at the current utilization, as trying to fill them up any more just slows down to an absolute crawl.
I'm adding a few more OSDs and plan on doing the dance of removing one OSD at a time and replacing it with another one to hopefully improve the situation, but obviously this is going to take forever.
Is there any plan for offering a defrag tool of some sort for bluestore?
- Hector _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 25/05/2023 01.40, 胡 玮文 wrote:
Hi Hector,
Not related to fragmentation. But I see you mentioned CephFS, and your OSDs are at high utilization. Is your pool NEAR FULL? CephFS write performance is severely degraded if the pool is NEAR FULL. Buffered write will be disabled, and every single write() system call needs to wait for reply from OSD.
If this is the case, use “ceph osd set-nearfull-ratio” to get normal performance.
I learned about this after the issue; they did become nearfull at one point and I changed the threshold, but I don't think this explains the behavior I was seeing because I was trying to do bulk writes (which should use very large write sizes even without buffering). What happened was usually a single OSD would immediately go to 100% utilization, but not the rest, which is what I'd expect if that one OSD was the one with the most fragmented free space ending up pathologically slowing down writes. - Hector
participants (7)
-
Fox, Kevin M
-
Hector Martin
-
Igor Fedotov
-
Konstantin Shalygin
-
Mark Nelson
-
Stefan Kooman
-
胡 玮文