24 May
2023
24 May
'23
7:17 p.m.
Hi,
I've been seeing relatively large fragmentation numbers on all my OSDs:
ceph daemon osd.13 bluestore allocator score block
{
"fragmentation_rating": 0.77251526920454427
}
These aren't that old, as I recreated them all around July last year.
They mostly hold CephFS data with erasure coding, with a mix of large
and small files. The OSDs are at around 80%-85% utilization right now.
Most of the data was written sequentially when the OSDs were created (I
rsynced everything from a remote backup). Since then more data has been
added, but not particularly quickly.
At some point I noticed pathologically slow writes, and I couldn't
figure out what was wrong. Eventually I did some block tracing and
noticed the I/Os were very small, even though CephFS-side I was just
writing one large file sequentially, and that's when I stumbled upon the
free space fragmentation problem. Indeed, deleting some large files
opened up some larger free extents and resolved the problem, but only
until those get filled up and I'm back to fragmented tiny extents. So
effectively I'm stuck at the current utilization, as trying to fill them
up any more just slows down to an absolute crawl.
I'm adding a few more OSDs and plan on doing the dance of removing one
OSD at a time and replacing it with another one to hopefully improve the
situation, but obviously this is going to take forever.
Is there any plan for offering a defrag tool of some sort for bluestore?
- Hector
24 May
24 May
8:07 p.m.
Yep, bluestore fragmentation is an issue. It's sort of a natural result
of using copy-on-write and never implementing any kind of
defragmentation scheme. Adam and I have been talking about doing it
now, probably piggybacking on scrub or other operations that already
area reading all of the extents for an object anyway.
I wrote a very simply prototype for clone to speed up the rbd mirror use
case here:
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
Adam ended up going the extra mile and completely changed how shared
blobs works which probably eliminates the need to do defrag on clone
anymore from an rbd-mirror perspective, but I think we still need to
identify any times we are doing full object reads of fragmented objects
and consider defragmenting at that time. It might be clone, or scrub,
or other things, but the point is that if we are already doing most of
the work (seeks on HDD especially!) the extra cost of a large write to
clean it up isn't that bad, especially if we are doing it over the
course of months or years and can help keep freespace less fragmented.
Mark
On 5/24/23 07:17, Hector Martin wrote:
Hi,
I've been seeing relatively large fragmentation numbers on all my OSDs:
ceph daemon osd.13 bluestore allocator score block
{
"fragmentation_rating": 0.77251526920454427
}
These aren't that old, as I recreated them all around July last year.
They mostly hold CephFS data with erasure coding, with a mix of large
and small files. The OSDs are at around 80%-85% utilization right now.
Most of the data was written sequentially when the OSDs were created (I
rsynced everything from a remote backup). Since then more data has been
added, but not particularly quickly.
At some point I noticed pathologically slow writes, and I couldn't
figure out what was wrong. Eventually I did some block tracing and
noticed the I/Os were very small, even though CephFS-side I was just
writing one large file sequentially, and that's when I stumbled upon the
free space fragmentation problem. Indeed, deleting some large files
opened up some larger free extents and resolved the problem, but only
until those get filled up and I'm back to fragmented tiny extents. So
effectively I'm stuck at the current utilization, as trying to fill them
up any more just slows down to an absolute crawl.
I'm adding a few more OSDs and plan on doing the dance of removing one
OSD at a time and replacing it with another one to hopefully improve the
situation, but obviously this is going to take forever.
Is there any plan for offering a defrag tool of some sort for bluestore?
- Hector
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Mark Nelson
Head of R&D (USA)
Clyso GmbH
p: +49 89 21552391 12
a: Loristraße 8 | 80335 München | Germany
w: https://clyso.com | e: mark.nelson(a)clyso.com
We are hiring: https://www.clyso.com/jobs/
9:18 p.m.
On 24/05/2023 22.07, Mark Nelson wrote:
Yep, bluestore fragmentation is an issue. It's
sort of a natural result
of using copy-on-write and never implementing any kind of
defragmentation scheme. Adam and I have been talking about doing it
now, probably piggybacking on scrub or other operations that already
area reading all of the extents for an object anyway.
I wrote a very simply prototype for clone to speed up the rbd mirror use
case here:
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
Adam ended up going the extra mile and completely changed how shared
blobs works which probably eliminates the need to do defrag on clone
anymore from an rbd-mirror perspective, but I think we still need to
identify any times we are doing full object reads of fragmented objects
and consider defragmenting at that time. It might be clone, or scrub,
or other things, but the point is that if we are already doing most of
the work (seeks on HDD especially!) the extra cost of a large write to
clean it up isn't that bad, especially if we are doing it over the
course of months or years and can help keep freespace less fragmented.
Note that my particular issue seemed to specifically be free space
fragmentation. I don't use RBD mirror and I would not *expect* most of
my cephfs use cases to lead to any weird cow/fragmentation issues with
objects other than those forced by the free space becoming fragmented
(unless there is some weird pathological use case I'm hitting). Most of
my write workloads are just copying files in bulk and incrementally
writing out files.
Would simply defragging objects during scrub/etc help with free space
fragmentation itself? Those seem like two somewhat unrelated issues...
note that if free space is already fragmented, you wouldn't even have a
place to put down a defragmented object.
Are there any stats I can look at to figure out how bad object and free
space fragmentation is? It would be nice to have some clearer data
beyond my hunch/deduction after seeing the I/O patterns and the sole
fragmentation number :). Also would be interesting to get some kind of
trace of the bluestore ops the OSD is doing, so I can find out whether
it's doing something pathological that causes more fragmentation for
some reason.
Mark
On 5/24/23 07:17, Hector Martin wrote:
- Hector
Hi,
I've been seeing relatively large fragmentation numbers on all my OSDs:
ceph daemon osd.13 bluestore allocator score block
{
"fragmentation_rating": 0.77251526920454427
}
These aren't that old, as I recreated them all around July last year.
They mostly hold CephFS data with erasure coding, with a mix of large
and small files. The OSDs are at around 80%-85% utilization right now.
Most of the data was written sequentially when the OSDs were created (I
rsynced everything from a remote backup). Since then more data has been
added, but not particularly quickly.
At some point I noticed pathologically slow writes, and I couldn't
figure out what was wrong. Eventually I did some block tracing and
noticed the I/Os were very small, even though CephFS-side I was just
writing one large file sequentially, and that's when I stumbled upon the
free space fragmentation problem. Indeed, deleting some large files
opened up some larger free extents and resolved the problem, but only
until those get filled up and I'm back to fragmented tiny extents. So
effectively I'm stuck at the current utilization, as trying to fill them
up any more just slows down to an absolute crawl.
I'm adding a few more OSDs and plan on doing the dance of removing one
OSD at a time and replacing it with another one to hopefully improve the
situation, but obviously this is going to take forever.
Is there any plan for offering a defrag tool of some sort for bluestore?
- Hector
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 May
25 May
9:11 p.m.
On 5/24/23 09:18, Hector Martin wrote:
On 24/05/2023 22.07, Mark Nelson wrote:
That is indeed one of the big issues. If you've already fragmented
freespace, it becomes much harder. The approach I've been advocating
for this is that when we scrub and encounter a heavily fragmented
object, we do a quick search to see if we can easily find contiguous
freespace for the whole object and if we can, we do that. If we can't,
we look and see if any of extents have space next to them that would
allow an adjacent range to fit in that would allow us to improve the
amount of contiguous space used (ie eliminating extents is a win,
splitting an extent to add to another extent is a judgement call). The
idea here is that we'd try to repair the holes that get left behind when
we do COW lazily after the fact. When we punch a hole we'd mark it as a
range that a new extent might easily fit back into to make things
contiguous again.
Long term I want to restructure how bluestore works. I think we should
write overwrite extents to the fast device unless they are quite large
and once we have space pressure on the fast device we write large blobs
or perhaps whole objects to the slow device and try to keep everything
neatly aligned (perhaps segment the disk for different object sizes, or
even put some small objects on the fast device). We take out the
current blob level compression, leave small extents uncompressed on the
fast device, and only compress when we write large extents or whole
objects to the slow device. The goal would be to keep fragmentation on
the slow device low, improve behavior on HDD and QLC flash, and make
better use of the DB/WAL devices that people put in their systems.
Mark
Yep, bluestore fragmentation is an issue.
It's sort of a natural result
of using copy-on-write and never implementing any kind of
defragmentation scheme. Adam and I have been talking about doing it
now, probably piggybacking on scrub or other operations that already
area reading all of the extents for an object anyway.
I wrote a very simply prototype for clone to speed up the rbd mirror use
case here:
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
Adam ended up going the extra mile and completely changed how shared
blobs works which probably eliminates the need to do defrag on clone
anymore from an rbd-mirror perspective, but I think we still need to
identify any times we are doing full object reads of fragmented objects
and consider defragmenting at that time. It might be clone, or scrub,
or other things, but the point is that if we are already doing most of
the work (seeks on HDD especially!) the extra cost of a large write to
clean it up isn't that bad, especially if we are doing it over the
course of months or years and can help keep freespace less fragmented.
Note that
my particular issue seemed to specifically be free space
fragmentation. I don't use RBD mirror and I would not *expect* most of
my cephfs use cases to lead to any weird cow/fragmentation issues with
objects other than those forced by the free space becoming fragmented
(unless there is some weird pathological use case I'm hitting). Most of
my write workloads are just copying files in bulk and incrementally
writing out files.
Would simply defragging objects during scrub/etc help with free space
fragmentation itself? Those seem like two somewhat unrelated issues...
note that if free space is already fragmented, you wouldn't even have a
place to put down a defragmented object.
Are there any stats I can look at to figure out how bad object and free
space fragmentation is? It would be nice to have some clearer data
beyond my hunch/deduction after seeing the I/O patterns and the sole
fragmentation number :). Also would be interesting to get some kind of
trace of the bluestore ops the OSD is doing, so I can find out whether
it's doing something pathological that causes more fragmentation for
some reason.
--
Best Regards,
Mark Nelson
Head of R&D (USA)
Clyso GmbH
p: +49 89 21552391 12
a: Loristraße 8 | 80335 München | Germany
w: https://clyso.com | e: mark.nelson(a)clyso.com
We are hiring: https://www.clyso.com/jobs/
Mark
On 5/24/23 07:17, Hector Martin wrote:
> Hi,
>
> I've been seeing relatively large fragmentation numbers on all my OSDs:
>
> ceph daemon osd.13 bluestore allocator score block
> {
> "fragmentation_rating": 0.77251526920454427
> }
>
> These aren't that old, as I recreated them all around July last year.
> They mostly hold CephFS data with erasure coding, with a mix of large
> and small files. The OSDs are at around 80%-85% utilization right now.
> Most of the data was written sequentially when the OSDs were created (I
> rsynced everything from a remote backup). Since then more data has been
> added, but not particularly quickly.
>
> At some point I noticed pathologically slow writes, and I couldn't
> figure out what was wrong. Eventually I did some block tracing and
> noticed the I/Os were very small, even though CephFS-side I was just
> writing one large file sequentially, and that's when I stumbled upon the
> free space fragmentation problem. Indeed, deleting some large files
> opened up some larger free extents and resolved the problem, but only
> until those get filled up and I'm back to fragmented tiny extents. So
> effectively I'm stuck at the current utilization, as trying to fill them
> up any more just slows down to an absolute crawl.
>
> I'm adding a few more OSDs and plan on doing the dance of removing one
> OSD at a time and replacing it with another one to hopefully improve the
> situation, but obviously this is going to take forever.
>
> Is there any plan for offering a defrag tool of some sort for bluestore?
>
> - Hector
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
- Hector
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
10:29 p.m.
Hi Hector,
I can advise two tools for further fragmentation analysis:
1) One might want to use ceph-bluestore-tool's free-dump command to get
a list of free chunks for an OSD and try to analyze whether it's really
highly fragmented and lacks long enough extents. free-dump just returns
a list of extents in json format, I can take a look to the output if
shared...
2) You might want to look for allocation probs in OSD logs and see how
fragmentation in allocated chunks has evolved.
E.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
probe -1: 35168547, 46401246, 1199516209152
probe -3: 27275094, 35681802, 200121712640
probe -5: 34847167, 52539758, 271272230912
probe -9: 44291522, 60025613, 523997483008
probe -17: 10646313, 10646313, 155178434560
The first probe refers to the last day while others match days (or
rather probes) -1, -3, -5, -9, -17
'cnt' column represents the amount of allocations performed in the
previous 24 hours and 'frags' one shows amount of fragments in the
resulted allocations. So significant mismatch between frags and cnt
might indicate some issues with high fragmentation indeed.
Apart from retrospective analysis you might also want how OSD behavior
changes after reboot - e.g. wouldn't rebooted OSD produce less
fragmentation... Which in turn might indicate some issues with BlueStore
allocator..
Just FYI: allocation probe printing interval is controlled by
bluestore_alloc_stats_dump_interval parameter.
Thanks,
Igor
On 24/05/2023 17:18, Hector Martin wrote:
> On 24/05/2023 22.07, Mark Nelson wrote:
>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>> of using copy-on-write and never implementing any kind of
>> defragmentation scheme. Adam and I have been talking about doing it
>> now, probably piggybacking on scrub or other operations that already
>> area reading all of the extents for an object anyway.
>>
>>
>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>> case here:
>>
>> https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>
>>
>> Adam ended up going the extra mile and completely changed how shared
>> blobs works which probably eliminates the need to do defrag on clone
>> anymore from an rbd-mirror perspective, but I think we still need to
>> identify any times we are doing full object reads of fragmented objects
>> and consider defragmenting at that time. It might be clone, or scrub,
>> or other things, but the point is that if we are already doing most of
>> the work (seeks on HDD especially!) the extra cost of a large write to
>> clean it up isn't that bad, especially if we are doing it over the
>> course of months or years and can help keep freespace less fragmented.
> Note that my particular issue seemed to specifically be free space
> fragmentation. I don't use RBD mirror and I would not *expect* most of
> my cephfs use cases to lead to any weird cow/fragmentation issues with
> objects other than those forced by the free space becoming fragmented
> (unless there is some weird pathological use case I'm hitting). Most of
> my write workloads are just copying files in bulk and incrementally
> writing out files.
>
> Would simply defragging objects during scrub/etc help with free space
> fragmentation itself? Those seem like two somewhat unrelated issues...
> note that if free space is already fragmented, you wouldn't even have a
> place to put down a defragmented object.
>
> Are there any stats I can look at to figure out how bad object and free
> space fragmentation is? It would be nice to have some clearer data
> beyond my hunch/deduction after seeing the I/O patterns and the sole
> fragmentation number :). Also would be interesting to get some kind of
> trace of the bluestore ops the OSD is doing, so I can find out whether
> it's doing something pathological that causes more fragmentation for
> some reason.
>
>> Mark
>>
>>
>> On 5/24/23 07:17, Hector Martin wrote:
>>> Hi,
>>>
>>> I've been seeing relatively large fragmentation numbers on all my OSDs:
>>>
>>> ceph daemon osd.13 bluestore allocator score block
>>> {
>>> "fragmentation_rating": 0.77251526920454427
>>> }
>>>
>>> These aren't that old, as I recreated them all around July last year.
>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>> Most of the data was written sequentially when the OSDs were created (I
>>> rsynced everything from a remote backup). Since then more data has been
>>> added, but not particularly quickly.
>>>
>>> At some point I noticed pathologically slow writes, and I couldn't
>>> figure out what was wrong. Eventually I did some block tracing and
>>> noticed the I/Os were very small, even though CephFS-side I was just
>>> writing one large file sequentially, and that's when I stumbled upon the
>>> free space fragmentation problem. Indeed, deleting some large files
>>> opened up some larger free extents and resolved the problem, but only
>>> until those get filled up and I'm back to fragmented tiny extents. So
>>> effectively I'm stuck at the current utilization, as trying to fill them
>>> up any more just slows down to an absolute crawl.
>>>
>>> I'm adding a few more OSDs and plan on doing the dance of removing one
>>> OSD at a time and replacing it with another one to hopefully improve the
>>> situation, but obviously this is going to take forever.
>>>
>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> - Hector
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:41 p.m.
Is this related to https://tracker.ceph.com/issues/58022 ?
We still see run away osds at times, somewhat randomly, that causes runaway fragmentation
issues.
Thanks,
Kevin
________________________________________
From: Igor Fedotov <igor.fedotov(a)croit.io>
Sent: Thursday, May 25, 2023 8:29 AM
To: Hector Martin; ceph-users(a)ceph.io
Subject: [ceph-users] Re: BlueStore fragmentation woes
Check twice before you click! This email originated from outside PNNL.
Hi Hector,
I can advise two tools for further fragmentation analysis:
1) One might want to use ceph-bluestore-tool's free-dump command to get
a list of free chunks for an OSD and try to analyze whether it's really
highly fragmented and lacks long enough extents. free-dump just returns
a list of extents in json format, I can take a look to the output if
shared...
2) You might want to look for allocation probs in OSD logs and see how
fragmentation in allocated chunks has evolved.
E.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
probe -1: 35168547, 46401246, 1199516209152
probe -3: 27275094, 35681802, 200121712640
probe -5: 34847167, 52539758, 271272230912
probe -9: 44291522, 60025613, 523997483008
probe -17: 10646313, 10646313, 155178434560
The first probe refers to the last day while others match days (or
rather probes) -1, -3, -5, -9, -17
'cnt' column represents the amount of allocations performed in the
previous 24 hours and 'frags' one shows amount of fragments in the
resulted allocations. So significant mismatch between frags and cnt
might indicate some issues with high fragmentation indeed.
Apart from retrospective analysis you might also want how OSD behavior
changes after reboot - e.g. wouldn't rebooted OSD produce less
fragmentation... Which in turn might indicate some issues with BlueStore
allocator..
Just FYI: allocation probe printing interval is controlled by
bluestore_alloc_stats_dump_interval parameter.
Thanks,
Igor
On 24/05/2023 17:18, Hector Martin wrote:
On 24/05/2023 22.07, Mark Nelson wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Yep, bluestore fragmentation is an issue.
It's sort of a natural result
of using copy-on-write and never implementing any kind of
defragmentation scheme. Adam and I have been talking about doing it
now, probably piggybacking on scrub or other operations that already
area reading all of the extents for an object anyway.
I wrote a very simply prototype for clone to speed up the rbd mirror use
case here:
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
Adam ended up going the extra mile and completely changed how shared
blobs works which probably eliminates the need to do defrag on clone
anymore from an rbd-mirror perspective, but I think we still need to
identify any times we are doing full object reads of fragmented objects
and consider defragmenting at that time. It might be clone, or scrub,
or other things, but the point is that if we are already doing most of
the work (seeks on HDD especially!) the extra cost of a large write to
clean it up isn't that bad, especially if we are doing it over the
course of months or years and can help keep freespace less fragmented.
Note that
my particular issue seemed to specifically be free space
fragmentation. I don't use RBD mirror and I would not *expect* most of
my cephfs use cases to lead to any weird cow/fragmentation issues with
objects other than those forced by the free space becoming fragmented
(unless there is some weird pathological use case I'm hitting). Most of
my write workloads are just copying files in bulk and incrementally
writing out files.
Would simply defragging objects during scrub/etc help with free space
fragmentation itself? Those seem like two somewhat unrelated issues...
note that if free space is already fragmented, you wouldn't even have a
place to put down a defragmented object.
Are there any stats I can look at to figure out how bad object and free
space fragmentation is? It would be nice to have some clearer data
beyond my hunch/deduction after seeing the I/O patterns and the sole
fragmentation number :). Also would be interesting to get some kind of
trace of the bluestore ops the OSD is doing, so I can find out whether
it's doing something pathological that causes more fragmentation for
some reason.
Mark
On 5/24/23 07:17, Hector Martin wrote:
> Hi,
>
> I've been seeing relatively large fragmentation numbers on all my OSDs:
>
> ceph daemon osd.13 bluestore allocator score block
> {
> "fragmentation_rating": 0.77251526920454427
> }
>
> These aren't that old, as I recreated them all around July last year.
> They mostly hold CephFS data with erasure coding, with a mix of large
> and small files. The OSDs are at around 80%-85% utilization right now.
> Most of the data was written sequentially when the OSDs were created (I
> rsynced everything from a remote backup). Since then more data has been
> added, but not particularly quickly.
>
> At some point I noticed pathologically slow writes, and I couldn't
> figure out what was wrong. Eventually I did some block tracing and
> noticed the I/Os were very small, even though CephFS-side I was just
> writing one large file sequentially, and that's when I stumbled upon the
> free space fragmentation problem. Indeed, deleting some large files
> opened up some larger free extents and resolved the problem, but only
> until those get filled up and I'm back to fragmented tiny extents. So
> effectively I'm stuck at the current utilization, as trying to fill them
> up any more just slows down to an absolute crawl.
>
> I'm adding a few more OSDs and plan on doing the dance of removing one
> OSD at a time and replacing it with another one to hopefully improve the
> situation, but obviously this is going to take forever.
>
> Is there any plan for offering a defrag tool of some sort for bluestore?
>
> - Hector
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
- Hector
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:17 p.m.
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO
it's mostly like a very imprecise first turn marker to alert that
something might be wrong. But not a real quantitative high-quality estimate.
So in fact I'd like to see a series of allocation probes showing
eventual degradation without OSD restart and immediate severe
improvement after the restart.
Can you try to collect something like that? Would the same behavior
persist with an alternative allocator?
Thanks,
Igor
On 25/05/2023 18:41, Fox, Kevin M wrote:
> Is this related to https://tracker.ceph.com/issues/58022 ?
>
> We still see run away osds at times, somewhat randomly, that causes runaway
fragmentation issues.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov(a)croit.io>
> Sent: Thursday, May 25, 2023 8:29 AM
> To: Hector Martin; ceph-users(a)ceph.io
> Subject: [ceph-users] Re: BlueStore fragmentation woes
>
> Check twice before you click! This email originated from outside PNNL.
>
>
> Hi Hector,
>
> I can advise two tools for further fragmentation analysis:
>
> 1) One might want to use ceph-bluestore-tool's free-dump command to get
> a list of free chunks for an OSD and try to analyze whether it's really
> highly fragmented and lacks long enough extents. free-dump just returns
> a list of extents in json format, I can take a look to the output if
> shared...
>
> 2) You might want to look for allocation probs in OSD logs and see how
> fragmentation in allocated chunks has evolved.
>
> E.g.
>
> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
> probe -1: 35168547, 46401246, 1199516209152
> probe -3: 27275094, 35681802, 200121712640
> probe -5: 34847167, 52539758, 271272230912
> probe -9: 44291522, 60025613, 523997483008
> probe -17: 10646313, 10646313, 155178434560
>
> The first probe refers to the last day while others match days (or
> rather probes) -1, -3, -5, -9, -17
>
> 'cnt' column represents the amount of allocations performed in the
> previous 24 hours and 'frags' one shows amount of fragments in the
> resulted allocations. So significant mismatch between frags and cnt
> might indicate some issues with high fragmentation indeed.
>
> Apart from retrospective analysis you might also want how OSD behavior
> changes after reboot - e.g. wouldn't rebooted OSD produce less
> fragmentation... Which in turn might indicate some issues with BlueStore
> allocator..
>
> Just FYI: allocation probe printing interval is controlled by
> bluestore_alloc_stats_dump_interval parameter.
>
>
> Thanks,
>
> Igor
>
>
>
> On 24/05/2023 17:18, Hector Martin wrote:
>> On 24/05/2023 22.07, Mark Nelson wrote:
>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>> of using copy-on-write and never implementing any kind of
>>> defragmentation scheme. Adam and I have been talking about doing it
>>> now, probably piggybacking on scrub or other operations that already
>>> area reading all of the extents for an object anyway.
>>>
>>>
>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>> case here:
>>>
>>>
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>>
>>>
>>> Adam ended up going the extra mile and completely changed how shared
>>> blobs works which probably eliminates the need to do defrag on clone
>>> anymore from an rbd-mirror perspective, but I think we still need to
>>> identify any times we are doing full object reads of fragmented objects
>>> and consider defragmenting at that time. It might be clone, or scrub,
>>> or other things, but the point is that if we are already doing most of
>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>> clean it up isn't that bad, especially if we are doing it over the
>>> course of months or years and can help keep freespace less fragmented.
>> Note that my particular issue seemed to specifically be free space
>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>> objects other than those forced by the free space becoming fragmented
>> (unless there is some weird pathological use case I'm hitting). Most of
>> my write workloads are just copying files in bulk and incrementally
>> writing out files.
>>
>> Would simply defragging objects during scrub/etc help with free space
>> fragmentation itself? Those seem like two somewhat unrelated issues...
>> note that if free space is already fragmented, you wouldn't even have a
>> place to put down a defragmented object.
>>
>> Are there any stats I can look at to figure out how bad object and free
>> space fragmentation is? It would be nice to have some clearer data
>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>> fragmentation number :). Also would be interesting to get some kind of
>> trace of the bluestore ops the OSD is doing, so I can find out whether
>> it's doing something pathological that causes more fragmentation for
>> some reason.
>>
>>> Mark
>>>
>>>
>>> On 5/24/23 07:17, Hector Martin wrote:
>>>> Hi,
>>>>
>>>> I've been seeing relatively large fragmentation numbers on all my
OSDs:
>>>>
>>>> ceph daemon osd.13 bluestore allocator score block
>>>> {
>>>> "fragmentation_rating": 0.77251526920454427
>>>> }
>>>>
>>>> These aren't that old, as I recreated them all around July last
year.
>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>> Most of the data was written sequentially when the OSDs were created (I
>>>> rsynced everything from a remote backup). Since then more data has been
>>>> added, but not particularly quickly.
>>>>
>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>> figure out what was wrong. Eventually I did some block tracing and
>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>> writing one large file sequentially, and that's when I stumbled upon
the
>>>> free space fragmentation problem. Indeed, deleting some large files
>>>> opened up some larger free extents and resolved the problem, but only
>>>> until those get filled up and I'm back to fragmented tiny extents.
So
>>>> effectively I'm stuck at the current utilization, as trying to fill
them
>>>> up any more just slows down to an absolute crawl.
>>>>
>>>> I'm adding a few more OSDs and plan on doing the dance of removing
one
>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>> situation, but obviously this is going to take forever.
>>>>
>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> - Hector
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:20 p.m.
If you can give me instructions on what you want me to gather before the restart and after
restart I can do it. I have some running away right now.
Thanks,
Kevin
________________________________________
From: Igor Fedotov <igor.fedotov(a)croit.io>
Sent: Thursday, May 25, 2023 9:17 AM
To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO
it's mostly like a very imprecise first turn marker to alert that
something might be wrong. But not a real quantitative high-quality estimate.
So in fact I'd like to see a series of allocation probes showing
eventual degradation without OSD restart and immediate severe
improvement after the restart.
Can you try to collect something like that? Would the same behavior
persist with an alternative allocator?
Thanks,
Igor
On 25/05/2023 18:41, Fox, Kevin M wrote:
> Is this related to https://tracker.ceph.com/issues/58022 ?
>
> We still see run away osds at times, somewhat randomly, that causes runaway
fragmentation issues.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov(a)croit.io>
> Sent: Thursday, May 25, 2023 8:29 AM
> To: Hector Martin; ceph-users(a)ceph.io
> Subject: [ceph-users] Re: BlueStore fragmentation woes
>
> Check twice before you click! This email originated from outside PNNL.
>
>
> Hi Hector,
>
> I can advise two tools for further fragmentation analysis:
>
> 1) One might want to use ceph-bluestore-tool's free-dump command to get
> a list of free chunks for an OSD and try to analyze whether it's really
> highly fragmented and lacks long enough extents. free-dump just returns
> a list of extents in json format, I can take a look to the output if
> shared...
>
> 2) You might want to look for allocation probs in OSD logs and see how
> fragmentation in allocated chunks has evolved.
>
> E.g.
>
> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
> probe -1: 35168547, 46401246, 1199516209152
> probe -3: 27275094, 35681802, 200121712640
> probe -5: 34847167, 52539758, 271272230912
> probe -9: 44291522, 60025613, 523997483008
> probe -17: 10646313, 10646313, 155178434560
>
> The first probe refers to the last day while others match days (or
> rather probes) -1, -3, -5, -9, -17
>
> 'cnt' column represents the amount of allocations performed in the
> previous 24 hours and 'frags' one shows amount of fragments in the
> resulted allocations. So significant mismatch between frags and cnt
> might indicate some issues with high fragmentation indeed.
>
> Apart from retrospective analysis you might also want how OSD behavior
> changes after reboot - e.g. wouldn't rebooted OSD produce less
> fragmentation... Which in turn might indicate some issues with BlueStore
> allocator..
>
> Just FYI: allocation probe printing interval is controlled by
> bluestore_alloc_stats_dump_interval parameter.
>
>
> Thanks,
>
> Igor
>
>
>
> On 24/05/2023 17:18, Hector Martin wrote:
>> On 24/05/2023 22.07, Mark Nelson wrote:
>>> Yep, bluestore fragmentation is an issue. It's sort of a natural result
>>> of using copy-on-write and never implementing any kind of
>>> defragmentation scheme. Adam and I have been talking about doing it
>>> now, probably piggybacking on scrub or other operations that already
>>> area reading all of the extents for an object anyway.
>>>
>>>
>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>> case here:
>>>
>>>
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>>
>>>
>>> Adam ended up going the extra mile and completely changed how shared
>>> blobs works which probably eliminates the need to do defrag on clone
>>> anymore from an rbd-mirror perspective, but I think we still need to
>>> identify any times we are doing full object reads of fragmented objects
>>> and consider defragmenting at that time. It might be clone, or scrub,
>>> or other things, but the point is that if we are already doing most of
>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>> clean it up isn't that bad, especially if we are doing it over the
>>> course of months or years and can help keep freespace less fragmented.
>> Note that my particular issue seemed to specifically be free space
>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>> objects other than those forced by the free space becoming fragmented
>> (unless there is some weird pathological use case I'm hitting). Most of
>> my write workloads are just copying files in bulk and incrementally
>> writing out files.
>>
>> Would simply defragging objects during scrub/etc help with free space
>> fragmentation itself? Those seem like two somewhat unrelated issues...
>> note that if free space is already fragmented, you wouldn't even have a
>> place to put down a defragmented object.
>>
>> Are there any stats I can look at to figure out how bad object and free
>> space fragmentation is? It would be nice to have some clearer data
>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>> fragmentation number :). Also would be interesting to get some kind of
>> trace of the bluestore ops the OSD is doing, so I can find out whether
>> it's doing something pathological that causes more fragmentation for
>> some reason.
>>
>>> Mark
>>>
>>>
>>> On 5/24/23 07:17, Hector Martin wrote:
>>>> Hi,
>>>>
>>>> I've been seeing relatively large fragmentation numbers on all my
OSDs:
>>>>
>>>> ceph daemon osd.13 bluestore allocator score block
>>>> {
>>>> "fragmentation_rating": 0.77251526920454427
>>>> }
>>>>
>>>> These aren't that old, as I recreated them all around July last
year.
>>>> They mostly hold CephFS data with erasure coding, with a mix of large
>>>> and small files. The OSDs are at around 80%-85% utilization right now.
>>>> Most of the data was written sequentially when the OSDs were created (I
>>>> rsynced everything from a remote backup). Since then more data has been
>>>> added, but not particularly quickly.
>>>>
>>>> At some point I noticed pathologically slow writes, and I couldn't
>>>> figure out what was wrong. Eventually I did some block tracing and
>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>> writing one large file sequentially, and that's when I stumbled upon
the
>>>> free space fragmentation problem. Indeed, deleting some large files
>>>> opened up some larger free extents and resolved the problem, but only
>>>> until those get filled up and I'm back to fragmented tiny extents.
So
>>>> effectively I'm stuck at the current utilization, as trying to fill
them
>>>> up any more just slows down to an absolute crawl.
>>>>
>>>> I'm adding a few more OSDs and plan on doing the dance of removing
one
>>>> OSD at a time and replacing it with another one to hopefully improve the
>>>> situation, but obviously this is going to take forever.
>>>>
>>>> Is there any plan for offering a defrag tool of some sort for bluestore?
>>>>
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> - Hector
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:29 p.m.
Just run through available logs for a specific OSD (which you suspect
suffer from high fragmentation) and collect all allocation stats probes
you can find ("allocation stats probe" string is a perfect grep pattern,
please append lines with historic probes following day-0 line as well.
Given this is printed once per day there wouldn't be too many).
Then do OSD restart and wait a couple more days. Would allocation stats
show much better disparity between cnt and frags columns?
Is the similar pattern (eventual degradation in stats prior to restart
and severe improvement afterwards) be observed for other OSDs?
On 25/05/2023 19:20, Fox, Kevin M wrote:
> If you can give me instructions on what you want me to gather before the restart and
after restart I can do it. I have some running away right now.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov(a)croit.io>
> Sent: Thursday, May 25, 2023 9:17 AM
> To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Perhaps...
>
> I don't like the idea to use fragmentation score as a real index. IMO
> it's mostly like a very imprecise first turn marker to alert that
> something might be wrong. But not a real quantitative high-quality estimate.
>
> So in fact I'd like to see a series of allocation probes showing
> eventual degradation without OSD restart and immediate severe
> improvement after the restart.
>
> Can you try to collect something like that? Would the same behavior
> persist with an alternative allocator?
>
>
> Thanks,
>
> Igor
>
>
> On 25/05/2023 18:41, Fox, Kevin M wrote:
>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>
>> We still see run away osds at times, somewhat randomly, that causes runaway
fragmentation issues.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov(a)croit.io>
>> Sent: Thursday, May 25, 2023 8:29 AM
>> To: Hector Martin; ceph-users(a)ceph.io
>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Check twice before you click! This email originated from outside PNNL.
>>
>>
>> Hi Hector,
>>
>> I can advise two tools for further fragmentation analysis:
>>
>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>> a list of free chunks for an OSD and try to analyze whether it's really
>> highly fragmented and lacks long enough extents. free-dump just returns
>> a list of extents in json format, I can take a look to the output if
>> shared...
>>
>> 2) You might want to look for allocation probs in OSD logs and see how
>> fragmentation in allocated chunks has evolved.
>>
>> E.g.
>>
>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>> probe -1: 35168547, 46401246, 1199516209152
>> probe -3: 27275094, 35681802, 200121712640
>> probe -5: 34847167, 52539758, 271272230912
>> probe -9: 44291522, 60025613, 523997483008
>> probe -17: 10646313, 10646313, 155178434560
>>
>> The first probe refers to the last day while others match days (or
>> rather probes) -1, -3, -5, -9, -17
>>
>> 'cnt' column represents the amount of allocations performed in the
>> previous 24 hours and 'frags' one shows amount of fragments in the
>> resulted allocations. So significant mismatch between frags and cnt
>> might indicate some issues with high fragmentation indeed.
>>
>> Apart from retrospective analysis you might also want how OSD behavior
>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>> fragmentation... Which in turn might indicate some issues with BlueStore
>> allocator..
>>
>> Just FYI: allocation probe printing interval is controlled by
>> bluestore_alloc_stats_dump_interval parameter.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>> On 24/05/2023 17:18, Hector Martin wrote:
>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural
result
>>>> of using copy-on-write and never implementing any kind of
>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>> now, probably piggybacking on scrub or other operations that already
>>>> area reading all of the extents for an object anyway.
>>>>
>>>>
>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>> case here:
>>>>
>>>>
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>>>
>>>>
>>>> Adam ended up going the extra mile and completely changed how shared
>>>> blobs works which probably eliminates the need to do defrag on clone
>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>> identify any times we are doing full object reads of fragmented objects
>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>> or other things, but the point is that if we are already doing most of
>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>> clean it up isn't that bad, especially if we are doing it over the
>>>> course of months or years and can help keep freespace less fragmented.
>>> Note that my particular issue seemed to specifically be free space
>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>> objects other than those forced by the free space becoming fragmented
>>> (unless there is some weird pathological use case I'm hitting). Most of
>>> my write workloads are just copying files in bulk and incrementally
>>> writing out files.
>>>
>>> Would simply defragging objects during scrub/etc help with free space
>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>> note that if free space is already fragmented, you wouldn't even have a
>>> place to put down a defragmented object.
>>>
>>> Are there any stats I can look at to figure out how bad object and free
>>> space fragmentation is? It would be nice to have some clearer data
>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>> fragmentation number :). Also would be interesting to get some kind of
>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>> it's doing something pathological that causes more fragmentation for
>>> some reason.
>>>
>>>> Mark
>>>>
>>>>
>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>> Hi,
>>>>>
>>>>> I've been seeing relatively large fragmentation numbers on all my
OSDs:
>>>>>
>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>> {
>>>>> "fragmentation_rating": 0.77251526920454427
>>>>> }
>>>>>
>>>>> These aren't that old, as I recreated them all around July last
year.
>>>>> They mostly hold CephFS data with erasure coding, with a mix of
large
>>>>> and small files. The OSDs are at around 80%-85% utilization right
now.
>>>>> Most of the data was written sequentially when the OSDs were created
(I
>>>>> rsynced everything from a remote backup). Since then more data has
been
>>>>> added, but not particularly quickly.
>>>>>
>>>>> At some point I noticed pathologically slow writes, and I
couldn't
>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>> writing one large file sequentially, and that's when I stumbled
upon the
>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>> opened up some larger free extents and resolved the problem, but
only
>>>>> until those get filled up and I'm back to fragmented tiny
extents. So
>>>>> effectively I'm stuck at the current utilization, as trying to
fill them
>>>>> up any more just slows down to an absolute crawl.
>>>>>
>>>>> I'm adding a few more OSDs and plan on doing the dance of
removing one
>>>>> OSD at a time and replacing it with another one to hopefully improve
the
>>>>> situation, but obviously this is going to take forever.
>>>>>
>>>>> Is there any plan for offering a defrag tool of some sort for
bluestore?
>>>>>
>>>>> - Hector
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:36 p.m.
Ok, I'm gathering the "allocation stats probe" stuff. Not sure I follow what
you mean by the historic probes. just:
| egrep "allocation stats probe|probe" ?
That gets something like:
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 24503, 24503, 58141900800
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 24594, 24594, 56951898112
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-6: 19737, 19737, 37299027968
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-14: 20373, 20373, 35302801408
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-30: 19072, 19072, 33645854720
if that is the right query, then I'll gather the metrics, restart and gather some more
after and let you know.
Thanks,
Kevin
________________________________________
From: Igor Fedotov <igor.fedotov(a)croit.io>
Sent: Thursday, May 25, 2023 9:29 AM
To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
Just run through available logs for a specific OSD (which you suspect
suffer from high fragmentation) and collect all allocation stats probes
you can find ("allocation stats probe" string is a perfect grep pattern,
please append lines with historic probes following day-0 line as well.
Given this is printed once per day there wouldn't be too many).
Then do OSD restart and wait a couple more days. Would allocation stats
show much better disparity between cnt and frags columns?
Is the similar pattern (eventual degradation in stats prior to restart
and severe improvement afterwards) be observed for other OSDs?
On 25/05/2023 19:20, Fox, Kevin M wrote:
> If you can give me instructions on what you want me to gather before the restart and
after restart I can do it. I have some running away right now.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov(a)croit.io>
> Sent: Thursday, May 25, 2023 9:17 AM
> To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Perhaps...
>
> I don't like the idea to use fragmentation score as a real index. IMO
> it's mostly like a very imprecise first turn marker to alert that
> something might be wrong. But not a real quantitative high-quality estimate.
>
> So in fact I'd like to see a series of allocation probes showing
> eventual degradation without OSD restart and immediate severe
> improvement after the restart.
>
> Can you try to collect something like that? Would the same behavior
> persist with an alternative allocator?
>
>
> Thanks,
>
> Igor
>
>
> On 25/05/2023 18:41, Fox, Kevin M wrote:
>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>
>> We still see run away osds at times, somewhat randomly, that causes runaway
fragmentation issues.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov(a)croit.io>
>> Sent: Thursday, May 25, 2023 8:29 AM
>> To: Hector Martin; ceph-users(a)ceph.io
>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Check twice before you click! This email originated from outside PNNL.
>>
>>
>> Hi Hector,
>>
>> I can advise two tools for further fragmentation analysis:
>>
>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>> a list of free chunks for an OSD and try to analyze whether it's really
>> highly fragmented and lacks long enough extents. free-dump just returns
>> a list of extents in json format, I can take a look to the output if
>> shared...
>>
>> 2) You might want to look for allocation probs in OSD logs and see how
>> fragmentation in allocated chunks has evolved.
>>
>> E.g.
>>
>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>> probe -1: 35168547, 46401246, 1199516209152
>> probe -3: 27275094, 35681802, 200121712640
>> probe -5: 34847167, 52539758, 271272230912
>> probe -9: 44291522, 60025613, 523997483008
>> probe -17: 10646313, 10646313, 155178434560
>>
>> The first probe refers to the last day while others match days (or
>> rather probes) -1, -3, -5, -9, -17
>>
>> 'cnt' column represents the amount of allocations performed in the
>> previous 24 hours and 'frags' one shows amount of fragments in the
>> resulted allocations. So significant mismatch between frags and cnt
>> might indicate some issues with high fragmentation indeed.
>>
>> Apart from retrospective analysis you might also want how OSD behavior
>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>> fragmentation... Which in turn might indicate some issues with BlueStore
>> allocator..
>>
>> Just FYI: allocation probe printing interval is controlled by
>> bluestore_alloc_stats_dump_interval parameter.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>> On 24/05/2023 17:18, Hector Martin wrote:
>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural
result
>>>> of using copy-on-write and never implementing any kind of
>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>> now, probably piggybacking on scrub or other operations that already
>>>> area reading all of the extents for an object anyway.
>>>>
>>>>
>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>> case here:
>>>>
>>>>
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>>>
>>>>
>>>> Adam ended up going the extra mile and completely changed how shared
>>>> blobs works which probably eliminates the need to do defrag on clone
>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>> identify any times we are doing full object reads of fragmented objects
>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>> or other things, but the point is that if we are already doing most of
>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>> clean it up isn't that bad, especially if we are doing it over the
>>>> course of months or years and can help keep freespace less fragmented.
>>> Note that my particular issue seemed to specifically be free space
>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>> objects other than those forced by the free space becoming fragmented
>>> (unless there is some weird pathological use case I'm hitting). Most of
>>> my write workloads are just copying files in bulk and incrementally
>>> writing out files.
>>>
>>> Would simply defragging objects during scrub/etc help with free space
>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>> note that if free space is already fragmented, you wouldn't even have a
>>> place to put down a defragmented object.
>>>
>>> Are there any stats I can look at to figure out how bad object and free
>>> space fragmentation is? It would be nice to have some clearer data
>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>> fragmentation number :). Also would be interesting to get some kind of
>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>> it's doing something pathological that causes more fragmentation for
>>> some reason.
>>>
>>>> Mark
>>>>
>>>>
>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>> Hi,
>>>>>
>>>>> I've been seeing relatively large fragmentation numbers on all my
OSDs:
>>>>>
>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>> {
>>>>> "fragmentation_rating": 0.77251526920454427
>>>>> }
>>>>>
>>>>> These aren't that old, as I recreated them all around July last
year.
>>>>> They mostly hold CephFS data with erasure coding, with a mix of
large
>>>>> and small files. The OSDs are at around 80%-85% utilization right
now.
>>>>> Most of the data was written sequentially when the OSDs were created
(I
>>>>> rsynced everything from a remote backup). Since then more data has
been
>>>>> added, but not particularly quickly.
>>>>>
>>>>> At some point I noticed pathologically slow writes, and I
couldn't
>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>> writing one large file sequentially, and that's when I stumbled
upon the
>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>> opened up some larger free extents and resolved the problem, but
only
>>>>> until those get filled up and I'm back to fragmented tiny
extents. So
>>>>> effectively I'm stuck at the current utilization, as trying to
fill them
>>>>> up any more just slows down to an absolute crawl.
>>>>>
>>>>> I'm adding a few more OSDs and plan on doing the dance of
removing one
>>>>> OSD at a time and replacing it with another one to hopefully improve
the
>>>>> situation, but obviously this is going to take forever.
>>>>>
>>>>> Is there any plan for offering a defrag tool of some sort for
bluestore?
>>>>>
>>>>> - Hector
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
26 May
26 May
12:14 a.m.
Yeah this looks fine. Please collect all of them for a given OSD.
Then restart OSD, wait more to come (1-2 days) and collect them too.
A side note - in the attached probe I can't see any fragmentation at all
- amount of allocations is equal to amount of fragments, e.g.
cnt: 27637 frags: 27637
And the average requested chunk is 63777406976 / 27637 = ~2308 bytes.
I.e. in average one needed less than a single alloc unit. Which would
tell us nothing about the fragmentation...
Thanks,
Igor
On 25/05/2023 19:36, Fox, Kevin M wrote:
> Ok, I'm gathering the "allocation stats probe" stuff. Not sure I follow
what you mean by the historic probes. just:
> | egrep "allocation stats probe|probe" ?
>
> That gets something like:
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 24503, 24503, 58141900800
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 24594, 24594, 56951898112
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-6: 19737, 19737, 37299027968
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-14: 20373, 20373, 35302801408
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-30: 19072, 19072, 33645854720
>
> if that is the right query, then I'll gather the metrics, restart and gather some
more after and let you know.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov(a)croit.io>
> Sent: Thursday, May 25, 2023 9:29 AM
> To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Just run through available logs for a specific OSD (which you suspect
> suffer from high fragmentation) and collect all allocation stats probes
> you can find ("allocation stats probe" string is a perfect grep pattern,
> please append lines with historic probes following day-0 line as well.
> Given this is printed once per day there wouldn't be too many).
>
> Then do OSD restart and wait a couple more days. Would allocation stats
> show much better disparity between cnt and frags columns?
>
> Is the similar pattern (eventual degradation in stats prior to restart
> and severe improvement afterwards) be observed for other OSDs?
>
>
> On 25/05/2023 19:20, Fox, Kevin M wrote:
>> If you can give me instructions on what you want me to gather before the restart
and after restart I can do it. I have some running away right now.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov(a)croit.io>
>> Sent: Thursday, May 25, 2023 9:17 AM
>> To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
>> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Perhaps...
>>
>> I don't like the idea to use fragmentation score as a real index. IMO
>> it's mostly like a very imprecise first turn marker to alert that
>> something might be wrong. But not a real quantitative high-quality estimate.
>>
>> So in fact I'd like to see a series of allocation probes showing
>> eventual degradation without OSD restart and immediate severe
>> improvement after the restart.
>>
>> Can you try to collect something like that? Would the same behavior
>> persist with an alternative allocator?
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>> On 25/05/2023 18:41, Fox, Kevin M wrote:
>>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>>
>>> We still see run away osds at times, somewhat randomly, that causes runaway
fragmentation issues.
>>>
>>> Thanks,
>>> Kevin
>>>
>>> ________________________________________
>>> From: Igor Fedotov <igor.fedotov(a)croit.io>
>>> Sent: Thursday, May 25, 2023 8:29 AM
>>> To: Hector Martin; ceph-users(a)ceph.io
>>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>>
>>> Check twice before you click! This email originated from outside PNNL.
>>>
>>>
>>> Hi Hector,
>>>
>>> I can advise two tools for further fragmentation analysis:
>>>
>>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>>> a list of free chunks for an OSD and try to analyze whether it's really
>>> highly fragmented and lacks long enough extents. free-dump just returns
>>> a list of extents in json format, I can take a look to the output if
>>> shared...
>>>
>>> 2) You might want to look for allocation probs in OSD logs and see how
>>> fragmentation in allocated chunks has evolved.
>>>
>>> E.g.
>>>
>>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>>> probe -1: 35168547, 46401246, 1199516209152
>>> probe -3: 27275094, 35681802, 200121712640
>>> probe -5: 34847167, 52539758, 271272230912
>>> probe -9: 44291522, 60025613, 523997483008
>>> probe -17: 10646313, 10646313, 155178434560
>>>
>>> The first probe refers to the last day while others match days (or
>>> rather probes) -1, -3, -5, -9, -17
>>>
>>> 'cnt' column represents the amount of allocations performed in the
>>> previous 24 hours and 'frags' one shows amount of fragments in the
>>> resulted allocations. So significant mismatch between frags and cnt
>>> might indicate some issues with high fragmentation indeed.
>>>
>>> Apart from retrospective analysis you might also want how OSD behavior
>>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>>> fragmentation... Which in turn might indicate some issues with BlueStore
>>> allocator..
>>>
>>> Just FYI: allocation probe printing interval is controlled by
>>> bluestore_alloc_stats_dump_interval parameter.
>>>
>>>
>>> Thanks,
>>>
>>> Igor
>>>
>>>
>>>
>>> On 24/05/2023 17:18, Hector Martin wrote:
>>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural
result
>>>>> of using copy-on-write and never implementing any kind of
>>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>>> now, probably piggybacking on scrub or other operations that already
>>>>> area reading all of the extents for an object anyway.
>>>>>
>>>>>
>>>>> I wrote a very simply prototype for clone to speed up the rbd mirror
use
>>>>> case here:
>>>>>
>>>>>
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>>>>
>>>>>
>>>>> Adam ended up going the extra mile and completely changed how shared
>>>>> blobs works which probably eliminates the need to do defrag on clone
>>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>>> identify any times we are doing full object reads of fragmented
objects
>>>>> and consider defragmenting at that time. It might be clone, or
scrub,
>>>>> or other things, but the point is that if we are already doing most
of
>>>>> the work (seeks on HDD especially!) the extra cost of a large write
to
>>>>> clean it up isn't that bad, especially if we are doing it over
the
>>>>> course of months or years and can help keep freespace less
fragmented.
>>>> Note that my particular issue seemed to specifically be free space
>>>> fragmentation. I don't use RBD mirror and I would not *expect* most
of
>>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>>> objects other than those forced by the free space becoming fragmented
>>>> (unless there is some weird pathological use case I'm hitting). Most
of
>>>> my write workloads are just copying files in bulk and incrementally
>>>> writing out files.
>>>>
>>>> Would simply defragging objects during scrub/etc help with free space
>>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>>> note that if free space is already fragmented, you wouldn't even have
a
>>>> place to put down a defragmented object.
>>>>
>>>> Are there any stats I can look at to figure out how bad object and free
>>>> space fragmentation is? It would be nice to have some clearer data
>>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>>> fragmentation number :). Also would be interesting to get some kind of
>>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>>> it's doing something pathological that causes more fragmentation for
>>>> some reason.
>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've been seeing relatively large fragmentation numbers on
all my OSDs:
>>>>>>
>>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>>> {
>>>>>> "fragmentation_rating": 0.77251526920454427
>>>>>> }
>>>>>>
>>>>>> These aren't that old, as I recreated them all around July
last year.
>>>>>> They mostly hold CephFS data with erasure coding, with a mix of
large
>>>>>> and small files. The OSDs are at around 80%-85% utilization right
now.
>>>>>> Most of the data was written sequentially when the OSDs were
created (I
>>>>>> rsynced everything from a remote backup). Since then more data
has been
>>>>>> added, but not particularly quickly.
>>>>>>
>>>>>> At some point I noticed pathologically slow writes, and I
couldn't
>>>>>> figure out what was wrong. Eventually I did some block tracing
and
>>>>>> noticed the I/Os were very small, even though CephFS-side I was
just
>>>>>> writing one large file sequentially, and that's when I
stumbled upon the
>>>>>> free space fragmentation problem. Indeed, deleting some large
files
>>>>>> opened up some larger free extents and resolved the problem, but
only
>>>>>> until those get filled up and I'm back to fragmented tiny
extents. So
>>>>>> effectively I'm stuck at the current utilization, as trying
to fill them
>>>>>> up any more just slows down to an absolute crawl.
>>>>>>
>>>>>> I'm adding a few more OSDs and plan on doing the dance of
removing one
>>>>>> OSD at a time and replacing it with another one to hopefully
improve the
>>>>>> situation, but obviously this is going to take forever.
>>>>>>
>>>>>> Is there any plan for offering a defrag tool of some sort for
bluestore?
>>>>>>
>>>>>> - Hector
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
31 May
31 May
5:11 a.m.
Ok, I restarted it May 25th, ~11:30, let it run over the long weekend and just checked on
it. Data attached.
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 107: cnt: 17991 fr
ags: 17991 size: 32016760832
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 20267, 20267, 39482425344
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 19737, 19737, 37299027968
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-7: 18498, 18498, 32395558912
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-11: 20373, 20373, 35302801408
May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-27: 19072, 19072, 33645854720
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 108: cnt: 24594 fr
ags: 24594 size: 56951898112
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 17991, 17991, 32016760832
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 20267, 20267, 39482425344
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-4: 19737, 19737, 37299027968
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-12: 20373, 20373, 35302801408
May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-28: 19072, 19072, 33645854720
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 109: cnt: 24503 frags: 24503 size: 58141900800
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 24594, 24594, 56951898112
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 20267, 20267, 39482425344
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-5: 19737, 19737, 37299027968
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-13: 20373, 20373, 35302801408
May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-29: 19072, 19072, 33645854720
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 24503, 24503, 58141900800
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 24594, 24594, 56951898112
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-6: 19737, 19737, 37299027968
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-14: 20373, 20373, 35302801408
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-30: 19072, 19072, 33645854720
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 111: cnt: 22136 frags: 22136 size: 48656023552
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 27637, 27637, 63777406976
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 24594, 24594, 56951898112
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-7: 19737, 19737, 37299027968
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-15: 20373, 20373, 35302801408
May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-31: 19072, 19072, 33645854720
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 0: cnt: 21986 frags: 21986 size: 47407562752
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 0, 0, 0
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 0, 0, 0
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-4: 0, 0, 0
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-8: 0, 0, 0
May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-16: 0, 0, 0
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 1: cnt: 21145 frags: 21145 size: 44858146816
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 21986, 21986, 47407562752
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 0, 0, 0
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-5: 0, 0, 0
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-9: 0, 0, 0
May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-17: 0, 0, 0
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 2: cnt: 17987 frags: 17987 size: 32446676992
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 21145, 21145, 44858146816
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 21986, 21986, 47407562752
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-6: 0, 0, 0
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-10: 0, 0, 0
May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-18: 0, 0, 0
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 3: cnt: 17509 frags: 17509 size: 31015436288
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 17987, 17987, 32446676992
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 21986, 21986, 47407562752
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-7: 0, 0, 0
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-11: 0, 0, 0
May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-19: 0, 0, 0
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 4: cnt: 21016 frags: 21016 size: 45432438784
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 17509, 17509, 31015436288
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 17987, 17987, 32446676992
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-4: 21986, 21986, 47407562752
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-12: 0, 0, 0
May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-20: 0, 0, 0
Thanks,
Kevin
________________________________________
From: Fox, Kevin M <Kevin.Fox(a)pnnl.gov>
Sent: Thursday, May 25, 2023 9:36 AM
To: Igor Fedotov; Hector Martin; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
Ok, I'm gathering the "allocation stats probe" stuff. Not sure I follow what
you mean by the historic probes. just:
| egrep "allocation stats probe|probe" ?
That gets something like:
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 24503, 24503, 58141900800
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 24594, 24594, 56951898112
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-6: 19737, 19737, 37299027968
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-14: 20373, 20373, 35302801408
May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-30: 19072, 19072, 33645854720
if that is the right query, then I'll gather the metrics, restart and gather some more
after and let you know.
Thanks,
Kevin
________________________________________
From: Igor Fedotov <igor.fedotov(a)croit.io>
Sent: Thursday, May 25, 2023 9:29 AM
To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
Just run through available logs for a specific OSD (which you suspect
suffer from high fragmentation) and collect all allocation stats probes
you can find ("allocation stats probe" string is a perfect grep pattern,
please append lines with historic probes following day-0 line as well.
Given this is printed once per day there wouldn't be too many).
Then do OSD restart and wait a couple more days. Would allocation stats
show much better disparity between cnt and frags columns?
Is the similar pattern (eventual degradation in stats prior to restart
and severe improvement afterwards) be observed for other OSDs?
On 25/05/2023 19:20, Fox, Kevin M wrote:
> If you can give me instructions on what you want me to gather before the restart and
after restart I can do it. I have some running away right now.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov(a)croit.io>
> Sent: Thursday, May 25, 2023 9:17 AM
> To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Perhaps...
>
> I don't like the idea to use fragmentation score as a real index. IMO
> it's mostly like a very imprecise first turn marker to alert that
> something might be wrong. But not a real quantitative high-quality estimate.
>
> So in fact I'd like to see a series of allocation probes showing
> eventual degradation without OSD restart and immediate severe
> improvement after the restart.
>
> Can you try to collect something like that? Would the same behavior
> persist with an alternative allocator?
>
>
> Thanks,
>
> Igor
>
>
> On 25/05/2023 18:41, Fox, Kevin M wrote:
>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>
>> We still see run away osds at times, somewhat randomly, that causes runaway
fragmentation issues.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov(a)croit.io>
>> Sent: Thursday, May 25, 2023 8:29 AM
>> To: Hector Martin; ceph-users(a)ceph.io
>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Check twice before you click! This email originated from outside PNNL.
>>
>>
>> Hi Hector,
>>
>> I can advise two tools for further fragmentation analysis:
>>
>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>> a list of free chunks for an OSD and try to analyze whether it's really
>> highly fragmented and lacks long enough extents. free-dump just returns
>> a list of extents in json format, I can take a look to the output if
>> shared...
>>
>> 2) You might want to look for allocation probs in OSD logs and see how
>> fragmentation in allocated chunks has evolved.
>>
>> E.g.
>>
>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>> probe -1: 35168547, 46401246, 1199516209152
>> probe -3: 27275094, 35681802, 200121712640
>> probe -5: 34847167, 52539758, 271272230912
>> probe -9: 44291522, 60025613, 523997483008
>> probe -17: 10646313, 10646313, 155178434560
>>
>> The first probe refers to the last day while others match days (or
>> rather probes) -1, -3, -5, -9, -17
>>
>> 'cnt' column represents the amount of allocations performed in the
>> previous 24 hours and 'frags' one shows amount of fragments in the
>> resulted allocations. So significant mismatch between frags and cnt
>> might indicate some issues with high fragmentation indeed.
>>
>> Apart from retrospective analysis you might also want how OSD behavior
>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>> fragmentation... Which in turn might indicate some issues with BlueStore
>> allocator..
>>
>> Just FYI: allocation probe printing interval is controlled by
>> bluestore_alloc_stats_dump_interval parameter.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>> On 24/05/2023 17:18, Hector Martin wrote:
>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural
result
>>>> of using copy-on-write and never implementing any kind of
>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>> now, probably piggybacking on scrub or other operations that already
>>>> area reading all of the extents for an object anyway.
>>>>
>>>>
>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>> case here:
>>>>
>>>>
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>>>
>>>>
>>>> Adam ended up going the extra mile and completely changed how shared
>>>> blobs works which probably eliminates the need to do defrag on clone
>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>> identify any times we are doing full object reads of fragmented objects
>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>> or other things, but the point is that if we are already doing most of
>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>> clean it up isn't that bad, especially if we are doing it over the
>>>> course of months or years and can help keep freespace less fragmented.
>>> Note that my particular issue seemed to specifically be free space
>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>> objects other than those forced by the free space becoming fragmented
>>> (unless there is some weird pathological use case I'm hitting). Most of
>>> my write workloads are just copying files in bulk and incrementally
>>> writing out files.
>>>
>>> Would simply defragging objects during scrub/etc help with free space
>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>> note that if free space is already fragmented, you wouldn't even have a
>>> place to put down a defragmented object.
>>>
>>> Are there any stats I can look at to figure out how bad object and free
>>> space fragmentation is? It would be nice to have some clearer data
>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>> fragmentation number :). Also would be interesting to get some kind of
>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>> it's doing something pathological that causes more fragmentation for
>>> some reason.
>>>
>>>> Mark
>>>>
>>>>
>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>> Hi,
>>>>>
>>>>> I've been seeing relatively large fragmentation numbers on all my
OSDs:
>>>>>
>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>> {
>>>>> "fragmentation_rating": 0.77251526920454427
>>>>> }
>>>>>
>>>>> These aren't that old, as I recreated them all around July last
year.
>>>>> They mostly hold CephFS data with erasure coding, with a mix of
large
>>>>> and small files. The OSDs are at around 80%-85% utilization right
now.
>>>>> Most of the data was written sequentially when the OSDs were created
(I
>>>>> rsynced everything from a remote backup). Since then more data has
been
>>>>> added, but not particularly quickly.
>>>>>
>>>>> At some point I noticed pathologically slow writes, and I
couldn't
>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>> writing one large file sequentially, and that's when I stumbled
upon the
>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>> opened up some larger free extents and resolved the problem, but
only
>>>>> until those get filled up and I'm back to fragmented tiny
extents. So
>>>>> effectively I'm stuck at the current utilization, as trying to
fill them
>>>>> up any more just slows down to an absolute crawl.
>>>>>
>>>>> I'm adding a few more OSDs and plan on doing the dance of
removing one
>>>>> OSD at a time and replacing it with another one to hopefully improve
the
>>>>> situation, but obviously this is going to take forever.
>>>>>
>>>>> Is there any plan for offering a defrag tool of some sort for
bluestore?
>>>>>
>>>>> - Hector
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:33 p.m.
Hi Kevin,
according to the shared probes there were no fragmented allocations -
cnt = frags for all the probes. And average allocation request is
pretty large - more than 1.5 MB for the probes I checked.
So to me it looks like your disk fragmentation (at least for new
allocations) is of little significance at the moment - it doesn't affect
write requests.
As I mentioned before for further analysis you might want to run through
the output from 'ceph tell osd.N bluestore allocator dump block' command.
This is my recent commit to build free space histogram from it:
https://github.com/ceph/ceph/pull/51820
One can use this as an example and create a script to do the same (just
to avoid all the tricks with building/upgrading Ceph binaries) or
backport and build custom Ceph image.
Thanks,
Igor
On 31/05/2023 01:11, Fox, Kevin M wrote:
> Ok, I restarted it May 25th, ~11:30, let it run over the long weekend and just
checked on it. Data attached.
>
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 107: cnt: 17991 fr
> ags: 17991 size: 32016760832
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 20267, 20267, 39482425344
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 19737, 19737, 37299027968
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-7: 18498, 18498, 32395558912
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-11: 20373, 20373, 35302801408
> May 21 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-21T18:24:34.040+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-27: 19072, 19072, 33645854720
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 108: cnt: 24594 fr
> ags: 24594 size: 56951898112
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 17991, 17991, 32016760832
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 20267, 20267, 39482425344
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-4: 19737, 19737, 37299027968
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-12: 20373, 20373, 35302801408
> May 22 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-22T18:24:34.057+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-28: 19072, 19072, 33645854720
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 109: cnt: 24503 frags: 24503 size: 58141900800
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 24594, 24594, 56951898112
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 20267, 20267, 39482425344
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-5: 19737, 19737, 37299027968
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-13: 20373, 20373, 35302801408
> May 23 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-23T18:24:34.095+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-29: 19072, 19072, 33645854720
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 24503, 24503, 58141900800
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 24594, 24594, 56951898112
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-6: 19737, 19737, 37299027968
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-14: 20373, 20373, 35302801408
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-30: 19072, 19072, 33645854720
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 111: cnt: 22136 frags: 22136 size: 48656023552
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 27637, 27637, 63777406976
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 24594, 24594, 56951898112
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-7: 19737, 19737, 37299027968
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-15: 20373, 20373, 35302801408
> May 25 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-25T18:24:34.151+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-31: 19072, 19072, 33645854720
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 0: cnt: 21986 frags: 21986 size: 47407562752
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 0, 0, 0
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 0, 0, 0
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-4: 0, 0, 0
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-8: 0, 0, 0
> May 26 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-26T18:35:22.701+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-16: 0, 0, 0
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 1: cnt: 21145 frags: 21145 size: 44858146816
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 21986, 21986, 47407562752
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 0, 0, 0
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-5: 0, 0, 0
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-9: 0, 0, 0
> May 27 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-27T18:35:22.740+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-17: 0, 0, 0
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 2: cnt: 17987 frags: 17987 size: 32446676992
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 21145, 21145, 44858146816
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 21986, 21986, 47407562752
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-6: 0, 0, 0
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-10: 0, 0, 0
> May 28 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-28T18:35:22.790+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-18: 0, 0, 0
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 3: cnt: 17509 frags: 17509 size: 31015436288
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 17987, 17987, 32446676992
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-3: 21986, 21986, 47407562752
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-7: 0, 0, 0
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-11: 0, 0, 0
> May 29 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-29T18:35:22.815+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-19: 0, 0, 0
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 4: cnt: 21016 frags: 21016 size: 45432438784
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 17509, 17509, 31015436288
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 17987, 17987, 32446676992
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-4: 21986, 21986, 47407562752
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-12: 0, 0, 0
> May 30 11:35:22 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2490690]: debug
2023-05-30T18:35:22.826+0000 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-20: 0, 0, 0
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Fox, Kevin M <Kevin.Fox(a)pnnl.gov>
> Sent: Thursday, May 25, 2023 9:36 AM
> To: Igor Fedotov; Hector Martin; ceph-users(a)ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Ok, I'm gathering the "allocation stats probe" stuff. Not sure I follow
what you mean by the historic probes. just:
> | egrep "allocation stats probe|probe" ?
>
> That gets something like:
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183)
allocation stats probe 110: cnt: 27637 frags: 27637 size: 63777406976
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-1: 24503, 24503, 58141900800
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-2: 24594, 24594, 56951898112
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-6: 19737, 19737, 37299027968
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-14: 20373, 20373, 35302801408
> May 24 11:24:34 cf8 ceph-4e4184f5-7733-453b-b72c-2b43422fd027-osd-183[2282674]: debug
2023-05-24T18:24:34.105+0000 7f53603fc700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe
-30: 19072, 19072, 33645854720
>
> if that is the right query, then I'll gather the metrics, restart and gather some
more after and let you know.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Igor Fedotov <igor.fedotov(a)croit.io>
> Sent: Thursday, May 25, 2023 9:29 AM
> To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>
> Just run through available logs for a specific OSD (which you suspect
> suffer from high fragmentation) and collect all allocation stats probes
> you can find ("allocation stats probe" string is a perfect grep pattern,
> please append lines with historic probes following day-0 line as well.
> Given this is printed once per day there wouldn't be too many).
>
> Then do OSD restart and wait a couple more days. Would allocation stats
> show much better disparity between cnt and frags columns?
>
> Is the similar pattern (eventual degradation in stats prior to restart
> and severe improvement afterwards) be observed for other OSDs?
>
>
> On 25/05/2023 19:20, Fox, Kevin M wrote:
>> If you can give me instructions on what you want me to gather before the restart
and after restart I can do it. I have some running away right now.
>>
>> Thanks,
>> Kevin
>>
>> ________________________________________
>> From: Igor Fedotov <igor.fedotov(a)croit.io>
>> Sent: Thursday, May 25, 2023 9:17 AM
>> To: Fox, Kevin M; Hector Martin; ceph-users(a)ceph.io
>> Subject: Re: [ceph-users] Re: BlueStore fragmentation woes
>>
>> Perhaps...
>>
>> I don't like the idea to use fragmentation score as a real index. IMO
>> it's mostly like a very imprecise first turn marker to alert that
>> something might be wrong. But not a real quantitative high-quality estimate.
>>
>> So in fact I'd like to see a series of allocation probes showing
>> eventual degradation without OSD restart and immediate severe
>> improvement after the restart.
>>
>> Can you try to collect something like that? Would the same behavior
>> persist with an alternative allocator?
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>> On 25/05/2023 18:41, Fox, Kevin M wrote:
>>> Is this related to https://tracker.ceph.com/issues/58022 ?
>>>
>>> We still see run away osds at times, somewhat randomly, that causes runaway
fragmentation issues.
>>>
>>> Thanks,
>>> Kevin
>>>
>>> ________________________________________
>>> From: Igor Fedotov <igor.fedotov(a)croit.io>
>>> Sent: Thursday, May 25, 2023 8:29 AM
>>> To: Hector Martin; ceph-users(a)ceph.io
>>> Subject: [ceph-users] Re: BlueStore fragmentation woes
>>>
>>> Check twice before you click! This email originated from outside PNNL.
>>>
>>>
>>> Hi Hector,
>>>
>>> I can advise two tools for further fragmentation analysis:
>>>
>>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>>> a list of free chunks for an OSD and try to analyze whether it's really
>>> highly fragmented and lacks long enough extents. free-dump just returns
>>> a list of extents in json format, I can take a look to the output if
>>> shared...
>>>
>>> 2) You might want to look for allocation probs in OSD logs and see how
>>> fragmentation in allocated chunks has evolved.
>>>
>>> E.g.
>>>
>>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>>> probe -1: 35168547, 46401246, 1199516209152
>>> probe -3: 27275094, 35681802, 200121712640
>>> probe -5: 34847167, 52539758, 271272230912
>>> probe -9: 44291522, 60025613, 523997483008
>>> probe -17: 10646313, 10646313, 155178434560
>>>
>>> The first probe refers to the last day while others match days (or
>>> rather probes) -1, -3, -5, -9, -17
>>>
>>> 'cnt' column represents the amount of allocations performed in the
>>> previous 24 hours and 'frags' one shows amount of fragments in the
>>> resulted allocations. So significant mismatch between frags and cnt
>>> might indicate some issues with high fragmentation indeed.
>>>
>>> Apart from retrospective analysis you might also want how OSD behavior
>>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>>> fragmentation... Which in turn might indicate some issues with BlueStore
>>> allocator..
>>>
>>> Just FYI: allocation probe printing interval is controlled by
>>> bluestore_alloc_stats_dump_interval parameter.
>>>
>>>
>>> Thanks,
>>>
>>> Igor
>>>
>>>
>>>
>>> On 24/05/2023 17:18, Hector Martin wrote:
>>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural
result
>>>>> of using copy-on-write and never implementing any kind of
>>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>>> now, probably piggybacking on scrub or other operations that already
>>>>> area reading all of the extents for an object anyway.
>>>>>
>>>>>
>>>>> I wrote a very simply prototype for clone to speed up the rbd mirror
use
>>>>> case here:
>>>>>
>>>>>
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>>>>
>>>>>
>>>>> Adam ended up going the extra mile and completely changed how shared
>>>>> blobs works which probably eliminates the need to do defrag on clone
>>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>>> identify any times we are doing full object reads of fragmented
objects
>>>>> and consider defragmenting at that time. It might be clone, or
scrub,
>>>>> or other things, but the point is that if we are already doing most
of
>>>>> the work (seeks on HDD especially!) the extra cost of a large write
to
>>>>> clean it up isn't that bad, especially if we are doing it over
the
>>>>> course of months or years and can help keep freespace less
fragmented.
>>>> Note that my particular issue seemed to specifically be free space
>>>> fragmentation. I don't use RBD mirror and I would not *expect* most
of
>>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>>> objects other than those forced by the free space becoming fragmented
>>>> (unless there is some weird pathological use case I'm hitting). Most
of
>>>> my write workloads are just copying files in bulk and incrementally
>>>> writing out files.
>>>>
>>>> Would simply defragging objects during scrub/etc help with free space
>>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>>> note that if free space is already fragmented, you wouldn't even have
a
>>>> place to put down a defragmented object.
>>>>
>>>> Are there any stats I can look at to figure out how bad object and free
>>>> space fragmentation is? It would be nice to have some clearer data
>>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>>> fragmentation number :). Also would be interesting to get some kind of
>>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>>> it's doing something pathological that causes more fragmentation for
>>>> some reason.
>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've been seeing relatively large fragmentation numbers on
all my OSDs:
>>>>>>
>>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>>> {
>>>>>> "fragmentation_rating": 0.77251526920454427
>>>>>> }
>>>>>>
>>>>>> These aren't that old, as I recreated them all around July
last year.
>>>>>> They mostly hold CephFS data with erasure coding, with a mix of
large
>>>>>> and small files. The OSDs are at around 80%-85% utilization right
now.
>>>>>> Most of the data was written sequentially when the OSDs were
created (I
>>>>>> rsynced everything from a remote backup). Since then more data
has been
>>>>>> added, but not particularly quickly.
>>>>>>
>>>>>> At some point I noticed pathologically slow writes, and I
couldn't
>>>>>> figure out what was wrong. Eventually I did some block tracing
and
>>>>>> noticed the I/Os were very small, even though CephFS-side I was
just
>>>>>> writing one large file sequentially, and that's when I
stumbled upon the
>>>>>> free space fragmentation problem. Indeed, deleting some large
files
>>>>>> opened up some larger free extents and resolved the problem, but
only
>>>>>> until those get filled up and I'm back to fragmented tiny
extents. So
>>>>>> effectively I'm stuck at the current utilization, as trying
to fill them
>>>>>> up any more just slows down to an absolute crawl.
>>>>>>
>>>>>> I'm adding a few more OSDs and plan on doing the dance of
removing one
>>>>>> OSD at a time and replacing it with another one to hopefully
improve the
>>>>>> situation, but obviously this is going to take forever.
>>>>>>
>>>>>> Is there any plan for offering a defrag tool of some sort for
bluestore?
>>>>>>
>>>>>> - Hector
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>> - Hector
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
26 May
26 May
12:36 a.m.
On 5/25/23 18:17, Igor Fedotov wrote:
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO
it's mostly like a very imprecise first turn marker to alert that
something might be wrong. But not a real quantitative high-quality
estimate.
Chiming in on the high fragmentation issue. We started collecting
"fragmentation_rating" of each OSD this afternoon. All OSDs that have
been provisioned a year ago have a fragmentation rating of ~ 0.9. Not
sure for how long they are on this level. And after reading your mails
it might not be a problem at all. But we will start collecting this
information in the coming weeks.
We will be re-provisioning all our OSDs, so that might be a good time to
look at the behavior and development of "cnt versus frags" ratio.
After we completely emptied a host, even after having the OSDs run idle
for a couple of hours, the fragmentation ratio would not drop lower than
0.27 for some OSDs, and up to 0.62 for others. Is it expected that this
will not go to ~ zero?
We have not noticed any performance degradation whatsoever. We are
running with bitmap for current OSDs, hybrid allocator for the new
provisioned OSDs (4k min alloc size).
Gr. Stefan
3:12 a.m.
On 25/05/2023 20:36, Stefan Kooman wrote:
On 5/25/23 18:17, Igor Fedotov wrote:
Could you please collect allocation probes from existing OSD logs? Just
a few samples from different OSDs...
Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO
it's mostly like a very imprecise first turn marker to alert that
something might be wrong. But not a real quantitative high-quality
estimate.
Chiming in on the high fragmentation issue. We started collecting
"fragmentation_rating" of each OSD this afternoon. All OSDs that have
been provisioned a year ago have a fragmentation rating of ~ 0.9. Not
sure for how long they are on this level. And after reading your mails it might not be a problem
at all. But we
will start collecting this information in the coming weeks.
We will be re-provisioning all our OSDs, so that might be a good time
to look at the behavior and development of "cnt versus frags" ratio.
After we completely emptied a host, even after having the OSDs run
idle for a couple of hours, the fragmentation ratio would not drop
lower than 0.27 for some OSDs, and up to 0.62 for others. Is it
expected that this will not go to ~ zero?
You might be facing the issue fixed by
https://github.com/ceph/ceph/pull/49885
>
> We have not noticed any performance degradation whatsoever. We are
> running with bitmap for current OSDs, hybrid allocator for the new
> provisioned OSDs (4k min alloc size).
>
> Gr. Stefan
1:39 p.m.
Hi Igor,
Should we backpot this to the p,q and reef release's?
Thanks,
k
Sent from my iPhone
On 25 May 2023, at 23:13, Igor Fedotov
<igor.fedotov(a)croit.io> wrote:
You might be facing the issue fixed by https://github.com/ceph/ceph/pull/49885
3:50 p.m.
yeah, definitely this makes sense
On 26/05/2023 09:39, Konstantin Shalygin wrote:
> Hi Igor,
>
> Should we backpot this to the p,q and reef release's?
>
>
> Thanks,
> k
> Sent from my iPhone
>
>> On 25 May 2023, at 23:13, Igor Fedotov <igor.fedotov(a)croit.io> wrote:
>>
>> You might be facing the issue fixed by https://github.com/ceph/ceph/pull/49885
8:03 p.m.
On 5/25/23 22:12, Igor Fedotov wrote:
Possibly.
I have read some tracker tickets that got mentioned in PRs [1,2]. The
problem seems to reveal itself in Pacific release. I wonder if this has
something to do with the change in default allocator: bitmap -> hybrid
in Pacific.
BlueFS 4K allocation unit will not be backported to Pacific [3]. Would
it make sense to skip re-provisiong OSDs in Pacific altogether and do
re-provisioning in Quincy release with BlueFS 4K alloc size support [4]?
Gr. Stefan
[1]: https://tracker.ceph.com/issues/58022
[2]: https://tracker.ceph.com/issues/57672
[3]: https://tracker.ceph.com/issues/58589
[4]: https://tracker.ceph.com/issues/58588
On 25/05/2023 20:36, Stefan Kooman wrote:
10 OSDs from one host, but I have checked other nodes and they are similar:
CNT FRAG Size Ratio Avg Frag size
21350923 37146899 317040259072 1.73982637659271 8534.77053554322
20951932 38122769 317841477632 1.8195347808498 8337.31352599283
21188454 37298950 278389411840 1.76034315670223 7463.73321072041
21605451 39369462 270427185152 1.82220042525379 6868.95810646333
19215230 36063713 290967818240 1.87682962941375 8068.16032059705
19293599 35464928 269238423552 1.83817068033807 7591.68109835159
19963538 36088151 315796836352 1.80770317365589 8750.70702159277
18030613 31753098 297826177024 1.76106591606176 9379.43683554909
17889602 31718012 299550142464 1.77298589426417 9444.16511551859
18475332 33264944 266053271552 1.80050588536109 7998.0074985847
18618154 31914219 254801883136 1.71414518324427 7983.96110323113
16437108 29421873 275350355968 1.78996651965784 9358.69568766067
17164338 28605353 249404649472 1.66655731202683 8718.81040838755
17895480 29658102 309047177216 1.65729569701399 10420.3288941416
19546560 34588509 301368737792 1.76954456436324 8712.97279081905
18525784 34806856 314875801600 1.87883309014075 9046.37297893266
18550989 35236438 273069948928 1.89943716747393 7749.64679823767
19085807 34605572 255512043520 1.81315738967705 7383.55209155335
17203820 31205542 277097357312 1.81387284916954 8879.74826112618
18003801 33723670 269696761856 1.87314167713807 7997.25420916525
18655425 33227176 306511810560 1.78109992133655 9224.7325069094
26380965 45627920 335281111040 1.72957736762093 7348.15680925188
24923956 44721109 328790982656 1.79430219664968 7352.03106559813
25312482 43035393 287792226304 1.70016488308021 6687.33817079351
25841471 46276699 288168476672 1.79079197929561 6227.07502693742
25618384 43785917 321591488512 1.70915999229303 7344.63294469772
26006097 45056206 298747666432 1.73252472295247 6630.55532088077
26684805 45196730 351100243968 1.69372532420604 7768.26650883814
24025872 42450135 353265467392 1.76685095966548 8321.89267223768
24080466 45510525 371726323712 1.88993539410741 8167.91991988666
23195936 45095051 326473826304 1.94409274969546 7239.68193990955
23653302 43312705 307549573120 1.83114835298683 7100.67803707942
21589455 40034670 322982109184 1.85436223378497 8067.56017182107
22469039 42042723 314323701760 1.87114023879704 7476.29266924504
23647633 43486098 370003841024 1.83891969230071 8508.55464254346
23750561 37387139 320471453696 1.57415814304344 8571.70305799542
23142315 38640274 329341046784 1.66968058294946 8523.25857689312
23539469 39573256 292528910336 1.68114480407353 7392.08596674481
23810938 37968499 277270380544 1.59458224619291 7302.64266027477
19361754 33610252 286391676928 1.73590946357443 8520.96190555191
20331818 34119736 256076865536 1.67814486633709 7505.24170339419
21017537 35862221 318755282944 1.70629988661374 8888.33078531305
21660731 42648077 329217507328 1.96891217567865 7719.39863380007
20708620 42285124 344562262016 2.04190931119505 8148.54562129225
21371937 43158447 312754188288 2.01939800777066 7246.65065654471
21447150 40034134 283613331456 1.86664120873869 7084.28790931259
18906469 36598724 302526169088 1.93577785465916 8266.03050663734
20086704 36824872 280208515072 1.83329589563325 7609.21898308296
20912511 40116356 340019290112 1.91829455582833 8475.82691987278
17728197 30717152 270751887360 1.73267208165613 8814.35516417668
16778676 30875765 267493560320 1.84017886751017 8663.54437922429
17700395 31528725 239652761600 1.78124414737637 7601.09270514428
17727766 31338207 232399462400 1.76774710361136 7415.85063880649
15488369 27225173 246367821824 1.75778179096844 9049.26561252705
16332731 29287976 227973730304 1.7932075168568 7783.86769724204
17043318 31659676 274151649280 1.85760049774346 8659.33211950748
21627836 34504152 279215091712 1.59535850003671 8092.2171833697
21244729 35619286 303324131328 1.67661757417569 8515.72744405938
22132156 38534232 281272401920 1.74109707160929 7299.28656473548
22035014 34627308 246920048640 1.57146748352418 7130.78962534425
20277457 33126067 265162657792 1.63364010585746 8004.65258347754
20669142 34587911 254815776768 1.67340816566067 7367.1918714027
21648239 34364823 292156514304 1.58741886580243 8501.61557078295
21117643 34737044 292367892480 1.64492997632359 8416.60253186771
20531946 37038043 292538568704 1.8039226773731 7898.32682855301
21393711 35682241 257189515264 1.66788459468299 7207.77361668512
21738966 34753281 252140285952 1.59866301828707 7255.1505554828
19197606 32922066 269381632000 1.71490476468785 8182.40361950553
20044574 33864896 245486792704 1.68947945713389 7249.00477190304
20601681 35851902 305202065408 1.74024158514055 8512.85561943129
1.76995040322111 8014.69622126768
So average fragment size is around 8 KiB, and the ratio between requests
/ fragments a bit lower than two.
On 5/25/23 18:17, Igor Fedotov wrote:
Could you please collect allocation probes from existing OSD logs? Just
a few samples from different OSDs... Perhaps...
I don't like the idea to use fragmentation score as a real index. IMO
it's mostly like a very imprecise first turn marker to alert that
something might be wrong. But not a real quantitative high-quality
estimate.
Chiming in on the high fragmentation issue. We started collecting
"fragmentation_rating" of each OSD this afternoon. All OSDs that have
been provisioned a year ago have a fragmentation rating of ~ 0.9. Not
sure for how long they are on this level. And after reading your mails it might not be a
problem at all. But we
will start collecting this information in the coming weeks.
We will be re-provisioning all our OSDs, so that might be a good time
to look at the behavior and development of "cnt versus frags" ratio.
After we completely emptied a host, even after having the OSDs run
idle for a couple of hours, the fragmentation ratio would not drop
lower than 0.27 for some OSDs, and up to 0.62 for others. Is it
expected that this will not go to ~ zero?
You might be facing the issue fixed by
https://github.com/ceph/ceph/pull/49885
29 May
29 May
8:52 p.m.
Hi Stefan,
given that allocation probes include every allocation (including short
4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there is
negative trend in it..
As I mentioned in my reply to Hector one might want to make further
investigation by e.g. building a histogram (chunk-size, num chanks)
using the output from 'ceph tell osd.N bluestore allocator dump block'
command and monitoring how it evolves over time. Script to build such a
histogram still to be written. ;)
As for Pacific release being a culprit - likely it is. But there were
two major updates which could have the impact. Both came in the same PR
(https://github.com/ceph/ceph/pull/34588):
1. 4K allocation unit for spinners
2. Switch to avl/hybrid allocator.
Honestly I'd rather bet on 1.
BlueFS 4K allocation unit will not be backported to
Pacific [3]. Would
it make sense to skip re-provisiong OSDs in Pacific altogether
and do
re-provisioning in Quincy release with BlueFS 4K alloc size support [4]?
IIRC this feature doesn't require OSD redeployment - new superblock
format is applied on-the-fly and 4K allocations are enabled immediately.
So there is no specific requirement to re-provision OSD at Quincy+.
Hence you're free to go with Pacific and enable 4K for BlueFS later in
Quincy.
Thanks,
Igor
On 26/05/2023 16:03, Stefan Kooman wrote:
On 5/25/23 22:12, Igor Fedotov wrote:
Possibly.
I have read some tracker tickets that got mentioned in PRs [1,2]. The
problem seems to reveal itself in Pacific release. I wonder if this
has something to do with the change in default allocator: bitmap ->
hybrid in Pacific.
>
> BlueFS 4K allocation unit will not be backported to Pacific [3]. Would
> it make sense to skip re-provisiong OSDs in Pacific altogether and do
> re-provisioning in Quincy release with BlueFS 4K alloc size support [4]?
>
> Gr. Stefan
>
> [1]: https://tracker.ceph.com/issues/58022
> [2]: https://tracker.ceph.com/issues/57672
> [3]: https://tracker.ceph.com/issues/58589
> [4]: https://tracker.ceph.com/issues/58588
On 25/05/2023 20:36, Stefan Kooman wrote:
10 OSDs from one host, but I have checked other nodes and they are
similar:
CNT FRAG Size Ratio Avg Frag size
21350923 37146899 317040259072 1.73982637659271 8534.77053554322
20951932 38122769 317841477632 1.8195347808498 8337.31352599283
21188454 37298950 278389411840 1.76034315670223 7463.73321072041
21605451 39369462 270427185152 1.82220042525379 6868.95810646333
19215230 36063713 290967818240 1.87682962941375 8068.16032059705
19293599 35464928 269238423552 1.83817068033807 7591.68109835159
19963538 36088151 315796836352 1.80770317365589 8750.70702159277
18030613 31753098 297826177024 1.76106591606176 9379.43683554909
17889602 31718012 299550142464 1.77298589426417 9444.16511551859
18475332 33264944 266053271552 1.80050588536109 7998.0074985847
18618154 31914219 254801883136 1.71414518324427 7983.96110323113
16437108 29421873 275350355968 1.78996651965784 9358.69568766067
17164338 28605353 249404649472 1.66655731202683 8718.81040838755
17895480 29658102 309047177216 1.65729569701399 10420.3288941416
19546560 34588509 301368737792 1.76954456436324 8712.97279081905
18525784 34806856 314875801600 1.87883309014075 9046.37297893266
18550989 35236438 273069948928 1.89943716747393 7749.64679823767
19085807 34605572 255512043520 1.81315738967705 7383.55209155335
17203820 31205542 277097357312 1.81387284916954 8879.74826112618
18003801 33723670 269696761856 1.87314167713807 7997.25420916525
18655425 33227176 306511810560 1.78109992133655 9224.7325069094
26380965 45627920 335281111040 1.72957736762093 7348.15680925188
24923956 44721109 328790982656 1.79430219664968 7352.03106559813
25312482 43035393 287792226304 1.70016488308021 6687.33817079351
25841471 46276699 288168476672 1.79079197929561 6227.07502693742
25618384 43785917 321591488512 1.70915999229303 7344.63294469772
26006097 45056206 298747666432 1.73252472295247 6630.55532088077
26684805 45196730 351100243968 1.69372532420604 7768.26650883814
24025872 42450135 353265467392 1.76685095966548 8321.89267223768
24080466 45510525 371726323712 1.88993539410741 8167.91991988666
23195936 45095051 326473826304 1.94409274969546 7239.68193990955
23653302 43312705 307549573120 1.83114835298683 7100.67803707942
21589455 40034670 322982109184 1.85436223378497 8067.56017182107
22469039 42042723 314323701760 1.87114023879704 7476.29266924504
23647633 43486098 370003841024 1.83891969230071 8508.55464254346
23750561 37387139 320471453696 1.57415814304344 8571.70305799542
23142315 38640274 329341046784 1.66968058294946 8523.25857689312
23539469 39573256 292528910336 1.68114480407353 7392.08596674481
23810938 37968499 277270380544 1.59458224619291 7302.64266027477
19361754 33610252 286391676928 1.73590946357443 8520.96190555191
20331818 34119736 256076865536 1.67814486633709 7505.24170339419
21017537 35862221 318755282944 1.70629988661374 8888.33078531305
21660731 42648077 329217507328 1.96891217567865 7719.39863380007
20708620 42285124 344562262016 2.04190931119505 8148.54562129225
21371937 43158447 312754188288 2.01939800777066 7246.65065654471
21447150 40034134 283613331456 1.86664120873869 7084.28790931259
18906469 36598724 302526169088 1.93577785465916 8266.03050663734
20086704 36824872 280208515072 1.83329589563325 7609.21898308296
20912511 40116356 340019290112 1.91829455582833 8475.82691987278
17728197 30717152 270751887360 1.73267208165613 8814.35516417668
16778676 30875765 267493560320 1.84017886751017 8663.54437922429
17700395 31528725 239652761600 1.78124414737637 7601.09270514428
17727766 31338207 232399462400 1.76774710361136 7415.85063880649
15488369 27225173 246367821824 1.75778179096844 9049.26561252705
16332731 29287976 227973730304 1.7932075168568 7783.86769724204
17043318 31659676 274151649280 1.85760049774346 8659.33211950748
21627836 34504152 279215091712 1.59535850003671 8092.2171833697
21244729 35619286 303324131328 1.67661757417569 8515.72744405938
22132156 38534232 281272401920 1.74109707160929 7299.28656473548
22035014 34627308 246920048640 1.57146748352418 7130.78962534425
20277457 33126067 265162657792 1.63364010585746 8004.65258347754
20669142 34587911 254815776768 1.67340816566067 7367.1918714027
21648239 34364823 292156514304 1.58741886580243 8501.61557078295
21117643 34737044 292367892480 1.64492997632359 8416.60253186771
20531946 37038043 292538568704 1.8039226773731 7898.32682855301
21393711 35682241 257189515264 1.66788459468299 7207.77361668512
21738966 34753281 252140285952 1.59866301828707 7255.1505554828
19197606 32922066 269381632000 1.71490476468785 8182.40361950553
20044574 33864896 245486792704 1.68947945713389 7249.00477190304
20601681 35851902 305202065408 1.74024158514055 8512.85561943129
1.76995040322111 8014.69622126768
So average fragment size is around 8 KiB, and the ratio between
requests / fragments a bit lower than two.
On 5/25/23 18:17, Igor Fedotov wrote:
Could you please collect allocation probes from existing OSD logs?
Just a few samples from different OSDs... Perhaps...
I don't like the idea to use fragmentation score as a real index.
IMO it's mostly like a very imprecise first turn marker to alert
that something might be wrong. But not a real quantitative
high-quality estimate.
Chiming in on the high fragmentation issue. We started collecting
"fragmentation_rating" of each OSD this afternoon. All OSDs that
have been provisioned a year ago have a fragmentation rating of ~
0.9. Not sure for how long they are on this level. And after reading your mails it might not be a
problem at all. But
we will start collecting this information in the coming weeks.
We will be re-provisioning all our OSDs, so that might be a good
time to look at the behavior and development of "cnt versus frags"
ratio.
After we completely emptied a host, even after having the OSDs run
idle for a couple of hours, the fragmentation ratio would not drop
lower than 0.27 for some OSDs, and up to 0.62 for others. Is it
expected that this will not go to ~ zero?
You might be facing the issue fixed by
https://github.com/ceph/ceph/pull/49885
31 May
31 May
7:26 p.m.
On 5/29/23 15:52, Igor Fedotov wrote:
Ah, that's good to know.
Gr. Stefan
Hi Stefan,
given that allocation probes include every allocation (including short
4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there is
negative trend in it..
I did not omit them. We (currently) don't store logs for longer than 7
days. I will increase the interval in which the probes get created
(every hour).
As I mentioned in my reply to Hector one might want to make further
investigation by e.g. building a histogram (chunk-size, num chanks)
using the output from 'ceph tell osd.N bluestore allocator dump block'
command and monitoring how it evolves over time. Script to build such a
histogram still to be written. ;).
We started to investigate such a script. But when we issue a "ceph tell
osd.N bluestore allocator dump block" on OSDs that are primary for three
or more CephFS metadata PGs, that will cause a massive amount of slow
ops (thousands), osd op tp threads will time out
(2023-05-31T11:52:35.454+0200 7fee13285700 1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7fedf6fa5700' had timed out after
15.000000954s) and the OSD will reboot itself. This is true for SSD as
well as NVMe OSDs. So it seems that the whole OSD is just busy
processing this data, and production IO (client / rep ops) are just
starved. Ideally this call would be asynchronous,processed in batches,
and not hinder IO in any way. Should I open a tracker for this?
So for us this is not a suitable way of obtaining this data. The offline
way of doing this, ceph-bluestore-tool --path
/var/lib/ceph/osd/ceph-$id/ --allocator block free-dump >
/root/osd.$id_free_dump did work and resulted in a 2.7 GiB file of JSON
data. So that's quite a bit of data to process ...
As for Pacific release being a culprit - likely it is.
But there were
two major updates which could have the impact. Both came in the same PR
(https://github.com/ceph/ceph/pull/34588):
1. 4K allocation unit for spinners
@Kevin: what drive types do you use in the clusters that are suffering
from this problem? Did only HDD suffer from this after upgrading to Pacific?
2. Switch to avl/hybrid allocator.
Honestly I'd rather bet on 1.
We have no spinners. We have 4K alloc size since Luminous, bitmap since
Luminous (12.2.13?). Not sure if we are suffering (more or less) on the
3 nodes that got provisioned / filled with hybrid allocator in use. We
plan to do some experiments though: fill an OSD with PGs with bitmap
allocator. At certain amount of PGs dump the free extents, until all PGs
are present. Repeat this process with the same PGs on an OSD with hybrid
allocator. My bet is on # 2 ;-).
BlueFS 4K
allocation unit will not be backported to Pacific [3]. Would
it make sense to skip
re-provisiong OSDs in Pacific altogether and do
re-provisioning in Quincy release with BlueFS 4K alloc size support [4]?
IIRC this feature doesn't require OSD redeployment - new superblock
format is applied on-the-fly and 4K allocations are enabled immediately.
So there is no specific requirement to re-provision OSD at Quincy+.
Hence you're free to go with Pacific and enable 4K for BlueFS later in
Quincy.
9:15 p.m.
On 31/05/2023 15:26, Stefan Kooman wrote:
ah... this makes sense, good to know.. I knew that this dump might be
huge but never heard it causes that drastic impact.. Perhaps it's really
big this time or you're writing it to slow device..
Unfortunately there is no simple enough way to process that in batches
since we should collect a complete consistent snapshot made at a given
point in time. Processing in batches would create potentially
inconsistent chunks since allocation map is permanently updated by OSD
which is processing regular user ops..
Looking forward for the results... ;) Knowing internal design for both
bitmap and hybrid allocator I'd be very surprised the latter one is
worse in this regard...
>
>> >BlueFS 4K allocation unit will not be backported to Pacific [3].
>> Would it make sense to skip re-provisiong OSDs in Pacific altogether
>> and do re-provisioning in Quincy release with BlueFS 4K alloc size
>> support [4]?
>>
>> IIRC this feature doesn't require OSD redeployment - new superblock
>> format is applied on-the-fly and 4K allocations are enabled
>> immediately. So there is no specific requirement to re-provision OSD
>> at Quincy+. Hence you're free to go with Pacific and enable 4K for
>> BlueFS later in Quincy.
>
> Ah, that's good to know.
>
> Gr. Stefan
On 5/29/23 15:52, Igor Fedotov wrote:
Allocation probe contains historic data on its own, e.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
probe -1: 35168547, 46401246, 1199516209152
probe -3: 27275094, 35681802, 200121712640
probe -5: 34847167, 52539758, 271272230912
probe -9: 44291522, 60025613, 523997483008
probe -17: 10646313, 10646313, 155178434560
in the snippet above probes -1 through -17 are historic data 1 through
17 days (or more correctly probe attempts) back.
The major idea behind this representation is to try to visualize how
allocation fragmentation evolved without the need for grep through all
the logs.
From the info you shared it's unclear which records were for the
current day and which were historic ones if any.
Hence no way to estimate the degradation over time.
Please note that probes are collected since OSD restart. Hence some
historic records might be void if restart occurred not long ago.
Hi Stefan,
given that allocation probes include every allocation (including
short 4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there is
negative trend in it..
I did not omit them. We (currently) don't store logs for longer than 7
days. I will increase the interval in which the probes get created
(every hour).
As I mentioned in my reply to Hector one might want to make further
investigation by e.g. building a histogram (chunk-size, num chanks)
using the output from 'ceph tell osd.N bluestore allocator dump
block' command and monitoring how it evolves over time. Script to
build such a histogram still to be written. ;).
We started to investigate such a script. But when we issue a "ceph
tell osd.N bluestore allocator dump block" on OSDs that are primary
for three or more CephFS metadata PGs, that will cause a massive
amount of slow ops (thousands), osd op tp threads will time out
(2023-05-31T11:52:35.454+0200 7fee13285700 1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7fedf6fa5700' had timed out after
15.000000954s) and the OSD will reboot itself. This is true for SSD as
well as NVMe OSDs. So it seems that the whole OSD is just busy
processing this data, and production IO (client / rep ops) are just
starved. Ideally this call would be asynchronous,processed in batches,
and not hinder IO in any way. Should I open a tracker for this?
So for us this is not a suitable way of obtaining this
data. The
offline way of doing this, ceph-bluestore-tool --path
/var/lib/ceph/osd/ceph-$id/ --allocator block free-dump >
/root/osd.$id_free_dump did work and resulted in a 2.7 GiB file of
JSON data. So that's quite a bit of data to process ...
Yeah, offline method is fine too. In fact Ceph codebase has a way to
convert this JSON file to a binary format which might drastically
improve processing time and save disk space.
The tool name is ceph_test_alloc_replay, it's primarily intended for dev
purposes hence it's not very user-friendly. And I'm not sure it's
included in regular ceph packages, perhaps you'll need to run it yourself.
As for Pacific release being a culprit - likely
it is. But there were
two major updates which could have the impact. Both came in the same
PR (https://github.com/ceph/ceph/pull/34588):
1. 4K allocation unit for spinners
@Kevin: what drive types do you use in the clusters that are suffering
from this problem? Did only HDD suffer from this after upgrading to
Pacific?
2. Switch to avl/hybrid allocator.
Honestly I'd rather bet on 1.
We have no spinners. We have 4K alloc size since Luminous, bitmap
since Luminous (12.2.13?). Not sure if we are suffering (more or less)
on the 3 nodes that got provisioned / filled with hybrid allocator in
use. We plan to do some experiments though: fill an OSD with PGs with
bitmap allocator. At certain amount of PGs dump the free extents,
until all PGs are present. Repeat this process with the same PGs on an
OSD with hybrid allocator. My bet is on # 2 ;-).
9:46 p.m.
On 5/31/23 09:15, Igor Fedotov wrote:
Ah, that's good to know.
Gr. Stefan _______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Mark Nelson
Head of R&D (USA)
Clyso GmbH
p: +49 89 21552391 12
a: Loristraße 8 | 80335 München | Germany
w: https://clyso.com | e: mark.nelson(a)clyso.com
We are hiring: https://www.clyso.com/jobs/
On 31/05/2023 15:26, Stefan Kooman wrote:
ah... this makes sense, good to know.. I knew that this dump might be
huge but never heard it causes that drastic impact.. Perhaps it's
really big this time or you're writing it to slow device..
Unfortunately there is no simple enough way to process that in batches
since we should collect a complete consistent snapshot made at a given
point in time. Processing in batches would create potentially
inconsistent chunks since allocation map is permanently updated by OSD
which is processing regular user ops..
Looking forward for the results... ;) Knowing internal design for both
bitmap and hybrid allocator I'd be very surprised the latter one is
worse in this regard...
Related to this, I was a little surprised to learn how the hybrid
allocator works. I figured we would do something like have a coarse
grained implementation of one data structure and implement the other at
the leaves. Adam was explaining that this isn't what we do though, we
just switch over at some point?
On 5/29/23 15:52, Igor Fedotov wrote:
Allocation probe contains historic data on its own, e.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
probe -1: 35168547, 46401246, 1199516209152
probe -3: 27275094, 35681802, 200121712640
probe -5: 34847167, 52539758, 271272230912
probe -9: 44291522, 60025613, 523997483008
probe -17: 10646313, 10646313, 155178434560
in the snippet above probes -1 through -17 are historic data 1 through
17 days (or more correctly probe attempts) back.
The major idea behind this representation is to try to visualize how
allocation fragmentation evolved without the need for grep through all
the logs.
From the info you shared it's unclear which records were for the
current day and which were historic ones if any.
Hence no way to estimate the degradation over time.
Please note that probes are collected since OSD restart. Hence some
historic records might be void if restart occurred not long ago.
Hi Stefan,
given that allocation probes include every allocation (including
short 4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there
is negative trend in it..
I did not omit them. We (currently) don't store logs for longer than
7 days. I will increase the interval in which the probes get created
(every hour).
As I mentioned in my reply to Hector one might want to make further
investigation by e.g. building a histogram (chunk-size, num chanks)
using the output from 'ceph tell osd.N bluestore allocator dump
block' command and monitoring how it evolves over time. Script to
build such a histogram still to be written. ;).
We started to investigate such a script. But when we issue a "ceph
tell osd.N bluestore allocator dump block" on OSDs that are primary
for three or more CephFS metadata PGs, that will cause a massive
amount of slow ops (thousands), osd op tp threads will time out
(2023-05-31T11:52:35.454+0200 7fee13285700 1 heartbeat_map
is_healthy 'OSD::osd_op_tp thread 0x7fedf6fa5700' had timed out after
15.000000954s) and the OSD will reboot itself. This is true for SSD
as well as NVMe OSDs. So it seems that the whole OSD is just busy
processing this data, and production IO (client / rep ops) are just
starved. Ideally this call would be asynchronous,processed in
batches, and not hinder IO in any way. Should I open a tracker for this?
So for us this is not a suitable way of obtaining
this data. The
offline way of doing this, ceph-bluestore-tool --path
/var/lib/ceph/osd/ceph-$id/ --allocator block free-dump >
/root/osd.$id_free_dump did work and resulted in a 2.7 GiB file of
JSON data. So that's quite a bit of data to process ...
Yeah, offline method is fine too. In fact Ceph codebase has a way to
convert this JSON file to a binary format which might drastically
improve processing time and save disk space.
The tool name is ceph_test_alloc_replay, it's primarily intended for
dev purposes hence it's not very user-friendly. And I'm not sure it's
included in regular ceph packages, perhaps you'll need to run it
yourself.
As for Pacific release being a culprit - likely
it is. But there
were two major updates which could have the impact. Both came in the
same PR (https://github.com/ceph/ceph/pull/34588):
1. 4K allocation unit for spinners
@Kevin: what drive types do you use in the clusters that are
suffering from this problem? Did only HDD suffer from this after
upgrading to Pacific?
2. Switch to avl/hybrid allocator.
Honestly I'd rather bet on 1.
We have no spinners. We have 4K alloc size since Luminous, bitmap
since Luminous (12.2.13?). Not sure if we are suffering (more or
less) on the 3 nodes that got provisioned / filled with hybrid
allocator in use. We plan to do some experiments though: fill an OSD
with PGs with bitmap allocator. At certain amount of PGs dump the
free extents, until all PGs are present. Repeat this process with the
same PGs on an OSD with hybrid allocator. My bet is on # 2 ;-).
BlueFS 4K
allocation unit will not be backported to Pacific [3].
Would it make sense to skip
re-provisiong OSDs in Pacific altogether
and do re-provisioning in Quincy release with BlueFS 4K alloc size
support [4]?
IIRC this feature doesn't require OSD redeployment - new superblock
format is applied on-the-fly and 4K allocations are enabled
immediately. So there is no specific requirement to re-provision OSD
at Quincy+. Hence you're free to go with Pacific and enable 4K for
BlueFS later in Quincy.
10:42 p.m.
On 5/31/23 16:15, Igor Fedotov wrote:
On 31/05/2023 15:26, Stefan Kooman wrote:
Ah, I just blindly applied your suggested grep keywords, but then you
don't get the historic probes, here we go:
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) allocation stats probe 15: cnt:
19940282 frags: 37474984 size: 331549941760
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -1: 19269005, 35807252,
265695309824
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -3: 18281841, 33452239,
321271795712
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -7: 21188454, 37298950,
278389411840
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -15: 20079763, 34770424,
306357551104
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -31: 0, 0, 0
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) allocation stats probe 15: cnt:
18130163 frags: 32547753 size: 286495289344
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -1: 16671297, 28781657,
219492544512
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -3: 16781500, 31159184,
306530078720
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -7: 18475332, 33264944,
266053271552
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -15: 18799693, 32644833,
270106509312
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -31: 0, 0, 0
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) allocation stats probe 15: cnt:
19073940 frags: 39349763 size: 350442409984
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -1: 19346357, 40814157,
291156762624
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -3: 18880370, 33120073,
329544183808
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -7: 18550989, 35236438,
273069948928
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -15: 18664247, 36208229,
327013150720
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -31: 0, 0, 0
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) allocation stats probe 15: cnt:
24318407 frags: 42340467 size: 324867026944
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -1: 24202548, 41934313,
263141662720
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -3: 24258584, 43985640,
348764803072
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -7: 25312482, 43035393,
287792226304
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -15: 25280732, 40994669,
298337083392
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -31: 0, 0, 0
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) allocation stats probe 15: cnt:
21755123 frags: 42279513 size: 364872880128
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -1: 20263259, 38412587,
291021299712
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -3: 19733485, 38065426,
353624813568
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -7: 23195936, 45095051,
326473826304
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -15: 24094232, 45266621,
367246442496
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -31: 0, 0, 0
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) allocation stats probe 15: cnt:
21035351 frags: 34478758 size: 296729862144
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -1: 20215704, 32981561,
229284646912
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -3: 19533561, 31158296,
311621177344
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -7: 23539469, 39573256,
292528910336
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -15: 21308764, 34003659,
272235884544
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -31: 0, 0, 0
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) allocation stats probe 15: cnt:
21619552 frags: 38792793 size: 319570493440
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -1: 19698381, 33288562,
245733462016
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -3: 19345976, 35673735,
327406686208
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -7: 21371937, 43158447,
312754188288
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -15: 23144793, 41159338,
309643296768
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -31: 0, 0, 0
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) allocation stats probe 15: cnt:
17348416 frags: 32011874 size: 257342664704
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -1: 16688730, 29763231,
204166496256
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -3: 15778722, 28658535,
269868609536
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -7: 17700395, 31528725,
239652761600
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -15: 17331958, 29820789,
239322513408
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -31: 0, 0, 0
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) allocation stats probe 15: cnt:
21817405 frags: 35609259 size: 304181972992
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -1: 21402843, 35001032,
247483416576
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -3: 20555960, 33268576,
297290342400
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -7: 22132156, 38534232,
281272401920
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -15: 22188633, 37241809,
290561724416
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -31: 0, 0, 0
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) allocation stats probe 15: cnt:
20080223 frags: 33329658 size: 278508113920
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -1: 20682155, 32852611,
219409920000
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -3: 19216200, 34658404,
296857645056
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -7: 21393711, 35682241,
257189515264
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -15: 20233752, 31906022,
256953790464
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -31: 0, 0, 0
Thanks,
Gr. Stefan
On 5/29/23 15:52, Igor Fedotov wrote:
Allocation probe contains historic data on its own, e.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
probe -1: 35168547, 46401246, 1199516209152
probe -3: 27275094, 35681802, 200121712640
probe -5: 34847167, 52539758, 271272230912
probe -9: 44291522, 60025613, 523997483008
probe -17: 10646313, 10646313, 155178434560
in the snippet above probes -1 through -17 are historic data 1 through
17 days (or more correctly probe attempts) back. Hi Stefan,
given that allocation probes include every allocation (including
short 4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there is
negative trend in it..
I did not omit them. We (currently) don't store logs for longer than 7
days. I will increase the interval in which the probes get created
(every hour).
28 May
28 May
12:31 p.m.
So chiming in, I think something is definitely wrong with at *least* the
frag score.
Here's what happened so far:
1. I had 8 OSDs (all 8T HDDs)
2. I added 2 more (osd.0,1) , with Quincy defaults
3. I marked 2 old ones out (the ones that seemed to be struggling the
most with IOPS)
4. I added 2 more (osd.2,3), but this time I had previously set
bluestore_min_alloc_size_hdd to 16K as an experiment
This has all happened in the space of a ~week. That means there was data
movement into the first 2 new OSDs, then before that completed I added 2
new OSDs. So I would expect some data thashing on the first 2, but
nothing extreme.
The fragmentation scores for the 4 new OSDs are, respectively:
0.746, 0.835, 0.160, 0.067
That seems ridiculous for the first two, it's only been a week. The
newest two seem in better shape, though those mostly would've seen only
data moving in, not out. The rebalance isn't done yet, but it's almost
done and all 4 OSDs have a similar fullness level at this time.
Looking at alloc stats:
ceph-0) allocation stats probe 6: cnt: 2219302 frags: 2328003 size:
1238454677504
ceph-0) probe -1: 1848577, 1970325, 1022324588544
ceph-0) probe -2: 848301, 862622, 505329963008
ceph-0) probe -6: 2187448, 2187448, 1055241568256
ceph-0) probe -14: 0, 0, 0
ceph-0) probe -22: 0, 0, 0
ceph-1) allocation stats probe 6: cnt: 1882396 frags: 1947321 size:
1054829641728
ceph-1) probe -1: 2212293, 2345923, 1215418728448
ceph-1) probe -2: 1471623, 1525498, 826984652800
ceph-1) probe -6: 2095298, 2095298, 1000065933312
ceph-1) probe -14: 0, 0, 0
ceph-1) probe -22: 0, 0, 0
ceph-2) allocation stats probe 3: cnt: 2760200 frags: 2760200 size:
1554513903616
ceph-2) probe -1: 2584046, 2584046, 1498140393472
ceph-2) probe -3: 1696921, 1696921, 869424496640
ceph-2) probe -7: 0, 0, 0
ceph-2) probe -11: 0, 0, 0
ceph-2) probe -19: 0, 0, 0
ceph-3) allocation stats probe 3: cnt: 2544818 frags: 2544818 size:
1432225021952
ceph-3) probe -1: 2688015, 2688015, 1515260739584
ceph-3) probe -3: 1086875, 1086875, 622025424896
ceph-3) probe -7: 0, 0, 0
ceph-3) probe -11: 0, 0, 0
ceph-3) probe -19: 0, 0, 0
So OSDs 2 and 3 (the latest ones to be added, note that these 4 new OSDs
are 0-3 since those IDs were free) are in good shape, but 0 and 1 are
already suffering from at least some fragmentation of objects, which is
a bit worrying when they are only ~70% full right now and only a week old.
I did delete a couple million small objects during the rebalance to try
to reduce load (I had some nasty directories), but that was cumulatively
only about 60GB of data. So while that could explain a high frag score
if there are now a million little holes in the free space map of the
OSDs (how is it calculated?), it should not actually cause new data
moving in to end up fragmented since there should be plenty of
unfragmented free space going around still.
I am now restarting OSDs 0 and 1 to see whether that makes the frag
score go down over time. I will do further analysis later with the raw
bluestore free space map, since I still have a bunch of rebalancing and
moving data around planned (I'm moving my cluster to new machines).
On 26/05/2023 00.29, Igor Fedotov wrote:
Hi Hector,
I can advise two tools for further fragmentation analysis:
1) One might want to use ceph-bluestore-tool's free-dump command to get
a list of free chunks for an OSD and try to analyze whether it's really
highly fragmented and lacks long enough extents. free-dump just returns
a list of extents in json format, I can take a look to the output if
shared...
2) You might want to look for allocation probs in OSD logs and see how
fragmentation in allocated chunks has evolved.
E.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
probe -1: 35168547, 46401246, 1199516209152
probe -3: 27275094, 35681802, 200121712640
probe -5: 34847167, 52539758, 271272230912
probe -9: 44291522, 60025613, 523997483008
probe -17: 10646313, 10646313, 155178434560
The first probe refers to the last day while others match days (or
rather probes) -1, -3, -5, -9, -17
'cnt' column represents the amount of allocations performed in the
previous 24 hours and 'frags' one shows amount of fragments in the
resulted allocations. So significant mismatch between frags and cnt
might indicate some issues with high fragmentation indeed.
Apart from retrospective analysis you might also want how OSD behavior
changes after reboot - e.g. wouldn't rebooted OSD produce less
fragmentation... Which in turn might indicate some issues with BlueStore
allocator..
Just FYI: allocation probe printing interval is controlled by
bluestore_alloc_stats_dump_interval parameter.
Thanks,
Igor
On 24/05/2023 17:18, Hector Martin wrote:
- Hector
On 24/05/2023 22.07, Mark Nelson wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Yep, bluestore fragmentation is an issue.
It's sort of a natural result
of using copy-on-write and never implementing any kind of
defragmentation scheme. Adam and I have been talking about doing it
now, probably piggybacking on scrub or other operations that already
area reading all of the extents for an object anyway.
I wrote a very simply prototype for clone to speed up the rbd mirror use
case here:
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
Adam ended up going the extra mile and completely changed how shared
blobs works which probably eliminates the need to do defrag on clone
anymore from an rbd-mirror perspective, but I think we still need to
identify any times we are doing full object reads of fragmented objects
and consider defragmenting at that time. It might be clone, or scrub,
or other things, but the point is that if we are already doing most of
the work (seeks on HDD especially!) the extra cost of a large write to
clean it up isn't that bad, especially if we are doing it over the
course of months or years and can help keep freespace less fragmented.
Note that
my particular issue seemed to specifically be free space
fragmentation. I don't use RBD mirror and I would not *expect* most of
my cephfs use cases to lead to any weird cow/fragmentation issues with
objects other than those forced by the free space becoming fragmented
(unless there is some weird pathological use case I'm hitting). Most of
my write workloads are just copying files in bulk and incrementally
writing out files.
Would simply defragging objects during scrub/etc help with free space
fragmentation itself? Those seem like two somewhat unrelated issues...
note that if free space is already fragmented, you wouldn't even have a
place to put down a defragmented object.
Are there any stats I can look at to figure out how bad object and free
space fragmentation is? It would be nice to have some clearer data
beyond my hunch/deduction after seeing the I/O patterns and the sole
fragmentation number :). Also would be interesting to get some kind of
trace of the bluestore ops the OSD is doing, so I can find out whether
it's doing something pathological that causes more fragmentation for
some reason.
Mark
On 5/24/23 07:17, Hector Martin wrote:
> Hi,
>
> I've been seeing relatively large fragmentation numbers on all my OSDs:
>
> ceph daemon osd.13 bluestore allocator score block
> {
> "fragmentation_rating": 0.77251526920454427
> }
>
> These aren't that old, as I recreated them all around July last year.
> They mostly hold CephFS data with erasure coding, with a mix of large
> and small files. The OSDs are at around 80%-85% utilization right now.
> Most of the data was written sequentially when the OSDs were created (I
> rsynced everything from a remote backup). Since then more data has been
> added, but not particularly quickly.
>
> At some point I noticed pathologically slow writes, and I couldn't
> figure out what was wrong. Eventually I did some block tracing and
> noticed the I/Os were very small, even though CephFS-side I was just
> writing one large file sequentially, and that's when I stumbled upon the
> free space fragmentation problem. Indeed, deleting some large files
> opened up some larger free extents and resolved the problem, but only
> until those get filled up and I'm back to fragmented tiny extents. So
> effectively I'm stuck at the current utilization, as trying to fill them
> up any more just slows down to an absolute crawl.
>
> I'm adding a few more OSDs and plan on doing the dance of removing one
> OSD at a time and replacing it with another one to hopefully improve the
> situation, but obviously this is going to take forever.
>
> Is there any plan for offering a defrag tool of some sort for bluestore?
>
> - Hector
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
- Hector
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
29 May
29 May
8:26 p.m.
So fragmentation score calculation was improved recently indeed,
seehttps://github.com/ceph/ceph/pull/49885
And yeah one can see some fragmentation in allocations for the first two
OSDs. Doesn't look that dramatic as fragmentation scores tell though.
Additionally you might want to collect free extents dump using 'ceph
tell osd.N ceph bluestore allocator dump block' command and do more
analysis on these data.
E.g. I'd recommend to build something like a histogram showing amount of
chunks for specific size range:
[1-4K]: N1 chunks
(4K-16]: N2 chunks
(16K-64K): N3
...
[16M-inf) : Nn chunks
This should be even more informative about fragmentation state -
particularly if observed in evolution.
Looking for volunteers to write a script for building such a histogram... ;)
Thanks,
Igor
On 28/05/2023 08:31, Hector Martin wrote:
> So chiming in, I think something is definitely wrong with at *least* the
> frag score.
>
> Here's what happened so far:
>
> 1. I had 8 OSDs (all 8T HDDs)
> 2. I added 2 more (osd.0,1) , with Quincy defaults
> 3. I marked 2 old ones out (the ones that seemed to be struggling the
> most with IOPS)
> 4. I added 2 more (osd.2,3), but this time I had previously set
> bluestore_min_alloc_size_hdd to 16K as an experiment
>
> This has all happened in the space of a ~week. That means there was data
> movement into the first 2 new OSDs, then before that completed I added 2
> new OSDs. So I would expect some data thashing on the first 2, but
> nothing extreme.
>
> The fragmentation scores for the 4 new OSDs are, respectively:
>
> 0.746, 0.835, 0.160, 0.067
>
> That seems ridiculous for the first two, it's only been a week. The
> newest two seem in better shape, though those mostly would've seen only
> data moving in, not out. The rebalance isn't done yet, but it's almost
> done and all 4 OSDs have a similar fullness level at this time.
>
> Looking at alloc stats:
>
> ceph-0) allocation stats probe 6: cnt: 2219302 frags: 2328003 size:
> 1238454677504
> ceph-0) probe -1: 1848577, 1970325, 1022324588544
> ceph-0) probe -2: 848301, 862622, 505329963008
> ceph-0) probe -6: 2187448, 2187448, 1055241568256
> ceph-0) probe -14: 0, 0, 0
> ceph-0) probe -22: 0, 0, 0
>
> ceph-1) allocation stats probe 6: cnt: 1882396 frags: 1947321 size:
> 1054829641728
> ceph-1) probe -1: 2212293, 2345923, 1215418728448
> ceph-1) probe -2: 1471623, 1525498, 826984652800
> ceph-1) probe -6: 2095298, 2095298, 1000065933312
> ceph-1) probe -14: 0, 0, 0
> ceph-1) probe -22: 0, 0, 0
>
> ceph-2) allocation stats probe 3: cnt: 2760200 frags: 2760200 size:
> 1554513903616
> ceph-2) probe -1: 2584046, 2584046, 1498140393472
> ceph-2) probe -3: 1696921, 1696921, 869424496640
> ceph-2) probe -7: 0, 0, 0
> ceph-2) probe -11: 0, 0, 0
> ceph-2) probe -19: 0, 0, 0
>
> ceph-3) allocation stats probe 3: cnt: 2544818 frags: 2544818 size:
> 1432225021952
> ceph-3) probe -1: 2688015, 2688015, 1515260739584
> ceph-3) probe -3: 1086875, 1086875, 622025424896
> ceph-3) probe -7: 0, 0, 0
> ceph-3) probe -11: 0, 0, 0
> ceph-3) probe -19: 0, 0, 0
>
> So OSDs 2 and 3 (the latest ones to be added, note that these 4 new OSDs
> are 0-3 since those IDs were free) are in good shape, but 0 and 1 are
> already suffering from at least some fragmentation of objects, which is
> a bit worrying when they are only ~70% full right now and only a week old.
>
> I did delete a couple million small objects during the rebalance to try
> to reduce load (I had some nasty directories), but that was cumulatively
> only about 60GB of data. So while that could explain a high frag score
> if there are now a million little holes in the free space map of the
> OSDs (how is it calculated?), it should not actually cause new data
> moving in to end up fragmented since there should be plenty of
> unfragmented free space going around still.
>
> I am now restarting OSDs 0 and 1 to see whether that makes the frag
> score go down over time. I will do further analysis later with the raw
> bluestore free space map, since I still have a bunch of rebalancing and
> moving data around planned (I'm moving my cluster to new machines).
>
> On 26/05/2023 00.29, Igor Fedotov wrote:
>> Hi Hector,
>>
>> I can advise two tools for further fragmentation analysis:
>>
>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>> a list of free chunks for an OSD and try to analyze whether it's really
>> highly fragmented and lacks long enough extents. free-dump just returns
>> a list of extents in json format, I can take a look to the output if
>> shared...
>>
>> 2) You might want to look for allocation probs in OSD logs and see how
>> fragmentation in allocated chunks has evolved.
>>
>> E.g.
>>
>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>> probe -1: 35168547, 46401246, 1199516209152
>> probe -3: 27275094, 35681802, 200121712640
>> probe -5: 34847167, 52539758, 271272230912
>> probe -9: 44291522, 60025613, 523997483008
>> probe -17: 10646313, 10646313, 155178434560
>>
>> The first probe refers to the last day while others match days (or
>> rather probes) -1, -3, -5, -9, -17
>>
>> 'cnt' column represents the amount of allocations performed in the
>> previous 24 hours and 'frags' one shows amount of fragments in the
>> resulted allocations. So significant mismatch between frags and cnt
>> might indicate some issues with high fragmentation indeed.
>>
>> Apart from retrospective analysis you might also want how OSD behavior
>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>> fragmentation... Which in turn might indicate some issues with BlueStore
>> allocator..
>>
>> Just FYI: allocation probe printing interval is controlled by
>> bluestore_alloc_stats_dump_interval parameter.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>> On 24/05/2023 17:18, Hector Martin wrote:
>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural
result
>>>> of using copy-on-write and never implementing any kind of
>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>> now, probably piggybacking on scrub or other operations that already
>>>> area reading all of the extents for an object anyway.
>>>>
>>>>
>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>> case here:
>>>>
>>>>
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>>>
>>>>
>>>> Adam ended up going the extra mile and completely changed how shared
>>>> blobs works which probably eliminates the need to do defrag on clone
>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>> identify any times we are doing full object reads of fragmented objects
>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>> or other things, but the point is that if we are already doing most of
>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>> clean it up isn't that bad, especially if we are doing it over the
>>>> course of months or years and can help keep freespace less fragmented.
>>> Note that my particular issue seemed to specifically be free space
>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>> objects other than those forced by the free space becoming fragmented
>>> (unless there is some weird pathological use case I'm hitting). Most of
>>> my write workloads are just copying files in bulk and incrementally
>>> writing out files.
>>>
>>> Would simply defragging objects during scrub/etc help with free space
>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>> note that if free space is already fragmented, you wouldn't even have a
>>> place to put down a defragmented object.
>>>
>>> Are there any stats I can look at to figure out how bad object and free
>>> space fragmentation is? It would be nice to have some clearer data
>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>> fragmentation number :). Also would be interesting to get some kind of
>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>> it's doing something pathological that causes more fragmentation for
>>> some reason.
>>>
>>>> Mark
>>>>
>>>>
>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>> Hi,
>>>>>
>>>>> I've been seeing relatively large fragmentation numbers on all my
OSDs:
>>>>>
>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>> {
>>>>> "fragmentation_rating": 0.77251526920454427
>>>>> }
>>>>>
>>>>> These aren't that old, as I recreated them all around July last
year.
>>>>> They mostly hold CephFS data with erasure coding, with a mix of
large
>>>>> and small files. The OSDs are at around 80%-85% utilization right
now.
>>>>> Most of the data was written sequentially when the OSDs were created
(I
>>>>> rsynced everything from a remote backup). Since then more data has
been
>>>>> added, but not particularly quickly.
>>>>>
>>>>> At some point I noticed pathologically slow writes, and I
couldn't
>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>> writing one large file sequentially, and that's when I stumbled
upon the
>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>> opened up some larger free extents and resolved the problem, but
only
>>>>> until those get filled up and I'm back to fragmented tiny
extents. So
>>>>> effectively I'm stuck at the current utilization, as trying to
fill them
>>>>> up any more just slows down to an absolute crawl.
>>>>>
>>>>> I'm adding a few more OSDs and plan on doing the dance of
removing one
>>>>> OSD at a time and replacing it with another one to hopefully improve
the
>>>>> situation, but obviously this is going to take forever.
>>>>>
>>>>> Is there any plan for offering a defrag tool of some sort for
bluestore?
>>>>>
>>>>> - Hector
>>>>> _______________________________________________
>>>>> ceph-users mailing list --ceph-users(a)ceph.io
>>>>> To unsubscribe send an email toceph-users-leave(a)ceph.io
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list --ceph-users(a)ceph.io
>>> To unsubscribe send an email toceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list --ceph-users(a)ceph.io
>> To unsubscribe send an email toceph-users-leave(a)ceph.io
> - Hector
>
9:44 p.m.
On 29/05/2023 22.26, Igor Fedotov wrote:
So fragmentation score calculation was improved
recently indeed, see
https://github.com/ceph/ceph/pull/49885
And yeah one can see some fragmentation in allocations for the first two
OSDs. Doesn't look that dramatic as fragmentation scores tell though.
Additionally you might want to collect free extents dump using 'ceph
tell osd.N ceph bluestore allocator dump block' command and do more
analysis on these data.
E.g. I'd recommend to build something like a histogram showing amount of
chunks for specific size range:
[1-4K]: N1 chunks
(4K-16]: N2 chunks
(16K-64K): N3
...
[16M-inf) : Nn chunks
This should be even more informative about fragmentation state -
particularly if observed in evolution.
Looking for volunteers to write a script for building such a histogram... ;)
I'm up for that, once I get through some other cluster maintenance I
need to deal with first :)
Backfill is almost done and I was finally able to destroy two OSDs, will
be doing a bunch of restructuring in the coming weeks. I can probably
get the script done partway through doing this, so I can see how the
distributions evolve over a bunch of data movement.
Thanks,
Igor
On 28/05/2023 08:31, Hector Martin wrote:
> So chiming in, I think something is definitely wrong with at *least* the
> frag score.
>
> Here's what happened so far:
>
> 1. I had 8 OSDs (all 8T HDDs)
> 2. I added 2 more (osd.0,1) , with Quincy defaults
> 3. I marked 2 old ones out (the ones that seemed to be struggling the
> most with IOPS)
> 4. I added 2 more (osd.2,3), but this time I had previously set
> bluestore_min_alloc_size_hdd to 16K as an experiment
>
> This has all happened in the space of a ~week. That means there was data
> movement into the first 2 new OSDs, then before that completed I added 2
> new OSDs. So I would expect some data thashing on the first 2, but
> nothing extreme.
>
> The fragmentation scores for the 4 new OSDs are, respectively:
>
> 0.746, 0.835, 0.160, 0.067
>
> That seems ridiculous for the first two, it's only been a week. The
> newest two seem in better shape, though those mostly would've seen only
> data moving in, not out. The rebalance isn't done yet, but it's almost
> done and all 4 OSDs have a similar fullness level at this time.
>
> Looking at alloc stats:
>
> ceph-0) allocation stats probe 6: cnt: 2219302 frags: 2328003 size:
> 1238454677504
> ceph-0) probe -1: 1848577, 1970325, 1022324588544
> ceph-0) probe -2: 848301, 862622, 505329963008
> ceph-0) probe -6: 2187448, 2187448, 1055241568256
> ceph-0) probe -14: 0, 0, 0
> ceph-0) probe -22: 0, 0, 0
>
> ceph-1) allocation stats probe 6: cnt: 1882396 frags: 1947321 size:
> 1054829641728
> ceph-1) probe -1: 2212293, 2345923, 1215418728448
> ceph-1) probe -2: 1471623, 1525498, 826984652800
> ceph-1) probe -6: 2095298, 2095298, 1000065933312
> ceph-1) probe -14: 0, 0, 0
> ceph-1) probe -22: 0, 0, 0
>
> ceph-2) allocation stats probe 3: cnt: 2760200 frags: 2760200 size:
> 1554513903616
> ceph-2) probe -1: 2584046, 2584046, 1498140393472
> ceph-2) probe -3: 1696921, 1696921, 869424496640
> ceph-2) probe -7: 0, 0, 0
> ceph-2) probe -11: 0, 0, 0
> ceph-2) probe -19: 0, 0, 0
>
> ceph-3) allocation stats probe 3: cnt: 2544818 frags: 2544818 size:
> 1432225021952
> ceph-3) probe -1: 2688015, 2688015, 1515260739584
> ceph-3) probe -3: 1086875, 1086875, 622025424896
> ceph-3) probe -7: 0, 0, 0
> ceph-3) probe -11: 0, 0, 0
> ceph-3) probe -19: 0, 0, 0
>
> So OSDs 2 and 3 (the latest ones to be added, note that these 4 new OSDs
> are 0-3 since those IDs were free) are in good shape, but 0 and 1 are
> already suffering from at least some fragmentation of objects, which is
> a bit worrying when they are only ~70% full right now and only a week old.
>
> I did delete a couple million small objects during the rebalance to try
> to reduce load (I had some nasty directories), but that was cumulatively
> only about 60GB of data. So while that could explain a high frag score
> if there are now a million little holes in the free space map of the
> OSDs (how is it calculated?), it should not actually cause new data
> moving in to end up fragmented since there should be plenty of
> unfragmented free space going around still.
>
> I am now restarting OSDs 0 and 1 to see whether that makes the frag
> score go down over time. I will do further analysis later with the raw
> bluestore free space map, since I still have a bunch of rebalancing and
> moving data around planned (I'm moving my cluster to new machines).
>
> On 26/05/2023 00.29, Igor Fedotov wrote:
>> Hi Hector,
>>
>> I can advise two tools for further fragmentation analysis:
>>
>> 1) One might want to use ceph-bluestore-tool's free-dump command to get
>> a list of free chunks for an OSD and try to analyze whether it's really
>> highly fragmented and lacks long enough extents. free-dump just returns
>> a list of extents in json format, I can take a look to the output if
>> shared...
>>
>> 2) You might want to look for allocation probs in OSD logs and see how
>> fragmentation in allocated chunks has evolved.
>>
>> E.g.
>>
>> allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
>> probe -1: 35168547, 46401246, 1199516209152
>> probe -3: 27275094, 35681802, 200121712640
>> probe -5: 34847167, 52539758, 271272230912
>> probe -9: 44291522, 60025613, 523997483008
>> probe -17: 10646313, 10646313, 155178434560
>>
>> The first probe refers to the last day while others match days (or
>> rather probes) -1, -3, -5, -9, -17
>>
>> 'cnt' column represents the amount of allocations performed in the
>> previous 24 hours and 'frags' one shows amount of fragments in the
>> resulted allocations. So significant mismatch between frags and cnt
>> might indicate some issues with high fragmentation indeed.
>>
>> Apart from retrospective analysis you might also want how OSD behavior
>> changes after reboot - e.g. wouldn't rebooted OSD produce less
>> fragmentation... Which in turn might indicate some issues with BlueStore
>> allocator..
>>
>> Just FYI: allocation probe printing interval is controlled by
>> bluestore_alloc_stats_dump_interval parameter.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>> On 24/05/2023 17:18, Hector Martin wrote:
>>> On 24/05/2023 22.07, Mark Nelson wrote:
>>>> Yep, bluestore fragmentation is an issue. It's sort of a natural
result
>>>> of using copy-on-write and never implementing any kind of
>>>> defragmentation scheme. Adam and I have been talking about doing it
>>>> now, probably piggybacking on scrub or other operations that already
>>>> area reading all of the extents for an object anyway.
>>>>
>>>>
>>>> I wrote a very simply prototype for clone to speed up the rbd mirror use
>>>> case here:
>>>>
>>>>
https://github.com/markhpc/ceph/commit/29fc1bfd4c90dd618eb9e0d4ae6474d8cfa5…
>>>>
>>>>
>>>> Adam ended up going the extra mile and completely changed how shared
>>>> blobs works which probably eliminates the need to do defrag on clone
>>>> anymore from an rbd-mirror perspective, but I think we still need to
>>>> identify any times we are doing full object reads of fragmented objects
>>>> and consider defragmenting at that time. It might be clone, or scrub,
>>>> or other things, but the point is that if we are already doing most of
>>>> the work (seeks on HDD especially!) the extra cost of a large write to
>>>> clean it up isn't that bad, especially if we are doing it over the
>>>> course of months or years and can help keep freespace less fragmented.
>>> Note that my particular issue seemed to specifically be free space
>>> fragmentation. I don't use RBD mirror and I would not *expect* most of
>>> my cephfs use cases to lead to any weird cow/fragmentation issues with
>>> objects other than those forced by the free space becoming fragmented
>>> (unless there is some weird pathological use case I'm hitting). Most of
>>> my write workloads are just copying files in bulk and incrementally
>>> writing out files.
>>>
>>> Would simply defragging objects during scrub/etc help with free space
>>> fragmentation itself? Those seem like two somewhat unrelated issues...
>>> note that if free space is already fragmented, you wouldn't even have a
>>> place to put down a defragmented object.
>>>
>>> Are there any stats I can look at to figure out how bad object and free
>>> space fragmentation is? It would be nice to have some clearer data
>>> beyond my hunch/deduction after seeing the I/O patterns and the sole
>>> fragmentation number :). Also would be interesting to get some kind of
>>> trace of the bluestore ops the OSD is doing, so I can find out whether
>>> it's doing something pathological that causes more fragmentation for
>>> some reason.
>>>
>>>> Mark
>>>>
>>>>
>>>> On 5/24/23 07:17, Hector Martin wrote:
>>>>> Hi,
>>>>>
>>>>> I've been seeing relatively large fragmentation numbers on all my
OSDs:
>>>>>
>>>>> ceph daemon osd.13 bluestore allocator score block
>>>>> {
>>>>> "fragmentation_rating": 0.77251526920454427
>>>>> }
>>>>>
>>>>> These aren't that old, as I recreated them all around July last
year.
>>>>> They mostly hold CephFS data with erasure coding, with a mix of
large
>>>>> and small files. The OSDs are at around 80%-85% utilization right
now.
>>>>> Most of the data was written sequentially when the OSDs were created
(I
>>>>> rsynced everything from a remote backup). Since then more data has
been
>>>>> added, but not particularly quickly.
>>>>>
>>>>> At some point I noticed pathologically slow writes, and I
couldn't
>>>>> figure out what was wrong. Eventually I did some block tracing and
>>>>> noticed the I/Os were very small, even though CephFS-side I was just
>>>>> writing one large file sequentially, and that's when I stumbled
upon the
>>>>> free space fragmentation problem. Indeed, deleting some large files
>>>>> opened up some larger free extents and resolved the problem, but
only
>>>>> until those get filled up and I'm back to fragmented tiny
extents. So
>>>>> effectively I'm stuck at the current utilization, as trying to
fill them
>>>>> up any more just slows down to an absolute crawl.
>>>>>
>>>>> I'm adding a few more OSDs and plan on doing the dance of
removing one
>>>>> OSD at a time and replacing it with another one to hopefully improve
the
>>>>> situation, but obviously this is going to take forever.
>>>>>
>>>>> Is there any plan for offering a defrag tool of some sort for
bluestore?
>>>>>
>>>>> - Hector
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> - Hector
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> - Hector
>
- Hector
24 May
24 May
11:40 p.m.
Hi Hector,
Not related to fragmentation. But I see you mentioned CephFS, and your OSDs are at high
utilization. Is your pool NEAR FULL? CephFS write performance is severely degraded if the
pool is NEAR FULL. Buffered write will be disabled, and every single write() system call
needs to wait for reply from OSD.
If this is the case, use “ceph osd set-nearfull-ratio” to get normal performance.
Weiwen Hu
在 2023年5月24日，20:19，Hector Martin
<marcan(a)marcan.st> 写道：
Hi,
I've been seeing relatively large fragmentation numbers on all my OSDs:
ceph daemon osd.13 bluestore allocator score block
{
"fragmentation_rating": 0.77251526920454427
}
These aren't that old, as I recreated them all around July last year.
They mostly hold CephFS data with erasure coding, with a mix of large
and small files. The OSDs are at around 80%-85% utilization right now.
Most of the data was written sequentially when the OSDs were created (I
rsynced everything from a remote backup). Since then more data has been
added, but not particularly quickly.
At some point I noticed pathologically slow writes, and I couldn't
figure out what was wrong. Eventually I did some block tracing and
noticed the I/Os were very small, even though CephFS-side I was just
writing one large file sequentially, and that's when I stumbled upon the
free space fragmentation problem. Indeed, deleting some large files
opened up some larger free extents and resolved the problem, but only
until those get filled up and I'm back to fragmented tiny extents. So
effectively I'm stuck at the current utilization, as trying to fill them
up any more just slows down to an absolute crawl.
I'm adding a few more OSDs and plan on doing the dance of removing one
OSD at a time and replacing it with another one to hopefully improve the
situation, but obviously this is going to take forever.
Is there any plan for offering a defrag tool of some sort for bluestore?
- Hector
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 May
25 May
8:46 a.m.
On 25/05/2023 01.40, 胡 玮文 wrote:
Hi Hector,
Not related to fragmentation. But I see you mentioned CephFS, and your OSDs are at high
utilization. Is your pool NEAR FULL? CephFS write performance is severely degraded if the
pool is NEAR FULL. Buffered write will be disabled, and every single write() system call
needs to wait for reply from OSD.
If this is the case, use “ceph osd set-nearfull-ratio” to get normal performance.
I learned about this after the issue; they did become nearfull at one
point and I changed the threshold, but I don't think this explains the
behavior I was seeing because I was trying to do bulk writes (which
should use very large write sizes even without buffering). What happened
was usually a single OSD would immediately go to 100% utilization, but
not the rest, which is what I'd expect if that one OSD was the one with
the most fragmented free space ending up pathologically slowing down writes.
- Hector
329
days inactive
336
days old
27 comments
7 participants
participants (7)
-
Fox, Kevin M -
Hector Martin -
Igor Fedotov -
Konstantin Shalygin -
Mark Nelson -
Stefan Kooman -
胡 玮文