6 Oct
2020
6 Oct
'20
10:09 a.m.
Hi all,
We have a Ceph cluster which has been expanded from 10 to 16 nodes.
Each node has between 14 and 16 OSDs of which 2 are NVMe disks.
Most disks (except NVMe's) are 16TB large.
The expansion of 16 nodes went ok, but we've configured the system to
prevent auto balance towards the new disks (weight was set to 0) so we
could control the expansion.
We started adding 6 disks last week (1 disk on each new node) which didn't
give a lot of issues.
When the Ceph status indicated the PG degraded was almost finished, we've
added 2 disks on each node again.
All seemed to go fine, till yesterday morning... IOs towards the system
were slowing down.
Diving onto the nodes we could see that the OSD daemons are consuming the
CPU power, resulting in average CPU loads going near 10 (!).
The RGWs nor monitors nor other involved servers are having CPU issues
(except for the management server which is fighting with Prometheus), so
it's latency seems to be related to the ODS hosts.
All of the hosts are interconnected with 25Gbit connections, no bottlenecks
are reached on the network either.
Important piece of information: We are using erasure coding (6/3), and we
do have a lot of small files...
The current health detail indicates degraded health redundancy where
1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized).
Diving into the historic ops of an OSD we can see that the main latency is
found between the event "queued_for_pg" and "reached_pg". (Averaging
+/- 3
secs)
As the system load is quite high I assume the systems are busy
recalculating the code chunks for using the new disks we've added (though
not sure), but I was wondering how I can better fine tune the system or
pinpoint the exact bottle neck.
Latency towards the disks doesn't seem an issue at first sight...
We are running Ceph 14.2.11
Who can give me some thoughts on how I can better pinpoint the bottle neck?
Thanks
Kristof
6 Oct
6 Oct
11:23 a.m.
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones)
utilization along with slow ops?
Thanks,
Igor
On 10/6/2020 11:09 AM, Kristof Coucke wrote:
> Hi all,
>
> We have a Ceph cluster which has been expanded from 10 to 16 nodes.
> Each node has between 14 and 16 OSDs of which 2 are NVMe disks.
> Most disks (except NVMe's) are 16TB large.
>
> The expansion of 16 nodes went ok, but we've configured the system to
> prevent auto balance towards the new disks (weight was set to 0) so we
> could control the expansion.
>
> We started adding 6 disks last week (1 disk on each new node) which didn't
> give a lot of issues.
> When the Ceph status indicated the PG degraded was almost finished, we've
> added 2 disks on each node again.
>
> All seemed to go fine, till yesterday morning... IOs towards the system
> were slowing down.
>
> Diving onto the nodes we could see that the OSD daemons are consuming the
> CPU power, resulting in average CPU loads going near 10 (!).
>
> The RGWs nor monitors nor other involved servers are having CPU issues
> (except for the management server which is fighting with Prometheus), so
> it's latency seems to be related to the ODS hosts.
> All of the hosts are interconnected with 25Gbit connections, no bottlenecks
> are reached on the network either.
>
> Important piece of information: We are using erasure coding (6/3), and we
> do have a lot of small files...
> The current health detail indicates degraded health redundancy where
> 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized).
>
> Diving into the historic ops of an OSD we can see that the main latency is
> found between the event "queued_for_pg" and "reached_pg".
(Averaging +/- 3
> secs)
>
> As the system load is quite high I assume the systems are busy
> recalculating the code chunks for using the new disks we've added (though
> not sure), but I was wondering how I can better fine tune the system or
> pinpoint the exact bottle neck.
> Latency towards the disks doesn't seem an issue at first sight...
>
> We are running Ceph 14.2.11
>
> Who can give me some thoughts on how I can better pinpoint the bottle neck?
>
> Thanks
>
> Kristof
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:41 a.m.
Yes, some disks are spiking near 100%... The delay I see with the iostat
(r_await) seems to be synchronised with the delays between queued_for_pg
and reached_pg events.
The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is also
128kb (os level) (for spinner disks). As we are dealing with smaller files,
this might also lead to a decrease of the performance.
I'm still investigating, but I'm wondering if the system is also reading
from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov(a)suse.de>de>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones)
utilization along with slow ops?
Thanks,
Igor
On 10/6/2020 11:09 AM, Kristof Coucke wrote:
Hi all,
We have a Ceph cluster which has been expanded from 10 to 16 nodes.
Each node has between 14 and 16 OSDs of which 2 are NVMe disks.
Most disks (except NVMe's) are 16TB large.
The expansion of 16 nodes went ok, but we've configured the system to
prevent auto balance towards the new disks (weight was set to 0) so we
could control the expansion.
We started adding 6 disks last week (1 disk on each new node) which
didn't
give a lot of issues.
When the Ceph status indicated the PG degraded was almost finished, we've
added 2 disks on each node again.
All seemed to go fine, till yesterday morning... IOs towards the system
were slowing down.
Diving onto the nodes we could see that the OSD daemons are consuming the
CPU power, resulting in average CPU loads going near 10 (!).
The RGWs nor monitors nor other involved servers are having CPU issues
(except for the management server which is fighting with Prometheus), so
it's latency seems to be related to the ODS hosts.
All of the hosts are interconnected with 25Gbit connections, no
bottlenecks
are reached on the network either.
Important piece of information: We are using erasure coding (6/3), and we
do have a lot of small files...
The current health detail indicates degraded health redundancy where
1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg
undersized).
Diving into the historic ops of an OSD we can see that the main latency
is
found between the event "queued_for_pg"
and "reached_pg". (Averaging +/-
3
secs)
As the system load is quite high I assume the systems are busy
recalculating the code chunks for using the new disks we've added (though
not sure), but I was wondering how I can better fine tune the system or
pinpoint the exact bottle neck.
Latency towards the disks doesn't seem an issue at first sight...
We are running Ceph 14.2.11
Who can give me some thoughts on how I can better pinpoint the bottle
neck?
Thanks
Kristof
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:04 p.m.
Another strange thing is going on:
No client software is using the system any longer, so we would expect that
all IOs are related to the recovery (fixing of the degraded PG).
However, the disks that are reaching high IO are not a member of the PGs
that are being fixed.
So, something is heavily using the disk, but I can't find the process
immediately. I've read something that there can be old client processes
that keep on connecting to an OSD for retrieving data for a specific PG
while that PG is no longer available on that disk.
Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke <kristof.coucke(a)gmail.com
:
> Yes, some disks are spiking near 100%... The delay I see with the iostat
> (r_await) seems to be synchronised with the delays between queued_for_pg
> and reached_pg events.
> The NVMe disks are not spiking, just the spinner disks.
>
> I know the rocksdb is only partial on the NVMe. The read-ahead is also
> 128kb (os level) (for spinner disks). As we are dealing with smaller files,
> this might also lead to a decrease of the performance.
>
> I'm still investigating, but I'm wondering if the system is also reading
> from disk for finding the KV pairs.
>
>
>
> Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov(a)suse.de :
>
>> Hi Kristof,
>>
>> are you seeing high (around 100%) OSDs' disks (main or DB ones)
>> utilization along with slow ops?
>>
>>
>> Thanks,
>>
>> Igor
>>
>> On 10/6/2020 11:09 AM, Kristof Coucke wrote:
>> > Hi all,
>> >
>> > We have a Ceph cluster which has been expanded from 10 to 16 nodes.
>> > Each node has between 14 and 16 OSDs of which 2 are NVMe disks.
>> > Most disks (except NVMe's) are 16TB large.
>> >
>> > The expansion of 16 nodes went ok, but we've configured the system to
>> > prevent auto balance towards the new disks (weight was set to 0) so we
>> > could control the expansion.
>> >
>> > We started adding 6 disks last week (1 disk on each new node) which
>> didn't
>> > give a lot of issues.
>> > When the Ceph status indicated the PG degraded was almost finished,
>> we've
>> > added 2 disks on each node again.
>> >
>> > All seemed to go fine, till yesterday morning... IOs towards the system
>> > were slowing down.
>> >
>> > Diving onto the nodes we could see that the OSD daemons are consuming
>> the
>> > CPU power, resulting in average CPU loads going near 10 (!).
>> >
>> > The RGWs nor monitors nor other involved servers are having CPU issues
>> > (except for the management server which is fighting with Prometheus), so
>> > it's latency seems to be related to the ODS hosts.
>> > All of the hosts are interconnected with 25Gbit connections, no
>> bottlenecks
>> > are reached on the network either.
>> >
>> > Important piece of information: We are using erasure coding (6/3), and
>> we
>> > do have a lot of small files...
>> > The current health detail indicates degraded health redundancy where
>> > 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg
>> undersized).
>> >
>> > Diving into the historic ops of an OSD we can see that the main latency
>> is
>> > found between the event "queued_for_pg" and
"reached_pg". (Averaging
>> +/- 3
>> > secs)
>> >
>> > As the system load is quite high I assume the systems are busy
>> > recalculating the code chunks for using the new disks we've added
>> (though
>> > not sure), but I was wondering how I can better fine tune the system or
>> > pinpoint the exact bottle neck.
>> > Latency towards the disks doesn't seem an issue at first sight...
>> >
>> > We are running Ceph 14.2.11
>> >
>> > Who can give me some thoughts on how I can better pinpoint the bottle
>> neck?
>> >
>> > Thanks
>> >
>> > Kristof
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users(a)ceph.io
>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>
1:05 p.m.
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect
that all IOs are related to the recovery (fixing of the degraded PG).
However, the disks that are reaching high IO are not a member of the
PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process
immediately. I've read something that there can be old client
processes that keep on connecting to an OSD for retrieving data for a
specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke
<kristof.coucke(a)gmail.com <mailto:kristof.coucke@gmail.com>>:
Yes, some disks are spiking near 100%... The delay I see with the
iostat (r_await) seems to be synchronised with the delays between
queued_for_pg and reached_pg events.
The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is
also 128kb (os level) (for spinner disks). As we are dealing with
smaller files, this might also lead to a decrease of the performance.
I'm still investigating, but I'm wondering if the system is also
reading from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov(a)suse.de
<mailto:ifedotov@suse.de>>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones)
utilization along with slow ops?
Thanks,
Igor
On 10/6/2020 11:09 AM, Kristof Coucke wrote:
Hi all,
We have a Ceph cluster which has been expanded from 10 to 16
nodes.
Each node has between 14 and 16 OSDs of which 2
are NVMe disks.
Most disks (except NVMe's) are 16TB large.
The expansion of 16 nodes went ok, but we've configured the
system to
prevent auto balance towards the new disks
(weight was set
to 0) so we
could control the expansion.
We started adding 6 disks last week (1 disk on each new
node) which
didn't
give a lot of issues.
When the Ceph status indicated the PG degraded was almost
finished,
we've
added 2 disks on each node again.
All seemed to go fine, till yesterday morning... IOs towards
the system
were slowing down.
Diving onto the nodes we could see that the OSD daemons are
consuming the
CPU power, resulting in average CPU loads going
near 10 (!).
The RGWs nor monitors nor other involved servers are having
CPU issues
(except for the management server which is
fighting with
Prometheus), so
it's latency seems to be related to the ODS
hosts.
All of the hosts are interconnected with 25Gbit connections,
no
bottlenecks
are reached on the network either.
Important piece of information: We are using erasure coding
(6/3), and we
do have a lot of small files...
The current health detail indicates degraded health
redundancy where
1192911/103387889228 objects are degraded. (1 pg
degraded, 1
pg undersized).
Diving into the historic ops of an OSD we can see that the
main latency
is
found between the event "queued_for_pg"
and "reached_pg".
(Averaging +/- 3
secs)
As the system load is quite high I assume the systems are busy
recalculating the code chunks for using the new disks we've
added
(though
not sure), but I was wondering how I can better
fine tune
the system or
pinpoint the exact bottle neck.
Latency towards the disks doesn't seem an issue at first
sight...
We are running Ceph 14.2.11
Who can give me some thoughts on how I can better pinpoint
the bottle
neck?
Thanks
Kristof
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
1:06 p.m.
Is there a way that I can check if this process is causing performance
issues?
Can I check somehow if this process is causing the issue?
Op di 6 okt. 2020 om 13:05 schreef Igor Fedotov <ifedotov(a)suse.de>de>:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect that
all IOs are related to the recovery (fixing of the degraded PG).
However, the disks that are reaching high IO are not a member of the PGs
that are being fixed.
So, something is heavily using the disk, but I can't find the process
immediately. I've read something that there can be old client processes
that keep on connecting to an OSD for retrieving data for a specific PG
while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke <
kristof.coucke(a)gmail.com>gt;:
> Yes, some disks are spiking near 100%... The delay I see with the iostat
> (r_await) seems to be synchronised with the delays between queued_for_pg
> and reached_pg events.
> The NVMe disks are not spiking, just the spinner disks.
>
> I know the rocksdb is only partial on the NVMe. The read-ahead is also
> 128kb (os level) (for spinner disks). As we are dealing with smaller files,
> this might also lead to a decrease of the performance.
>
> I'm still investigating, but I'm wondering if the system is also reading
> from disk for finding the KV pairs.
>
>
>
> Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov(a)suse.de>de>:
>
>> Hi Kristof,
>>
>> are you seeing high (around 100%) OSDs' disks (main or DB ones)
>> utilization along with slow ops?
>>
>>
>> Thanks,
>>
>> Igor
>>
>> On 10/6/2020 11:09 AM, Kristof Coucke wrote:
>> > Hi all,
>> >
>> > We have a Ceph cluster which has been expanded from 10 to 16 nodes.
>> > Each node has between 14 and 16 OSDs of which 2 are NVMe disks.
>> > Most disks (except NVMe's) are 16TB large.
>> >
>> > The expansion of 16 nodes went ok, but we've configured the system to
>> > prevent auto balance towards the new disks (weight was set to 0) so we
>> > could control the expansion.
>> >
>> > We started adding 6 disks last week (1 disk on each new node) which
>> didn't
>> > give a lot of issues.
>> > When the Ceph status indicated the PG degraded was almost finished,
>> we've
>> > added 2 disks on each node again.
>> >
>> > All seemed to go fine, till yesterday morning... IOs towards the system
>> > were slowing down.
>> >
>> > Diving onto the nodes we could see that the OSD daemons are consuming
>> the
>> > CPU power, resulting in average CPU loads going near 10 (!).
>> >
>> > The RGWs nor monitors nor other involved servers are having CPU issues
>> > (except for the management server which is fighting with Prometheus),
>> so
>> > it's latency seems to be related to the ODS hosts.
>> > All of the hosts are interconnected with 25Gbit connections, no
>> bottlenecks
>> > are reached on the network either.
>> >
>> > Important piece of information: We are using erasure coding (6/3), and
>> we
>> > do have a lot of small files...
>> > The current health detail indicates degraded health redundancy where
>> > 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg
>> undersized).
>> >
>> > Diving into the historic ops of an OSD we can see that the main
>> latency is
>> > found between the event "queued_for_pg" and
"reached_pg". (Averaging
>> +/- 3
>> > secs)
>> >
>> > As the system load is quite high I assume the systems are busy
>> > recalculating the code chunks for using the new disks we've added
>> (though
>> > not sure), but I was wondering how I can better fine tune the system or
>> > pinpoint the exact bottle neck.
>> > Latency towards the disks doesn't seem an issue at first sight...
>> >
>> > We are running Ceph 14.2.11
>> >
>> > Who can give me some thoughts on how I can better pinpoint the bottle
>> neck?
>> >
>> > Thanks
>> >
>> > Kristof
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users(a)ceph.io
>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>
1:16 p.m.
Unfortunately currently available Ceph releases lack any means to
monitor KV data removal. The only way is to set debug_bluestore to 20
(for a short period of time, e.g. 1 min) and inspect OSD log for
_remove/_do_remove/_omap_clear calls. Plenty of them within the
inspected period means ongoing removals.
A weak proof of the hypothesis would be having non-zero "numpg_removing"
performance counter...
On 10/6/2020 2:06 PM, Kristof Coucke wrote:
Is there a way that I can check if this process is
causing performance
issues?
Can I check somehow if this process is causing the issue?
Op di 6 okt. 2020 om 13:05 schreef Igor Fedotov <ifedotov(a)suse.de
<mailto:ifedotov@suse.de>>:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would
expect that all IOs are related to the recovery (fixing of the
degraded PG).
However, the disks that are reaching high IO are not a member of
the PGs that are being fixed.
So, something is heavily using the disk, but I can't find the
process immediately. I've read something that there can be old
client processes that keep on connecting to an OSD for retrieving
data for a specific PG while that PG is no longer available on
that disk.
I bet it's rather PG removal happening in background....
> Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke
> <kristof.coucke(a)gmail.com <mailto:kristof.coucke@gmail.com>>:
>
> Yes, some disks are spiking near 100%... The delay I see with
> the iostat (r_await) seems to be synchronised with the delays
> between queued_for_pg and reached_pg events.
> The NVMe disks are not spiking, just the spinner disks.
>
> I know the rocksdb is only partial on the NVMe. The
> read-ahead is also 128kb (os level) (for spinner disks). As
> we are dealing with smaller files, this might also lead to a
> decrease of the performance.
>
> I'm still investigating, but I'm wondering if the system is
> also reading from disk for finding the KV pairs.
>
>
>
> Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov
> <ifedotov(a)suse.de <mailto:ifedotov@suse.de>>:
>
> Hi Kristof,
>
> are you seeing high (around 100%) OSDs' disks (main or DB
> ones)
> utilization along with slow ops?
>
>
> Thanks,
>
> Igor
>
> On 10/6/2020 11:09 AM, Kristof Coucke wrote:
> > Hi all,
> >
> > We have a Ceph cluster which has been expanded from 10
> to 16 nodes.
> > Each node has between 14 and 16 OSDs of which 2 are
> NVMe disks.
> > Most disks (except NVMe's) are 16TB large.
> >
> > The expansion of 16 nodes went ok, but we've configured
> the system to
> > prevent auto balance towards the new disks (weight was
> set to 0) so we
> > could control the expansion.
> >
> > We started adding 6 disks last week (1 disk on each new
> node) which didn't
> > give a lot of issues.
> > When the Ceph status indicated the PG degraded was
> almost finished, we've
> > added 2 disks on each node again.
> >
> > All seemed to go fine, till yesterday morning... IOs
> towards the system
> > were slowing down.
> >
> > Diving onto the nodes we could see that the OSD daemons
> are consuming the
> > CPU power, resulting in average CPU loads going near 10
> (!).
> >
> > The RGWs nor monitors nor other involved servers are
> having CPU issues
> > (except for the management server which is fighting
> with Prometheus), so
> > it's latency seems to be related to the ODS hosts.
> > All of the hosts are interconnected with 25Gbit
> connections, no bottlenecks
> > are reached on the network either.
> >
> > Important piece of information: We are using erasure
> coding (6/3), and we
> > do have a lot of small files...
> > The current health detail indicates degraded health
> redundancy where
> > 1192911/103387889228 objects are degraded. (1 pg
> degraded, 1 pg undersized).
> >
> > Diving into the historic ops of an OSD we can see that
> the main latency is
> > found between the event "queued_for_pg" and
> "reached_pg". (Averaging +/- 3
> > secs)
> >
> > As the system load is quite high I assume the systems
> are busy
> > recalculating the code chunks for using the new disks
> we've added (though
> > not sure), but I was wondering how I can better fine
> tune the system or
> > pinpoint the exact bottle neck.
> > Latency towards the disks doesn't seem an issue at
> first sight...
> >
> > We are running Ceph 14.2.11
> >
> > Who can give me some thoughts on how I can better
> pinpoint the bottle neck?
> >
> > Thanks
> >
> > Kristof
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> <mailto:ceph-users@ceph.io>
> > To unsubscribe send an email to
> ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>
> 
1:18 p.m.
On 2020-10-06 13:05, Igor Fedotov wrote:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
^^ This, and probably the accompanying RocksDB housekeeping that goes
with it. As only removing PGs shouldn't be a too big a deal at all.
Especially with very small files (and a lot of them) you probably have a
lot of OMAP / META data, (ceph osd df will tell you).
If that's indeed the case than there is a (way) quicker option to get
out of this situation: offline compacting of the OSDs. This process
happens orders of magnitude faster than when the OSDs are still online.
To check if this hypothesis is true: are the OSD servers under CPU
stress where the PGs were located previously (and not the new hosts)?
Offline compaction per host:
systemctl stop ceph-osd.target
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv
/var/lib/ceph/osd/$osd compact &);done
Gr. Stefan
Another strange thing is going on:
No client software is using the system any longer, so we would expect
that all IOs are related to the recovery (fixing of the degraded PG).
However, the disks that are reaching high IO are not a member of the
PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process
immediately. I've read something that there can be old client
processes that keep on connecting to an OSD for retrieving data for a
specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
2:18 p.m.
Ok, I did the compact on 1 osd.
The utilization is back to normal, so that's good... Thumbs up to you guys!
Though, one thing I want to get out of the way before adapting the other
OSDs:
When I now get the RocksDb stats, my L1, L2 and L3 are gone:
db_statistics {
"rocksdb_compaction_statistics": "",
"": "",
"": "** Compaction Stats [default] **",
"": "Level Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"----------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0
0.0
0.0 0.0 0.0 0.0 0.0 0.00 0.00
0 0.000 0 0",
"": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0
0.0
0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0
0.0
0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": "",
"": "** Compaction Stats [default] **",
"": "Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0
0.0
0.0 0.0 0.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Flush(GB): cumulative 0.001, interval 0.001",
"": "AddFile(GB): cumulative 0.000, interval 0.000",
"": "AddFile(Total Files): cumulative 0, interval 0",
"": "AddFile(L0 Files): cumulative 0, interval 0",
"": "AddFile(Keys): cumulative 0, interval 0",
"": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB
read, 0.00 MB/s read, 0.0 seconds",
"": "Interval compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB
read,
0.00 MB/s read, 0.0 seconds",
"": "Stalls(count): 0 level0_slowdown, 0
level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count",
"": "",
"": "** File Read Latency Histogram By Level [default] **",
"": "** Level 0 read latency histogram (micros):",
"": "Count: 5 Average: 69.2000 StdDev: 85.92",
"": "Min: 0 Median: 1.5000 Max: 201",
"": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9: 201.00
P99.99: 201.00",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 2 40.000% 40.000% ########",
"": "( 1, 2 ] 1 20.000% 60.000% ####",
"": "( 110, 170 ] 1 20.000% 80.000% ####",
"": "( 170, 250 ] 1 20.000% 100.000% ####",
"": "",
"": "** Level 4 read latency histogram (micros):",
"": "Count: 4664 Average: 0.6895 StdDev: 0.82",
"": "Min: 0 Median: 0.5258 Max: 27",
"": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45 P99.99:
13.83",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 4435 95.090% 95.090%
###################",
"": "( 1, 2 ] 149 3.195% 98.285% #",
"": "( 2, 3 ] 55 1.179% 99.464% ",
"": "( 3, 4 ] 12 0.257% 99.721% ",
"": "( 4, 6 ] 8 0.172% 99.893% ",
"": "( 6, 10 ] 3 0.064% 99.957% ",
"": "( 10, 15 ] 2 0.043% 100.000% ",
"": "( 22, 34 ] 1 0.021% 100.021% ",
"": "",
"": "",
"": "** DB Stats **",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes
per commit group, ingest: 0.00 GB, 0.00 MB/s",
"": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written:
0.00 GB, 0.00 MB/s",
"": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent",
"": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes
per
commit group, ingest: 0.00 MB, 0.00 MB/s",
"": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written:
0.00 MB, 0.00 MB/s",
"": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent"
}
We use the NVMe's to store the RocksDb, but with the spillover towards the
spinning drives.
L4 is intended to be stored on the spinning drives...
Will the other levels be created automatically?
Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan(a)bit.nl>nl>:
On 2020-10-06 13:05, Igor Fedotov wrote:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
^^ This, and probably the accompanying RocksDB housekeeping that goes
with it. As only removing PGs shouldn't be a too big a deal at all.
Especially with very small files (and a lot of them) you probably have a
lot of OMAP / META data, (ceph osd df will tell you).
If that's indeed the case than there is a (way) quicker option to get
out of this situation: offline compacting of the OSDs. This process
happens orders of magnitude faster than when the OSDs are still online.
To check if this hypothesis is true: are the OSD servers under CPU
stress where the PGs were located previously (and not the new hosts)?
Offline compaction per host:
systemctl stop ceph-osd.target
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv
/var/lib/ceph/osd/$osd compact &);done
Gr. Stefan
Another strange thing is going on:
No client software is using the system any longer, so we would expect
that all IOs are related to the recovery (fixing of the degraded PG).
However, the disks that are reaching high IO are not a member of the
PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process
immediately. I've read something that there can be old client
processes that keep on connecting to an OSD for retrieving data for a
specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
2:36 p.m.
On 2020-10-06 14:18, Kristof Coucke wrote:
Ok, I did the compact on 1 osd.
The utilization is back to normal, so that's good... Thumbs up to you guys!
We learned the hard way, but happy to spot the issue and share the info.
Though, one thing I want to get out of the way before
adapting the other
OSDs:
When I now get the RocksDb stats, my L1, L2 and L3 are gone:
I guess that they have all been merged now. On a OSD with L0 and L1 I
see L0 disappear after a compact. After a restart, recovery, and then
dumping stats again it's there again. So yeah, it gets created
automatically.
We use the NVMe's to store the RocksDb, but with the spillover towards
the spinning drives.
L4 is intended to be stored on the spinning drives...
Will the other levels be created automatically?
Yeah pretty sure that's how RocksDB works, and tested that (see above).
Gr. Stefan
2:36 p.m.
I've seen similar reports after manual compactions as well. But it looks
like a presentation bug in RocksDB to me.
You can check if all the data is spilled over (as it ought to be for L4)
in bluefs section of OSD perf counters dump...
On 10/6/2020 3:18 PM, Kristof Coucke wrote:
Ok, I did the compact on 1 osd.
The utilization is back to normal, so that's good... Thumbs up to you
guys!
Though, one thing I want to get out of the way before adapting the
other OSDs:
When I now get the RocksDb stats, my L1, L2 and L3 are gone:
db_statistics {
"rocksdb_compaction_statistics": "",
"": "",
"": "** Compaction Stats [default] **",
"": "Level Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"----------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
0 0.000 0 0",
"": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": "",
"": "** Compaction Stats [default] **",
"": "Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Flush(GB): cumulative 0.001, interval 0.001",
"": "AddFile(GB): cumulative 0.000, interval 0.000",
"": "AddFile(Total Files): cumulative 0, interval 0",
"": "AddFile(L0 Files): cumulative 0, interval 0",
"": "AddFile(Keys): cumulative 0, interval 0",
"": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write, 0.00
GB read, 0.00 MB/s read, 0.0 seconds",
"": "Interval compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB
read, 0.00 MB/s read, 0.0 seconds",
"": "Stalls(count): 0 level0_slowdown, 0
level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes,
0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count",
"": "",
"": "** File Read Latency Histogram By Level [default] **",
"": "** Level 0 read latency histogram (micros):",
"": "Count: 5 Average: 69.2000 StdDev: 85.92",
"": "Min: 0 Median: 1.5000 Max: 201",
"": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9: 201.00
P99.99: 201.00",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 2 40.000% 40.000% ########",
"": "( 1, 2 ] 1 20.000% 60.000% ####",
"": "( 110, 170 ] 1 20.000% 80.000% ####",
"": "( 170, 250 ] 1 20.000% 100.000% ####",
"": "",
"": "** Level 4 read latency histogram (micros):",
"": "Count: 4664 Average: 0.6895 StdDev: 0.82",
"": "Min: 0 Median: 0.5258 Max: 27",
"": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45
P99.99: 13.83",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 4435 95.090% 95.090%
###################",
"": "( 1, 2 ] 149 3.195% 98.285% #",
"": "( 2, 3 ] 55 1.179% 99.464% ",
"": "( 3, 4 ] 12 0.257% 99.721% ",
"": "( 4, 6 ] 8 0.172% 99.893% ",
"": "( 6, 10 ] 3 0.064% 99.957% ",
"": "( 10, 15 ] 2 0.043% 100.000% ",
"": "( 22, 34 ] 1 0.021% 100.021% ",
"": "",
"": "",
"": "** DB Stats **",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0
writes per commit group, ingest: 0.00 GB, 0.00 MB/s",
"": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync,
written: 0.00 GB, 0.00 MB/s",
"": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent",
"": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0
writes per commit group, ingest: 0.00 MB, 0.00 MB/s",
"": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync,
written: 0.00 MB, 0.00 MB/s",
"": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent"
}
We use the NVMe's to store the RocksDb, but with the spillover towards
the spinning drives.
L4 is intended to be stored on the spinning drives...
Will the other levels be created automatically?
Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan(a)bit.nl
<mailto:stefan@bit.nl>>:
On 2020-10-06 13:05, Igor Fedotov wrote:
^^ This, and probably the accompanying RocksDB housekeeping that goes
with it. As only removing PGs shouldn't be a too big a deal at all.
Especially with very small files (and a lot of them) you probably
have a
lot of OMAP / META data, (ceph osd df will tell you).
If that's indeed the case than there is a (way) quicker option to get
out of this situation: offline compacting of the OSDs. This process
happens orders of magnitude faster than when the OSDs are still
online.
To check if this hypothesis is true: are the OSD servers under CPU
stress where the PGs were located previously (and not the new hosts)?
Offline compaction per host:
systemctl stop ceph-osd.target
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv
/var/lib/ceph/osd/$osd compact &);done
Gr. Stefan
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
> Another strange thing is going on:
>
> No client software is using the system any longer, so we would
expect
> that all IOs are related to the recovery
(fixing of the
degraded PG).
> However, the disks that are reaching high IO
are not a member
of the
> PGs that are being fixed.
>
> So, something is heavily using the disk, but I can't find the
process
> immediately. I've read something that
there can be old client
> processes that keep on connecting to an OSD for retrieving data
for a
specific
PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
3:24 p.m.
Hi Igor and Stefan,
Everything seems okay, so we'll now create a script to automate this on all
the nodes and we will also review the monitoring possibilities.
Thanks for your help, it was a time saver.
Does anyone know if this issue is better handled in the newer versions or
if this is planned in an upcoming release?
My best regards,
Kristof
Op di 6 okt. 2020 om 14:36 schreef Igor Fedotov <ifedotov(a)suse.de>de>:
I've seen similar reports after manual compactions
as well. But it looks
like a presentation bug in RocksDB to me.
You can check if all the data is spilled over (as it ought to be for L4)
in bluefs section of OSD perf counters dump...
On 10/6/2020 3:18 PM, Kristof Coucke wrote:
Ok, I did the compact on 1 osd.
The utilization is back to normal, so that's good... Thumbs up to you guys!
Though, one thing I want to get out of the way before adapting the other
OSDs:
When I now get the RocksDb stats, my L1, L2 and L3 are gone:
db_statistics {
"rocksdb_compaction_statistics": "",
"": "",
"": "** Compaction Stats [default] **",
"": "Level Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"----------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
0 0.000 0 0",
"": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": "",
"": "** Compaction Stats [default] **",
"": "Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Flush(GB): cumulative 0.001, interval 0.001",
"": "AddFile(GB): cumulative 0.000, interval 0.000",
"": "AddFile(Total Files): cumulative 0, interval 0",
"": "AddFile(L0 Files): cumulative 0, interval 0",
"": "AddFile(Keys): cumulative 0, interval 0",
"": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB
read, 0.00 MB/s read, 0.0 seconds",
"": "Interval compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB
read, 0.00 MB/s read, 0.0 seconds",
"": "Stalls(count): 0 level0_slowdown, 0
level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count",
"": "",
"": "** File Read Latency Histogram By Level [default] **",
"": "** Level 0 read latency histogram (micros):",
"": "Count: 5 Average: 69.2000 StdDev: 85.92",
"": "Min: 0 Median: 1.5000 Max: 201",
"": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9: 201.00
P99.99: 201.00",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 2 40.000% 40.000% ########",
"": "( 1, 2 ] 1 20.000% 60.000% ####",
"": "( 110, 170 ] 1 20.000% 80.000% ####",
"": "( 170, 250 ] 1 20.000% 100.000% ####",
"": "",
"": "** Level 4 read latency histogram (micros):",
"": "Count: 4664 Average: 0.6895 StdDev: 0.82",
"": "Min: 0 Median: 0.5258 Max: 27",
"": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45 P99.99:
13.83",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 4435 95.090% 95.090%
###################",
"": "( 1, 2 ] 149 3.195% 98.285% #",
"": "( 2, 3 ] 55 1.179% 99.464% ",
"": "( 3, 4 ] 12 0.257% 99.721% ",
"": "( 4, 6 ] 8 0.172% 99.893% ",
"": "( 6, 10 ] 3 0.064% 99.957% ",
"": "( 10, 15 ] 2 0.043% 100.000% ",
"": "( 22, 34 ] 1 0.021% 100.021% ",
"": "",
"": "",
"": "** DB Stats **",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes
per commit group, ingest: 0.00 GB, 0.00 MB/s",
"": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync,
written:
0.00 GB, 0.00 MB/s",
"": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent",
"": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes
per commit group, ingest: 0.00 MB, 0.00 MB/s",
"": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written:
0.00 MB, 0.00 MB/s",
"": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent"
}
We use the NVMe's to store the RocksDb, but with the spillover towards the
spinning drives.
L4 is intended to be stored on the spinning drives...
Will the other levels be created automatically?
Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan(a)bit.nl>nl>:
On 2020-10-06 13:05, Igor Fedotov wrote:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
^^ This, and probably the accompanying RocksDB housekeeping that goes
with it. As only removing PGs shouldn't be a too big a deal at all.
Especially with very small files (and a lot of them) you probably have a
lot of OMAP / META data, (ceph osd df will tell you).
If that's indeed the case than there is a (way) quicker option to get
out of this situation: offline compacting of the OSDs. This process
happens orders of magnitude faster than when the OSDs are still online.
To check if this hypothesis is true: are the OSD servers under CPU
stress where the PGs were located previously (and not the new hosts)?
Offline compaction per host:
systemctl stop ceph-osd.target
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv
/var/lib/ceph/osd/$osd compact &);done
Gr. Stefan
Another strange thing is going on:
No client software is using the system any longer, so we would expect
that all IOs are related to the recovery (fixing of the degraded PG).
However, the disks that are reaching high IO are not a member of the
PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process
immediately. I've read something that there can be old client
processes that keep on connecting to an OSD for retrieving data for a
specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
3:27 p.m.
I'm working on improving PG removal in master, see:
https://github.com/ceph/ceph/pull/37496
Hopefully this will help in case of "cleanup after rebalancing" issue
which you presumably had.
On 10/6/2020 4:24 PM, Kristof Coucke wrote:
Hi Igor and Stefan,
Everything seems okay, so we'll now create a script to automate this
on all the nodes and we will also review the monitoring possibilities.
Thanks for your help, it was a time saver.
Does anyone know if this issue is better handled in the newer versions
or if this is planned in an upcoming release?
My best regards,
Kristof
Op di 6 okt. 2020 om 14:36 schreef Igor Fedotov <ifedotov(a)suse.de
<mailto:ifedotov@suse.de>>:
I've seen similar reports after manual compactions as well. But it
looks like a presentation bug in RocksDB to me.
You can check if all the data is spilled over (as it ought to be
for L4) in bluefs section of OSD perf counters dump...
On 10/6/2020 3:18 PM, Kristof Coucke wrote:
> Ok, I did the compact on 1 osd.
> The utilization is back to normal, so that's good... Thumbs up to
> you guys!
> Though, one thing I want to get out of the way before adapting
> the other OSDs:
> When I now get the RocksDb stats, my L1, L2 and L3 are gone:
>
> db_statistics {
> "rocksdb_compaction_statistics": "",
> "": "",
> "": "** Compaction Stats [default] **",
> "": "Level Files Size Score Read(GB) Rn(GB)
> Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s)
> Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
> "":
>
"----------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
> "": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0
> 0.0 0.0 0.0 1.0 0.0 105.1 0.01
> 0.00 1 0.009 0 0",
> "": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.00
> 0.00 0 0.000 0 0",
> "": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0
> 0.0 0.0 0.0 1.0 0.0 105.1 0.01
> 0.00 1 0.009 0 0",
> "": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0
> 0.0 0.0 0.0 1.0 0.0 105.1 0.01
> 0.00 1 0.009 0 0",
> "": "",
> "": "** Compaction Stats [default] **",
> "": "Priority Files Size Score Read(GB) Rn(GB)
> Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s)
> Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
> "":
>
"-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
> "": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0
> 0.0 0.0 0.0 0.0 0.0 105.1 0.01
> 0.00 1 0.009 0 0",
> "": "Uptime(secs): 0.3 total, 0.3 interval",
> "": "Flush(GB): cumulative 0.001, interval 0.001",
> "": "AddFile(GB): cumulative 0.000, interval 0.000",
> "": "AddFile(Total Files): cumulative 0, interval 0",
> "": "AddFile(L0 Files): cumulative 0, interval 0",
> "": "AddFile(Keys): cumulative 0, interval 0",
> "": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write,
> 0.00 GB read, 0.00 MB/s read, 0.0 seconds",
> "": "Interval compaction: 0.00 GB write, 2.84 MB/s write,
> 0.00 GB read, 0.00 MB/s read, 0.0 seconds",
> "": "Stalls(count): 0 level0_slowdown, 0
> level0_slowdown_with_compaction, 0 level0_numfiles, 0
> level0_numfiles_with_compaction, 0 stop for
> pending_compaction_bytes, 0 slowdown for
> pending_compaction_bytes, 0 memtable_compaction, 0
> memtable_slowdown, interval 0 total count",
> "": "",
> "": "** File Read Latency Histogram By Level [default]
**",
> "": "** Level 0 read latency histogram (micros):",
> "": "Count: 5 Average: 69.2000 StdDev: 85.92",
> "": "Min: 0 Median: 1.5000 Max: 201",
> "": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9:
> 201.00 P99.99: 201.00",
> "":
"------------------------------------------------------",
> "": "[ 0, 1 ] 2 40.000% 40.000%
########",
> "": "( 1, 2 ] 1 20.000% 60.000%
####",
> "": "( 110, 170 ] 1 20.000% 80.000%
####",
> "": "( 170, 250 ] 1 20.000% 100.000%
####",
> "": "",
> "": "** Level 4 read latency histogram (micros):",
> "": "Count: 4664 Average: 0.6895 StdDev: 0.82",
> "": "Min: 0 Median: 0.5258 Max: 27",
> "": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45
> P99.99: 13.83",
> "":
"------------------------------------------------------",
> "": "[ 0, 1 ] 4435 95.090% 95.090%
> ###################",
> "": "( 1, 2 ] 149 3.195% 98.285% #",
> "": "( 2, 3 ] 55 1.179% 99.464% ",
> "": "( 3, 4 ] 12 0.257% 99.721% ",
> "": "( 4, 6 ] 8 0.172% 99.893% ",
> "": "( 6, 10 ] 3 0.064% 99.957% ",
> "": "( 10, 15 ] 2 0.043% 100.000% ",
> "": "( 22, 34 ] 1 0.021% 100.021% ",
> "": "",
> "": "",
> "": "** DB Stats **",
> "": "Uptime(secs): 0.3 total, 0.3 interval",
> "": "Cumulative writes: 0 writes, 0 keys, 0 commit groups,
> 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s",
> "": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync,
> written: 0.00 GB, 0.00 MB/s",
> "": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent",
> "": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0
> writes per commit group, ingest: 0.00 MB, 0.00 MB/s",
> "": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync,
> written: 0.00 MB, 0.00 MB/s",
> "": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent"
> }
>
> We use the NVMe's to store the RocksDb, but with the spillover
> towards the spinning drives.
> L4 is intended to be stored on the spinning drives...
> Will the other levels be created automatically?
>
>
> Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan(a)bit.nl
> <mailto:stefan@bit.nl>>:
>
> On 2020-10-06 13:05, Igor Fedotov wrote:
> >
> > On 10/6/2020 1:04 PM, Kristof Coucke wrote:
> >> Another strange thing is going on:
> >>
> >> No client software is using the system any longer, so we
> would expect
> >> that all IOs are related to the recovery (fixing of the
> degraded PG).
> >> However, the disks that are reaching high IO are not a
> member of the
> >> PGs that are being fixed.
> >>
> >> So, something is heavily using the disk, but I can't find
> the process
> >> immediately. I've read something that there can be old client
> >> processes that keep on connecting to an OSD for retrieving
> data for a
> >> specific PG while that PG is no longer available on that disk.
> >>
> >>
> > I bet it's rather PG removal happening in background....
>
> ^^ This, and probably the accompanying RocksDB housekeeping
> that goes
> with it. As only removing PGs shouldn't be a too big a deal
> at all.
> Especially with very small files (and a lot of them) you
> probably have a
> lot of OMAP / META data, (ceph osd df will tell you).
>
> If that's indeed the case than there is a (way) quicker
> option to get
> out of this situation: offline compacting of the OSDs. This
> process
> happens orders of magnitude faster than when the OSDs are
> still online.
>
> To check if this hypothesis is true: are the OSD servers
> under CPU
> stress where the PGs were located previously (and not the new
> hosts)?
>
> Offline compaction per host:
>
> systemctl stop ceph-osd.target
>
> for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool
> bluestore-kv
> /var/lib/ceph/osd/$osd compact &);done
>
> Gr. Stefan
>
3:43 p.m.
On 2020-10-06 15:27, Igor Fedotov wrote:
I'm working on improving PG removal in master,
see:
https://github.com/ceph/ceph/pull/37496
Hopefully this will help in case of "cleanup after rebalancing" issue
which you presumably had.
That would be great. Does the offline compaction with the
ceph-kvstore-tool follows a completely different removal procedure and
is that why it's that much faster?
@Kristof:
https://github.com/ceph/ceph/commit/93e4c56ecc13560e0dad69aaa67afc3ca053fb4c
is a commit by Wido that would help to enable compaction at OSD (re)start.
Gr. Stefan
7 Oct
7 Oct
12:33 p.m.
On 10/6/2020 4:43 PM, Stefan Kooman wrote:
On 2020-10-06 15:27, Igor Fedotov wrote:
First of all - compaction is primarily about rearranging DB data in an
optimal way not removing it. Compaction after massive removals just
optimizes DB data layout and eliminates all the leftovers from the
previous rewrites/removals that negatively impact DB performance.
Offline compaction work in an exclusive access mode and doesn't care
about parallel DB operations. Hence it's presumably much more effective
comparing to the online one. Not to mention that the latter is rather
partial...
>
> @Kristof:
> https://github.com/ceph/ceph/commit/93e4c56ecc13560e0dad69aaa67afc3ca053fb4c
> is a commit by Wido that would help to enable compaction at OSD (re)start.
>
> Gr. Stefan
I'm working on improving PG removal in
master, see:
https://github.com/ceph/ceph/pull/37496
Hopefully this will help in case of "cleanup after rebalancing" issue
which you presumably had.
That would be great. Does the offline compaction with
the
ceph-kvstore-tool follows a completely different removal procedure and
is that why it's that much faster? 
6 Oct
6 Oct
2:32 p.m.
We have similar with this issue last week. We have sluggish disk (10TB
SAS in RAID 0 mode) in half of node which affect performance of cluster.
These disk has high CPU usage and very high latency. Turns out there is
a process *patrol read* from RAID card that running automatically every
week. When we stop patrol read, everything is normal again.
We also running on Ceph 14.2.11. We don't have this issue with previous
Ceph version and never change setting of patrol read.
Thanks.
On 06/10/20 17.04, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect that
all IOs are related to the recovery (fixing of the degraded PG).
However, the disks that are reaching high IO are not a member of the PGs
that are being fixed.
So, something is heavily using the disk, but I can't find the process
immediately. I've read something that there can be old client processes
that keep on connecting to an OSD for retrieving data for a specific PG
while that PG is no longer available on that disk.
Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke <kristof.coucke(a)gmail.com
:
Yes, some disks are spiking near 100%... The delay I see with the iostat
(r_await) seems to be synchronised with the delays between queued_for_pg
and reached_pg events.
The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is also
128kb (os level) (for spinner disks). As we are dealing with smaller files,
this might also lead to a decrease of the performance.
I'm still investigating, but I'm wondering if the system is also reading
from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov(a)suse.de>de>:
> Hi Kristof,
>
> are you seeing high (around 100%) OSDs' disks (main or DB ones)
> utilization along with slow ops?
>
>
> Thanks,
>
> Igor
>
> On 10/6/2020 11:09 AM, Kristof Coucke wrote:
>> Hi all,
>>
>> We have a Ceph cluster which has been expanded from 10 to 16 nodes.
>> Each node has between 14 and 16 OSDs of which 2 are NVMe disks.
>> Most disks (except NVMe's) are 16TB large.
>>
>> The expansion of 16 nodes went ok, but we've configured the system to
>> prevent auto balance towards the new disks (weight was set to 0) so we
>> could control the expansion.
>>
>> We started adding 6 disks last week (1 disk on each new node) which
> didn't
>> give a lot of issues.
>> When the Ceph status indicated the PG degraded was almost finished,
> we've
>> added 2 disks on each node again.
>>
>> All seemed to go fine, till yesterday morning... IOs towards the system
>> were slowing down.
>>
>> Diving onto the nodes we could see that the OSD daemons are consuming
> the
>> CPU power, resulting in average CPU loads going near 10 (!).
>>
>> The RGWs nor monitors nor other involved servers are having CPU issues
>> (except for the management server which is fighting with Prometheus), so
>> it's latency seems to be related to the ODS hosts.
>> All of the hosts are interconnected with 25Gbit connections, no
> bottlenecks
>> are reached on the network either.
>>
>> Important piece of information: We are using erasure coding (6/3), and
> we
>> do have a lot of small files...
>> The current health detail indicates degraded health redundancy where
>> 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg
> undersized).
>> Diving into the historic ops of an OSD we can see that the main latency
> is
>> found between the event "queued_for_pg" and "reached_pg".
(Averaging
> +/- 3
>> secs)
>>
>> As the system load is quite high I assume the systems are busy
>> recalculating the code chunks for using the new disks we've added
> (though
>> not sure), but I was wondering how I can better fine tune the system or
>> pinpoint the exact bottle neck.
>> Latency towards the disks doesn't seem an issue at first sight...
>>
>> We are running Ceph 14.2.11
>>
>> Who can give me some thoughts on how I can better pinpoint the bottle
> neck?
>> Thanks
>>
>> Kristof
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:04 p.m.
I presume that this might be caused by massive KV data removal which was
initiated after(or during) data rebalance. We've seen multiple complains
about RocksDB's performance negatively affected by pool/pg removal. And
I expect data rebalance might suffer from the same...
You might want to run manual DB compaction using ceph-kvstore-tool for
every affected OSD to try to work around the issue. This would rather
help just temporarily if data removal is still ongoing though.
On 10/6/2020 12:41 PM, Kristof Coucke wrote:
Yes, some disks are spiking near 100%... The delay I
see with the
iostat (r_await) seems to be synchronised with the delays between
queued_for_pg and reached_pg events.
The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is also
128kb (os level) (for spinner disks). As we are dealing with smaller
files, this might also lead to a decrease of the performance.
Can you share the amount of DB data spilled over to spinners? You can
learn this from "bluefs" section in performance counters dump...
I'm still investigating, but I'm wondering if
the system is also
reading from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov(a)suse.de
<mailto:ifedotov@suse.de>>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones)
utilization along with slow ops?
Thanks,
Igor
On 10/6/2020 11:09 AM, Kristof Coucke wrote:
Hi all,
We have a Ceph cluster which has been expanded from 10 to 16 nodes.
Each node has between 14 and 16 OSDs of which 2 are NVMe disks.
Most disks (except NVMe's) are 16TB large.
The expansion of 16 nodes went ok, but we've configured the
system to
prevent auto balance towards the new disks
(weight was set to 0)
so we
could control the expansion.
We started adding 6 disks last week (1 disk on each new node)
which
didn't
give a lot of issues.
When the Ceph status indicated the PG degraded was almost
finished, we've
added 2 disks on each node again.
All seemed to go fine, till yesterday morning... IOs towards the
system
were slowing down.
Diving onto the nodes we could see that the OSD daemons are
consuming the
CPU power, resulting in average CPU loads going
near 10 (!).
The RGWs nor monitors nor other involved servers are having CPU
issues
(except for the management server which is
fighting with
Prometheus), so
it's latency seems to be related to the ODS
hosts.
All of the hosts are interconnected with 25Gbit connections, no
bottlenecks
are reached on the network either.
Important piece of information: We are using erasure coding
(6/3), and we
do have a lot of small files...
The current health detail indicates degraded health redundancy where
1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg
undersized).
Diving into the historic ops of an OSD we can see that the main
latency is
found between the event "queued_for_pg"
and "reached_pg".
(Averaging +/- 3
secs)
As the system load is quite high I assume the systems are busy
recalculating the code chunks for using the new disks we've
added (though
not sure), but I was wondering how I can better
fine tune the
system or
pinpoint the exact bottle neck.
Latency towards the disks doesn't seem an issue at first sight...
We are running Ceph 14.2.11
Who can give me some thoughts on how I can better pinpoint the
bottle neck?
Thanks
Kristof
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
1306
days inactive
1307
days old
16 comments
4 participants
participants (4)
-
Danni Setiawan -
Igor Fedotov -
Kristof Coucke -
Stefan Kooman