24 Jun
2020
24 Jun
'20
2:30 a.m.
Hi,
We have recently added a new storage node to our Luminous (12.2.13)
cluster. The prev nodes are all setup as Filestore: e.g 12 osds on hdd
(Seagate Constellations) with one NVMe (Intel P4600) journal. With the
new guy we decided to introduce Bluestore so it is configured as: (same
HW) 12 osd with data on hdd and db + wal on one NVMe.
We noticed there are periodic slow requests logged, and the implicated
osds are the Bluestore ones 98% of the time! This suggests that we need
to tweak our Bluestore settings in some way. Investigating I'm seeing:
- A great deal of rocksdb debug info in the logs - perhaps we should
tone that down? (debug_rocksdb 4/5 -> 1/5)
- We look to have the default cache settings
(bluestore_cache_size_hdd|ssd etc), we have memory to increase these
- There are some buffered io settings (bluefs_buffered_io,
bluestore_default_buffered_write), set to (default) false. Are these
safe (or useful) to change?
- We have default rocksdb options, should some of these be changed?
(bluestore_rocksdb_options, in particular max_background_compactions=2 -
should we have less, or more?)
Also, anything else we should be looking at?
regards
Mark
26 Jun
26 Jun
12:47 a.m.
Progress update:
- tweaked debug_rocksdb to 1/5. *possibly* helped, fewer slow requests
- will increase osd_memory_target from 4 to 16G, and observe
On 24/06/20 1:30 pm, Mark Kirkwood wrote:
> Hi,
>
> We have recently added a new storage node to our Luminous (12.2.13)
> cluster. The prev nodes are all setup as Filestore: e.g 12 osds on hdd
> (Seagate Constellations) with one NVMe (Intel P4600) journal. With the
> new guy we decided to introduce Bluestore so it is configured as:
> (same HW) 12 osd with data on hdd and db + wal on one NVMe.
>
> We noticed there are periodic slow requests logged, and the implicated
> osds are the Bluestore ones 98% of the time! This suggests that we
> need to tweak our Bluestore settings in some way. Investigating I'm
> seeing:
>
> - A great deal of rocksdb debug info in the logs - perhaps we should
> tone that down? (debug_rocksdb 4/5 -> 1/5)
>
> - We look to have the default cache settings
> (bluestore_cache_size_hdd|ssd etc), we have memory to increase these
>
> - There are some buffered io settings (bluefs_buffered_io,
> bluestore_default_buffered_write), set to (default) false. Are these
> safe (or useful) to change?
>
> - We have default rocksdb options, should some of these be changed?
> (bluestore_rocksdb_options, in particular max_background_compactions=2
> - should we have less, or more?)
>
> Also, anything else we should be looking at?
>
> regards
>
> Mark
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:59 a.m.
From my point of view, it's better to have no more than 6 osd wal/db on 1
nvme.
I think that's the root cause of the slow requests, maybe.
Mark Kirkwood <mark.kirkwood(a)catalyst.net.nz> 于2020年6月26日周五 上午7:47写道：
Progress update:
- tweaked debug_rocksdb to 1/5. *possibly* helped, fewer slow requests
- will increase osd_memory_target from 4 to 16G, and observe
On 24/06/20 1:30 pm, Mark Kirkwood wrote:
Hi,
We have recently added a new storage node to our Luminous (12.2.13)
cluster. The prev nodes are all setup as Filestore: e.g 12 osds on hdd
(Seagate Constellations) with one NVMe (Intel P4600) journal. With the
new guy we decided to introduce Bluestore so it is configured as:
(same HW) 12 osd with data on hdd and db + wal on one NVMe.
We noticed there are periodic slow requests logged, and the implicated
osds are the Bluestore ones 98% of the time! This suggests that we
need to tweak our Bluestore settings in some way. Investigating I'm
seeing:
- A great deal of rocksdb debug info in the logs - perhaps we should
tone that down? (debug_rocksdb 4/5 -> 1/5)
- We look to have the default cache settings
(bluestore_cache_size_hdd|ssd etc), we have memory to increase these
- There are some buffered io settings (bluefs_buffered_io,
bluestore_default_buffered_write), set to (default) false. Are these
safe (or useful) to change?
- We have default rocksdb options, should some of these be changed?
(bluestore_rocksdb_options, in particular max_background_compactions=2
- should we have less, or more?)
Also, anything else we should be looking at?
regards
Mark
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
29 Jun
29 Jun
11:01 p.m.
That is an interesting point. We are using 12 on 1 nvme journal for our
Filestore nodes (which seems to work ok). The workload for wal + db is
different so that could be a factor. However when I've looked at the IO
metrics for the nvme it seems to be only lightly loaded, so does not
appear to be the issue (at 1st sight anyway).
Also the particular nvme model could be a factor (in the 6 vs 12 area)
- what type are you using?
regards
Mark
On 26/06/20 8:59 pm, Zhenshi Zhou wrote:
> From my point of view, it's better to have no more than 6 osd wal/db
> on 1 nvme.
> I think that's the root cause of the slow requests, maybe.
>
> Mark Kirkwood <mark.kirkwood(a)catalyst.net.nz
> <mailto:mark.kirkwood@catalyst.net.nz>> 于2020年6月26日周五 上午7:47写道：
>
> Progress update:
>
> - tweaked debug_rocksdb to 1/5. *possibly* helped, fewer slow requests
>
> - will increase osd_memory_target from 4 to 16G, and observe
>
> On 24/06/20 1:30 pm, Mark Kirkwood wrote:
> > Hi,
> >
> > We have recently added a new storage node to our Luminous (12.2.13)
> > cluster. The prev nodes are all setup as Filestore: e.g 12 osds
> on hdd
> > (Seagate Constellations) with one NVMe (Intel P4600) journal.
> With the
> > new guy we decided to introduce Bluestore so it is configured as:
> > (same HW) 12 osd with data on hdd and db + wal on one NVMe.
> >
> > We noticed there are periodic slow requests logged, and the
> implicated
> > osds are the Bluestore ones 98% of the time! This suggests that we
> > need to tweak our Bluestore settings in some way. Investigating I'm
> > seeing:
> >
> > - A great deal of rocksdb debug info in the logs - perhaps we
> should
> > tone that down? (debug_rocksdb 4/5 -> 1/5)
> >
> > - We look to have the default cache settings
> > (bluestore_cache_size_hdd|ssd etc), we have memory to increase these
> >
> > - There are some buffered io settings (bluefs_buffered_io,
> > bluestore_default_buffered_write), set to (default) false. Are
> these
> > safe (or useful) to change?
> >
> > - We have default rocksdb options, should some of these be changed?
> > (bluestore_rocksdb_options, in particular
> max_background_compactions=2
> > - should we have less, or more?)
> >
> > Also, anything else we should be looking at?
> >
> > regards
> >
> > Mark
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> <mailto:ceph-users@ceph.io>
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> <mailto:ceph-users-leave@ceph.io>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> <mailto:ceph-users@ceph.io>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> <mailto:ceph-users-leave@ceph.io>
>
30 Jun
30 Jun
4:46 p.m.
That is an interesting point. We are using 12 on 1
nvme journal for our Filestore nodes (which seems to work ok). The workload for wal + db
is different so that could be a factor. However when I've looked at the IO metrics for
the nvme it seems to be only lightly loaded, so does not appear to be the issue (at 1st
sight anyway).
How are you determining “lightly loaded”. Not iostat %util I hope.
Also the particular nvme model could be a factor (in
the 6 vs 12 area) - what type are you using?
I suspect that SSD OSDs would favor the 6 end of the scale; 12 doesn’t seem wholly
unreasonable for HDD OSDs. In the end though, add the hassle factor and the cost of the
NVMe device — when provisioned with enough capacity — and maybe just using SATA SSDs
makese sease.
11:46 p.m.
On Wed, 1 Jul 2020 at 01:47, Anthony D'Atri <anthony.datri(a)gmail.com> wrote:
For reference, this explains why iostat for SSDs can be misleading:
https://brooker.co.za/blog/2014/07/04/iostat-pct.html
However when
I've looked at the IO metrics for the nvme it seems to be only lightly loaded, so does
not appear to be the issue (at 1st sight anyway).
How are you determining “lightly loaded”. Not iostat %util I hope.
1 Jul
1 Jul
12:03 a.m.
Increasing the memory target appears to have solved the issue.
On 26/06/20 11:47 am, Mark Kirkwood wrote:
> Progress update:
>
> - tweaked debug_rocksdb to 1/5. *possibly* helped, fewer slow requests
>
> - will increase osd_memory_target from 4 to 16G, and observe
>
> On 24/06/20 1:30 pm, Mark Kirkwood wrote:
>> Hi,
>>
>> We have recently added a new storage node to our Luminous (12.2.13)
>> cluster. The prev nodes are all setup as Filestore: e.g 12 osds on
>> hdd (Seagate Constellations) with one NVMe (Intel P4600) journal.
>> With the new guy we decided to introduce Bluestore so it is
>> configured as: (same HW) 12 osd with data on hdd and db + wal on one
>> NVMe.
>>
>> We noticed there are periodic slow requests logged, and the
>> implicated osds are the Bluestore ones 98% of the time! This suggests
>> that we need to tweak our Bluestore settings in some way.
>> Investigating I'm seeing:
>>
>> - A great deal of rocksdb debug info in the logs - perhaps we should
>> tone that down? (debug_rocksdb 4/5 -> 1/5)
>>
>> - We look to have the default cache settings
>> (bluestore_cache_size_hdd|ssd etc), we have memory to increase these
>>
>> - There are some buffered io settings (bluefs_buffered_io,
>> bluestore_default_buffered_write), set to (default) false. Are these
>> safe (or useful) to change?
>>
>> - We have default rocksdb options, should some of these be changed?
>> (bluestore_rocksdb_options, in particular
>> max_background_compactions=2 - should we have less, or more?)
>>
>> Also, anything else we should be looking at?
>>
>> regards
>>
>> Mark
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1394
days inactive
1400
days old
6 comments
4 participants
participants (4)
-
Anthony D'Atri -
Mark Kirkwood -
Nigel Williams -
Zhenshi Zhou