Hey Eugen,
On Wed, 2020-08-26 at 09:29 +0000, Eugen Block wrote:
Hi,
> root@cephosd01:~# ceph config get
mds.cephosd01 osd_op_queue
> wpq
> root@0cephosd01:~# ceph config get mds.cephosd01
> osd_op_queue_cut_off
> high
just to make sure, I referred to OSD not MDS settings, maybe check
again?
root@cephosd01:~# ceph config get osd.* osd_op_queue
wpq
root@cephosd01:~# ceph config get osd.* osd_op_queue_cut_off
high
root@cephosd01:~# ceph config get mon.* osd_op_queue
wpq
root@cephosd01:~# ceph config get mon.* osd_op_queue_cut_off
high
root@cephosd01:~# ceph config get mds.* osd_op_queue
wpq
root@cephosd01:~# ceph config get mds.* osd_op_queue_cut_off
high
root@cephosd01:~#
It seems no matter which setting I query, it's always the same.
Also, documentation for OSD clearly states[1] that it is the default.
I wouldn't focus too much on the MDS service, 64
GB RAM should be
enough, but you could and should also check the actual RAM usage,
of
course. But in our case it's pretty clear that the hard disks are
the
bottleneck although we have rocksDB on SSD for all OSDs. We seem
to
have a similar use case (we have nightly compile jobs running in
cephfs) just with fewer clients. Our HDDs are saturated especially
if
we also run deep-scrubs during the night, but the slow requests
have
been reduced since we changed the osd_op_queue settings for our OSDs.
Have you checked your disk utilization?
Disks are utilized roughly between 70 and 80 percent. Not sure why
would operations slow down when disks are getting more utilization.
If that would be the case, I'd expect Ceph to issue a warning.
Have I understood correctly that the expectation is that if I used
larger drives I wouldn't be seeing these warnings?
I can understand that adding more disks would create better
parallelisation, that's why I'm asking about larger drives.
Thank you for discussing this with me, it's highly appreciated.
<snip>
[1]
https://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/#opera…
Kind regards,
Momo.