On Mon, 2020-08-31 at 14:36 +0000, Eugen Block wrote:
Disks are
utilized roughly between 70 and 80 percent. Not sure why
would operations slow down when disks are getting more utilization.
If that would be the case, I'd expect Ceph to issue a warning.
It is warning you, that's why you see slow requests. ;-) But just
to
be clear, by utilization I mean more than just the filling level of
the OSD, have you watched iostat (or something similar) for your
disks
during usual and high load? Heavy metadata operation on rocksDB
increases the load on the main device. I'm not sure if you
mentioned
it before, do you have stand-alone OSDs or with faster db devices?
I
believe you only mentioned cephfs_metadata on SSD.
Indeed DB is stored on HDDs and only metadata resides on SSDs.
I accidentally stumbled upon someone mentioning disk caching should be
disabled to increase performance.
I'm now looking into how to configure that on these:
- PERC H730P Adapter (has cache on controller)
- Dell HBA330 Adp (doesn't have cache on controller)
It is not as easy as executing "hdparm" command :(
I'll also look into code that does the "truncating" as it may not be
Ceph-friendly :/
Have I
understood correctly that the expectation is that if I used
larger drives I wouldn't be seeing these warnings?
I can understand that adding more disks would create better
parallelisation, that's why I'm asking about larger drives.
I don't think larger drives would improve that, probably even the
opposite, depending on the drives, of course. More drives should
scale, yes, but there's more to it.
Zitat von Momčilo Medić <fedorauser(a)fedoraproject.org>rg>:
> Hey Eugen,
>
> On Wed, 2020-08-26 at 09:29 +0000, Eugen Block wrote:
> > Hi,
> >
> > > > root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue
> > > > wpq
> > > > root@0cephosd01:~# ceph config get mds.cephosd01
> > > > osd_op_queue_cut_off
> > > > high
> >
> > just to make sure, I referred to OSD not MDS settings, maybe
> > check
> > again?
>
> root@cephosd01:~# ceph config get osd.* osd_op_queue
> wpq
> root@cephosd01:~# ceph config get osd.* osd_op_queue_cut_off
> high
> root@cephosd01:~# ceph config get mon.* osd_op_queue
> wpq
> root@cephosd01:~# ceph config get mon.* osd_op_queue_cut_off
> high
> root@cephosd01:~# ceph config get mds.* osd_op_queue
> wpq
> root@cephosd01:~# ceph config get mds.* osd_op_queue_cut_off
> high
> root@cephosd01:~#
>
> It seems no matter which setting I query, it's always the same.
> Also, documentation for OSD clearly states[1] that it is the
> default.
>
> > I wouldn't focus too much on the MDS service, 64 GB RAM should be
> > enough, but you could and should also check the actual RAM usage,
> > of
> > course. But in our case it's pretty clear that the hard disks are
> > the
> > bottleneck although we have rocksDB on SSD for all OSDs. We seem
> > to
> > have a similar use case (we have nightly compile jobs running in
> > cephfs) just with fewer clients. Our HDDs are saturated
> > especially
> > if
> > we also run deep-scrubs during the night, but the slow requests
> > have
> > been reduced since we changed the osd_op_queue settings for our
> > OSDs.
> >
> > Have you checked your disk utilization?
>
> Disks are utilized roughly between 70 and 80 percent. Not sure why
> would operations slow down when disks are getting more utilization.
> If that would be the case, I'd expect Ceph to issue a warning.
>
Have I understood correctly that the expectation
is that if I used
larger drives I wouldn't be seeing these warnings?
I can understand that adding more disks would create better
parallelisation, that's why I'm asking about larger drives.
>
> Thank you for discussing this with me, it's highly appreciated.
>
> <snip>
>
> [1]
>
https://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/#opera…
Kind regards,
Momo.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io