I can report similar results, although it's probably not just due to cluster size. Our cluster has 1248 OSDs at the moment and we have three active MDSs to spread the metadata operations evenly. However, I noticed that it isn't spread evenly at all. Usually, it's just one MDS (in our case mds.1) which handles most of the load and slowing down the others as a result. What we see is a significantly higher latency curve for this one MDS than for the other two. All MDSs operate at 100-150% CPU utilisation when multiple clients (we have up to 320) are actively reading or writing data (note: we have quite an uneven data distribution, so directory pinning isn't really an option). In the end, it turned out that some clients were running updatedb processes which tried to index the CephFS. After fixing that, the constant request load went down and with it the CPU load on the MDSs, but the underlying problem isn't solved of course. We just don't have any clients constantly operating on some of our largest directories anymore. On 29/01/2020 20:28, Neha Ojha wrote:
Hi Joe,
Can you grab a wallclock profiler dump from the mgr process and share it with us? This was useful for us to get to the root cause of the issue in 14.2.5.
Quoting Mark's suggestion from "[ceph-users] High CPU usage by ceph-mgr in 14.2.5" below.
If you can get a wallclock profiler on the mgr process we might be able to figure out specifics of what's taking so much time (ie processing pg_summary or something else). Assuming you have gdb with the python bindings and the ceph debug packages installed, if you (are anyone) could try gdbpmp on the 100% mgr process that would be fantastic.
https://github.com/markhpc/gdbpmp
gdbpmp.py -p`pidof ceph-mgr` -n 1000 -o mgr.gdbpmp
If you want to view the results:
gdbpmp.py -i mgr.gdbpmp -t 1
Thanks, Neha
On Wed, Jan 29, 2020 at 7:35 AM <jbardgett@godaddy.com> wrote:
Modules that are normally enabled:
ceph mgr module ls | jq -r '.enabled_modules' [ "dashboard", "prometheus", "restful" ]
We did test with all modules disabled, restarted the mgrs and saw no difference.
Joe _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io