After upgrading one of our clusters from Luminous 12.2.12 to Nautilus 14.2.6, I am seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H'). The way we found this was due to Prometheus being unable to report out certain pieces of data, specifically OSD Usage, OSD Apply and Commit Latency. Which are all similar issues people were having in previous versions of Nautilus. Bryan Stillwell reported this previously on a separate cluster, 14.2.5, we have here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VW3GNVJGOOWA... That issue was resolved with the upgrade to 14.2.6. We are seeing a similar issue on this other cluster with a couple differences. This cluster has 1900+ OSD in it, the previous one had 300+ The top user is libceph-common, instead of mmap 4.86% libceph-common.so.0 [.] EventCenter::create_time_event 2.78% [kernel] [k] nmi 2.64% libstdc++.so.6.0.19 [.] __dynamic_cast On all our other clusters that have been upgraded to 14.2.6 we are not experiencing this issue, the next largest being 800+ OSD. We feel this is related to the size of the cluster, similarly to the previous report. Anyone else experiencing this and/or can provide some direction on how to go about resolving this? Thanks, Joe
On 2020-01-29 01:19, jbardgett@godaddy.com wrote:
We feel this is related to the size of the cluster, similarly to the previous report.
Anyone else experiencing this and/or can provide some direction on how to go about resolving this?
What Manager modules are enabled on that node? Have you tried disabling some of them, e.g. the Dashboard or Balancer module? Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)
Modules that are normally enabled: ceph mgr module ls | jq -r '.enabled_modules' [ "dashboard", "prometheus", "restful" ] We did test with all modules disabled, restarted the mgrs and saw no difference. Joe
Hi Joe, Can you grab a wallclock profiler dump from the mgr process and share it with us? This was useful for us to get to the root cause of the issue in 14.2.5. Quoting Mark's suggestion from "[ceph-users] High CPU usage by ceph-mgr in 14.2.5" below. If you can get a wallclock profiler on the mgr process we might be able to figure out specifics of what's taking so much time (ie processing pg_summary or something else). Assuming you have gdb with the python bindings and the ceph debug packages installed, if you (are anyone) could try gdbpmp on the 100% mgr process that would be fantastic. https://github.com/markhpc/gdbpmp gdbpmp.py -p`pidof ceph-mgr` -n 1000 -o mgr.gdbpmp If you want to view the results: gdbpmp.py -i mgr.gdbpmp -t 1 Thanks, Neha On Wed, Jan 29, 2020 at 7:35 AM <jbardgett@godaddy.com> wrote:
Modules that are normally enabled:
ceph mgr module ls | jq -r '.enabled_modules' [ "dashboard", "prometheus", "restful" ]
We did test with all modules disabled, restarted the mgrs and saw no difference.
Joe _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I can report similar results, although it's probably not just due to cluster size. Our cluster has 1248 OSDs at the moment and we have three active MDSs to spread the metadata operations evenly. However, I noticed that it isn't spread evenly at all. Usually, it's just one MDS (in our case mds.1) which handles most of the load and slowing down the others as a result. What we see is a significantly higher latency curve for this one MDS than for the other two. All MDSs operate at 100-150% CPU utilisation when multiple clients (we have up to 320) are actively reading or writing data (note: we have quite an uneven data distribution, so directory pinning isn't really an option). In the end, it turned out that some clients were running updatedb processes which tried to index the CephFS. After fixing that, the constant request load went down and with it the CPU load on the MDSs, but the underlying problem isn't solved of course. We just don't have any clients constantly operating on some of our largest directories anymore. On 29/01/2020 20:28, Neha Ojha wrote:
Hi Joe,
Can you grab a wallclock profiler dump from the mgr process and share it with us? This was useful for us to get to the root cause of the issue in 14.2.5.
Quoting Mark's suggestion from "[ceph-users] High CPU usage by ceph-mgr in 14.2.5" below.
If you can get a wallclock profiler on the mgr process we might be able to figure out specifics of what's taking so much time (ie processing pg_summary or something else). Assuming you have gdb with the python bindings and the ceph debug packages installed, if you (are anyone) could try gdbpmp on the 100% mgr process that would be fantastic.
https://github.com/markhpc/gdbpmp
gdbpmp.py -p`pidof ceph-mgr` -n 1000 -o mgr.gdbpmp
If you want to view the results:
gdbpmp.py -i mgr.gdbpmp -t 1
Thanks, Neha
On Wed, Jan 29, 2020 at 7:35 AM <jbardgett@godaddy.com> wrote:
Modules that are normally enabled:
ceph mgr module ls | jq -r '.enabled_modules' [ "dashboard", "prometheus", "restful" ]
We did test with all modules disabled, restarted the mgrs and saw no difference.
Joe _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Does anyone have access to libibverbs-debuginfo-22.1-3.el7.x86_64 and librdmacm-debuginfo-22.1-3.el7.x86_64? I cannot find them in any repo list out there and the gdbpmp.py requires them. Thanks, Joe
Has anyone attempted to use gdbpmp since 14.2.6 to grab data? I have not been able to successfully do it on my clusters. It has just been hanging at attaching to process. If you have been able to, would you be available for a discussion regarding your configuration? Thanks, Joe Bardgett Storage Operations jbardgett@godaddy.com Cell - 480-221-7337 Office - 602-420-4403 This email message and any attachments hereto is intended for use only by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you have received this email in error, please immediately notify the sender and permanently delete the original and any copy of this message and its attachments. -----Original Message----- From: Neha Ojha <nojha@redhat.com> Sent: Wednesday, January 29, 2020 12:29 PM To: Joe Bardgett <jbardgett@godaddy.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: High CPU usage by ceph-mgr in 14.2.6 Notice: This email is from an external sender. Hi Joe, Can you grab a wallclock profiler dump from the mgr process and share it with us? This was useful for us to get to the root cause of the issue in 14.2.5. Quoting Mark's suggestion from "[ceph-users] High CPU usage by ceph-mgr in 14.2.5" below. If you can get a wallclock profiler on the mgr process we might be able to figure out specifics of what's taking so much time (ie processing pg_summary or something else). Assuming you have gdb with the python bindings and the ceph debug packages installed, if you (are anyone) could try gdbpmp on the 100% mgr process that would be fantastic. https://github.com/markhpc/gdbpmp gdbpmp.py -p`pidof ceph-mgr` -n 1000 -o mgr.gdbpmp If you want to view the results: gdbpmp.py -i mgr.gdbpmp -t 1 Thanks, Neha On Wed, Jan 29, 2020 at 7:35 AM <jbardgett@godaddy.com> wrote:
Modules that are normally enabled:
ceph mgr module ls | jq -r '.enabled_modules' [ "dashboard", "prometheus", "restful" ]
We did test with all modules disabled, restarted the mgrs and saw no difference.
Joe _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm having the same issue on 14.2.4. Have you fixed it? I disabled all modules apart from pg balancer (can't be disabled). I opened a report with wallclock profiler on tracker https://tracker.ceph.com/issues/44496
participants (6)
-
danjou.philippe＠yahoo.de
-
Janek Bevendorff
-
jbardgett＠godaddy.com
-
Joe Bardgett
-
Lenz Grimmer
-
Neha Ojha