Emmanuel, You probably missed the "daemon" keyword after the "ceph" command name. Here's the docs for pacific: https://docs.ceph.com/en/pacific/cephfs/troubleshooting/ So, your command should've been: # ceph daemon mds.icadmin011 dump cache /tmp/dump.txt You could also dump the ops in flight with: # ceph daemon mds.icadmin011 dump_ops_in_flight On Wed, May 24, 2023 at 1:38 PM Emmanuel Jaep <emmanuel.jaep@gmail.com> wrote:
Hi,
we are running a cephfs cluster with the following version: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
Several MDSs are reporting slow requests: HEALTH_WARN 4 MDSs report slow requests [WRN] MDS_SLOW_REQUEST: 4 MDSs report slow requests mds.icadmin011(mds.5): 1 slow requests are blocked > 30 secs mds.icadmin015(mds.6): 2 slow requests are blocked > 30 secs mds.icadmin006(mds.4): 8 slow requests are blocked > 30 secs mds.icadmin007(mds.2): 2 slow requests are blocked > 30 secs
According to Quincy's documentation ( https://docs.ceph.com/en/quincy/cephfs/troubleshooting/), this can be investigated by issuing: ceph mds.icadmin011 dump cache /tmp/dump.txt
Unfortunately, this command fails: no valid command found; 10 closest matches: pg stat pg getmap pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] pg dump_json [all|summary|sum|pools|osds|pgs...] pg dump_pools_json pg ls-by-pool <poolstr> [<states>...] pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...] pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...] pg ls [<pool:int>] [<states>...] pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [<threshold:int>] Error EINVAL: invalid command
I imagine that it is related to the fact that we are running the Pacific version and not the Quincy version.
When looking at the Pacific's documentation ( https://docs.ceph.com/en/pacific/cephfs/health-messages/), I should:
Use the ops admin socket command to list outstanding metadata operations.
Unfortunately, I fail to really understand what I'm supposed to do. Can someone give a pointer?
Best,
Emmanuel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Milind