An MDS-wide lock is acquired before the cache dump is done. After the dump is complete, the lock is released. So, the MDS freezing temporarily during the cache dump is expected. On Fri, May 26, 2023 at 12:51 PM Emmanuel Jaep <emmanuel.jaep@gmail.com> wrote:
Hi Milind,
I finally managed to dump the cache and find the file. It generated a 1.5 GB file with about 7 Mio lines. It's kind of hard to know what is out of the ordinary…
Furthermore, I noticed that dumping the cache was actually stopping the MDS. Is it a normal behavior?
Best,
Emmanuel
On Thu, May 25, 2023 at 1:19 PM Milind Changire <mchangir@redhat.com> wrote:
try the command with the --id argument:
# ceph --id admin --cluster floki daemon mds.icadmin011 dump cache /tmp/dump.txt
I presume that your keyring has an appropriate entry for the client.admin user
On Wed, May 24, 2023 at 5:10 PM Emmanuel Jaep <emmanuel.jaep@gmail.com> wrote:
Absolutely! :-)
root@icadmin011:/tmp# ceph --cluster floki daemon mds.icadmin011 dump cache /tmp/dump.txt root@icadmin011:/tmp# ll total 48 drwxrwxrwt 12 root root 4096 May 24 13:23 ./ drwxr-xr-x 18 root root 4096 Jun 9 2022 ../ drwxrwxrwt 2 root root 4096 May 4 12:43 .ICE-unix/ drwxrwxrwt 2 root root 4096 May 4 12:43 .Test-unix/ drwxrwxrwt 2 root root 4096 May 4 12:43 .X11-unix/ drwxrwxrwt 2 root root 4096 May 4 12:43 .XIM-unix/ drwxrwxrwt 2 root root 4096 May 4 12:43 .font-unix/ drwx------ 2 root root 4096 May 24 13:23 ssh-Sl5AiotnXp/ drwx------ 3 root root 4096 May 8 13:26 'systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-ceph-mds@icadmin011.service-SGZrKf '/ drwx------ 3 root root 4096 May 4 12:43 systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-logind.service-uU1GAi/ drwx------ 3 root root 4096 May 4 12:43 systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-resolved.service-KYHd7f/ drwx------ 3 root root 4096 May 4 12:43 systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-timesyncd.service-1Qtj5i/
On Wed, May 24, 2023 at 1:17 PM Milind Changire <mchangir@redhat.com> wrote:
I hope the daemon mds.icadmin011 is running on the same machine that you are looking for /tmp/dump.txt, since the file is created on the system which has that daemon running.
On Wed, May 24, 2023 at 2:16 PM Emmanuel Jaep <emmanuel.jaep@gmail.com> wrote:
Hi Milind,
you are absolutely right.
The dump_ops_in_flight is giving a good hint about what's happening: { "ops": [ { "description": "internal op exportdir:mds.5:975673", "initiated_at": "2023-05-23T17:49:53.030611+0200", "age": 60596.355186077999, "duration": 60596.355234167997, "type_data": { "flag_point": "failed to wrlock, waiting", "reqid": "mds.5:975673", "op_type": "internal_op", "internal_op": 5377, "op_name": "exportdir", "events": [ { "time": "2023-05-23T17:49:53.030611+0200", "event": "initiated" }, { "time": "2023-05-23T17:49:53.030611+0200", "event": "throttled" }, { "time": "2023-05-23T17:49:53.030611+0200", "event": "header_read" }, { "time": "2023-05-23T17:49:53.030611+0200", "event": "all_read" }, { "time": "2023-05-23T17:49:53.030611+0200", "event": "dispatched" }, { "time": "2023-05-23T17:49:53.030657+0200", "event": "requesting remote authpins" }, { "time": "2023-05-23T17:49:53.050253+0200", "event": "failed to wrlock, waiting" } ] } } ], "num_ops": 1 }
However, the dump cache does not seem to produce an output: root@icadmin011:~# ceph --cluster floki daemon mds.icadmin011 dump cache /tmp/dump.txt root@icadmin011:~# ls /tmp ssh-cHvP3iF611
systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-ceph-mds@icadmin011.service-SGZrKf
systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-logind.service-uU1GAi
systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-resolved.service-KYHd7f
systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-timesyncd.service-1Qtj5i
Do you have any hint?
Best,
Emmanuel
On Wed, May 24, 2023 at 10:30 AM Milind Changire <mchangir@redhat.com> wrote:
Emmanuel, You probably missed the "daemon" keyword after the "ceph" command name. Here's the docs for pacific: https://docs.ceph.com/en/pacific/cephfs/troubleshooting/
So, your command should've been: # ceph daemon mds.icadmin011 dump cache /tmp/dump.txt
You could also dump the ops in flight with: # ceph daemon mds.icadmin011 dump_ops_in_flight
On Wed, May 24, 2023 at 1:38 PM Emmanuel Jaep < emmanuel.jaep@gmail.com> wrote:
> Hi, > > we are running a cephfs cluster with the following version: > ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific > (stable) > > Several MDSs are reporting slow requests: > HEALTH_WARN 4 MDSs report slow requests > [WRN] MDS_SLOW_REQUEST: 4 MDSs report slow requests > mds.icadmin011(mds.5): 1 slow requests are blocked > 30 secs > mds.icadmin015(mds.6): 2 slow requests are blocked > 30 secs > mds.icadmin006(mds.4): 8 slow requests are blocked > 30 secs > mds.icadmin007(mds.2): 2 slow requests are blocked > 30 secs > > According to Quincy's documentation ( > https://docs.ceph.com/en/quincy/cephfs/troubleshooting/), this can be > investigated by issuing: > ceph mds.icadmin011 dump cache /tmp/dump.txt > > Unfortunately, this command fails: > no valid command found; 10 closest matches: > pg stat > pg getmap > pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] > pg dump_json [all|summary|sum|pools|osds|pgs...] > pg dump_pools_json > pg ls-by-pool <poolstr> [<states>...] > pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...] > pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...] > pg ls [<pool:int>] [<states>...] > pg dump_stuck [inactive|unclean|stale|undersized|degraded...] > [<threshold:int>] > Error EINVAL: invalid command > > > I imagine that it is related to the fact that we are running the Pacific > version and not the Quincy version. > > When looking at the Pacific's documentation ( > https://docs.ceph.com/en/pacific/cephfs/health-messages/), I should: > > Use the ops admin socket command to list outstanding metadata operations. > > Unfortunately, I fail to really understand what I'm supposed to do. Can > someone give a pointer? > > Best, > > Emmanuel > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > >
-- Milind _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Milind
-- Milind
-- Milind