Hi Paul, Could you create a ceph tracker (tracker.ceph.com) and list out things that are suboptimal according to your investigation? We'd like to hear more on this. Alternatively, you could list the issues with mds here. Thanks, Milind On Sun, Jan 7, 2024 at 4:37 PM Paul Mezzanini <pfmeec@rit.edu> wrote:
We've seen it use as much as 1.6t of ram/swap. Swap makes it slow, but a slow recovery is better than no recovery. My coworker looked into it at the source code level and while it is doing some things suboptimal that's how it's currently written.
The MDS code needs some real love if ceph is going to offer file services that can match what the back end storage can actually provide.
--
Paul Mezzanini Platform Engineer III Research Computing
Rochester Institute of Technology
Sent from my phone, please excuse typos and brevity ________________________________ From: Lars Köppel <lars.koeppel@ariadne.ai> Sent: Sunday, January 7, 2024 4:20:05 AM To: Paul Mezzanini <pfmeec@rit.edu> Cc: Patrick Donnelly <pdonnell@redhat.com>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: mds crashes after up:replay state
Hi Paul,
your suggestion was correct. The mds went through the replay state and was a few minutes in the active state. But then it gets killed because of too high memory consumption. @mds.cephfs.storage01.pgperp.service: Main process exited, code=exited, status=137/n/a How could I raise the memory limit for the mds?
From the looks in htop. It looked like there is a memory leak, because it consumed over 200 GB of memory while reporting that it actually used 20 - 30 GB. Is this possible?
Best regardes Lars
[ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel@ariadne.ai<mailto:lars.koeppel@ariadne.ai> Phone: +49 6221 5993580<tel:+4962215993580> ariadne.ai<http://ariadne.ai> (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai
On Sat, Jan 6, 2024 at 3:33 PM Paul Mezzanini <pfmeec@rit.edu<mailto:pfmeec@rit.edu>> wrote: I'm replying from my phone so hopefully this works well. This sounds suspiciously similar to an issue we have run into where there is an internal loop in the MDS that doesn't have heartbeat in it. If that loop goes for too long, it is marked as failed and the process jumps to another server and starts again.
We get around it by "wedging it in a corner" and removing the ability to migrate. This is as simple as stopping all standby MDS services and just waiting for the MDS to complete.
--
Paul Mezzanini Platform Engineer III Research Computing
Rochester Institute of Technology
Sent from my phone, please excuse typos and brevity ________________________________ From: Lars Köppel <lars.koeppel@ariadne.ai<mailto:lars.koeppel@ariadne.ai>> Sent: Saturday, January 6, 2024 7:22:14 AM To: Patrick Donnelly <pdonnell@redhat.com<mailto:pdonnell@redhat.com>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> <ceph-users@ceph.io<mailto:ceph-users@ceph.io>> Subject: [ceph-users] Re: mds crashes after up:replay state
Hi Patrick,
thank you for your response. I already changed the mentioned settings, but I had no luck with this.
The journal inspection I had running yesterday finished with: 'Overall journal integrity: OK'. So you are probably right that the mds is crashing shortly after the replay finished.
I checked the logs and there is every few seconds a new FSMap epoch without any visible changes. One of the current epochs is at the end. Is there anything useful in it?
When the replay is finished the running mds goes to the state 'up:reconnect' and after a second to the state 'up:rejoin'. After this there is for ~20 min no new fsmap until this message pops up:
Jan 06 12:38:23 storage01 ceph-mds[223997]: mds.beacon.cephfs.storage01.pgperp Skipping beacon heartbeat to monitors (last acked 4.00012s ago); MDS internal heartbeat is not healthy!
A few seconds later (the heartbeat message is still there) a new fsmap is created with a new mds now in replay state. The last of the heartbeat messages is after 1446 seconds. Then it is gone and no more warnings or errors are displayed at this point. One minute after the last message the mds is back as standy mds.
Jan 06 13:02:26 storage01 ceph-mds[223997]: mds.beacon.cephfs.storage01.pgperp Skipping beacon heartbeat to monitors (last acked 1446.6s ago); MDS internal heartbeat is not healthy!
Also i can not find any warning in the logs when the mds crashes. What could I do to find the error for the crash?
Best regardes Lars
e205510
enable_multiple, ever_enabled_multiple: 1,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 3
Filesystem 'cephfs' (3) fs_name cephfs epoch 205510 flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay created 2023-06-06T11:44:03.651905+0000 modified 2024-01-06T10:28:14.676738+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 8796093022208 required_client_features {} last_failure 0 last_failure_osd_epoch 42962 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=2178448} failed damaged stopped data_pools [11,12] metadata_pool 10 inline_data disabled balancer standby_count_wanted 1 [mds.cephfs.storage01.pgperp{0:2178448} state up:replay seq 4484 join_fscid=3 addr [v2: 192.168.0.101:6800/855849996,v1:192.168.0.101:6801/855849996<http://192.168.0.101:6800/855849996,v1:192.168.0.101:6801/855849996>] compat {c=[1],r=[1],i=[7ff]}]
Filesystem 'cephfs_recovery' (4) fs_name cephfs_recovery epoch 193460 flags 13 allow_snaps allow_multimds_snaps created 2024-01-05T10:47:32.224388+0000 modified 2024-01-05T16:43:37.677241+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 42904 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {} failed damaged 0 stopped data_pools [11,12] metadata_pool 13 inline_data disabled balancer standby_count_wanted 1
Standby daemons:
[mds.cephfs.storage02.zopcif{-1:2356728} state up:standby seq 1 join_fscid=3 addr [v2: 192.168.0.102:6800/3567764205,v1:192.168.0.102:6801/3567764205<http://192.168.0.102:6800/3567764205,v1:192.168.0.102:6801/3567764205>] compat {c=[1],r=[1],i=[7ff]}] dumped fsmap epoch 205510
[image: ariadne.ai<http://ariadne.ai> Logo] Lars Köppel Developer Email: lars.koeppel@ariadne.ai<mailto:lars.koeppel@ariadne.ai> Phone: +49 6221 5993580 <+4962215993580> ariadne.ai<http://ariadne.ai> (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai
On Fri, Jan 5, 2024 at 7:52 PM Patrick Donnelly <pdonnell@redhat.com<mailto:pdonnell@redhat.com>> wrote:
Hi Lars,
On Fri, Jan 5, 2024 at 9:53 AM Lars Köppel <lars.koeppel@ariadne.ai<mailto:lars.koeppel@ariadne.ai>> wrote:
Hello everyone,
we are running a small cluster with 3 nodes and 25 osds per node. And
version 17.2.6. Recently the active mds crashed and since then the new starting mds has always been in the up:replay state. In the output of the command 'ceph tell mds.cephfs:0 status' you can see that the journal is completely read in. As soon as it's finished, the mds crashes and the next one starts reading
Ceph the
journal.
At the moment I have the journal inspection running ('cephfs-journal-tool --rank=cephfs:0 journal inspect').
Does anyone have any further suggestions on how I can get the cluster running again as quickly as possible?
Please review:
https://docs.ceph.com/en/reef/cephfs/troubleshooting/#stuck-during-recovery
Note: your MDS is probably not failing in up:replay but shortly after reaching one of the later states. Check the mon logs to see what the FSMap changes were.
Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Milind