MDS lost, Filesystem degraded and wont mount

List overview All Threads
Download

newer

older

CfP Software Defined Storage...

Garbage Collection on Luminous

Anton Aleksandrov

4 Dec 2020 4 Dec '20

12:49 p.m.

Hello community, we are on ceph 13.2.8 - today something happenned with one MDS and cephs status tells, that filesystem is degraded. It won't mount either. I have take server with MDS, that was not working down. There are 2 more MDS servers, but they stay in "rejoin" state. Also only 1 is shown in "services", even though there are 2. Both running MDS servers have these lines in their logs: heartbeat_map is_healthy 'MDSRank' had timed out after 15 mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked 28.8979s ago); MDS internal heartbeat is not healthy! On one of MDS nodes I enabled more detailed debug, so I am getting there also: mds.beacon.mds3 Sending beacon up:standby seq 178 mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 Makes no sense and too much stress in my head... Anyone could help please? Anton.

Show replies by date

Dan van der Ster

4 Dec 4 Dec

12:53 p.m.

How many active MDS's did you have? (max_mds == 1, right?) Stop the other two MDS's so you can focus on getting exactly one running. Tail the log file and see what it is reporting. Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS while it is rejoining. Is that single MDS running out of memory during the rejoin phase? -- dan On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > > Hello community, > > we are on ceph 13.2.8 - today something happenned with one MDS and cephs > status tells, that filesystem is degraded. It won't mount either. I have > take server with MDS, that was not working down. There are 2 more MDS > servers, but they stay in "rejoin" state. Also only 1 is shown in > "services", even though there are 2. > > Both running MDS servers have these lines in their logs: > > heartbeat_map is_healthy 'MDSRank' had timed out after 15 > mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked > 28.8979s ago); MDS internal heartbeat is not healthy! > > On one of MDS nodes I enabled more detailed debug, so I am getting there > also: > > mds.beacon.mds3 Sending beacon up:standby seq 178 > mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 > > Makes no sense and too much stress in my head... Anyone could help please? > > Anton. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Anton Aleksandrov

1:05 p.m.

Yes, MDS eats all memory+swap, stays like this for a moment and then frees memory. mds_beacon_grace was already set to 1800 Also on other it is seen this message: Map has assigned me to become a standby. Does it matter, which MDS we stop and which we leave running? Anton On 04.12.2020 11:53, Dan van der Ster wrote: > How many active MDS's did you have? (max_mds == 1, right?) > > Stop the other two MDS's so you can focus on getting exactly one running. > Tail the log file and see what it is reporting. > Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS > while it is rejoining. > > Is that single MDS running out of memory during the rejoin phase? > > -- dan > > On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >> Hello community, >> >> we are on ceph 13.2.8 - today something happenned with one MDS and cephs >> status tells, that filesystem is degraded. It won't mount either. I have >> take server with MDS, that was not working down. There are 2 more MDS >> servers, but they stay in "rejoin" state. Also only 1 is shown in >> "services", even though there are 2. >> >> Both running MDS servers have these lines in their logs: >> >> heartbeat_map is_healthy 'MDSRank' had timed out after 15 >> mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked >> 28.8979s ago); MDS internal heartbeat is not healthy! >> >> On one of MDS nodes I enabled more detailed debug, so I am getting there >> also: >> >> mds.beacon.mds3 Sending beacon up:standby seq 178 >> mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 >> >> Makes no sense and too much stress in my head... Anyone could help please? >> >> Anton. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Dan van der Ster

1:20 p.m.

Please also make sure the mds_beacon_grace is high on the mon's too. it doesn't matter which mds you select to be the running one. Is the processing getting killed, restarted? If you're confident that the mds is getting OOM killed during rejoin step, then you might find this useful: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028964.html Stop all MDS, then: # rados -p cephfs_metadata_pool rm mds0_openfiles.0 then start one MDS. -- Dan On Fri, Dec 4, 2020 at 11:05 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > > Yes, MDS eats all memory+swap, stays like this for a moment and then > frees memory. > > mds_beacon_grace was already set to 1800 > > Also on other it is seen this message: Map has assigned me to become a > standby. > > Does it matter, which MDS we stop and which we leave running? > > Anton > > > On 04.12.2020 11:53, Dan van der Ster wrote: > > How many active MDS's did you have? (max_mds == 1, right?) > > > > Stop the other two MDS's so you can focus on getting exactly one running. > > Tail the log file and see what it is reporting. > > Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS > > while it is rejoining. > > > > Is that single MDS running out of memory during the rejoin phase? > > > > -- dan > > > > On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > >> Hello community, > >> > >> we are on ceph 13.2.8 - today something happenned with one MDS and cephs > >> status tells, that filesystem is degraded. It won't mount either. I have > >> take server with MDS, that was not working down. There are 2 more MDS > >> servers, but they stay in "rejoin" state. Also only 1 is shown in > >> "services", even though there are 2. > >> > >> Both running MDS servers have these lines in their logs: > >> > >> heartbeat_map is_healthy 'MDSRank' had timed out after 15 > >> mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked > >> 28.8979s ago); MDS internal heartbeat is not healthy! > >> > >> On one of MDS nodes I enabled more detailed debug, so I am getting there > >> also: > >> > >> mds.beacon.mds3 Sending beacon up:standby seq 178 > >> mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 > >> > >> Makes no sense and too much stress in my head... Anyone could help please? > >> > >> Anton. > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Anton Aleksandrov

1:35 p.m.

Thank you very much! This solution helped: Stop all MDS, then: # rados -p cephfs_metadata_pool rm mds0_openfiles.0 then start one MDS. We are back online. Amazing!!! :) On 04.12.2020 12:20, Dan van der Ster wrote: > Please also make sure the mds_beacon_grace is high on the mon's too. > > it doesn't matter which mds you select to be the running one. > > Is the processing getting killed, restarted? > If you're confident that the mds is getting OOM killed during rejoin > step, then you might find this useful: > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028964.html > > Stop all MDS, then: > # rados -p cephfs_metadata_pool rm mds0_openfiles.0 > then start one MDS. > > -- Dan > > On Fri, Dec 4, 2020 at 11:05 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >> Yes, MDS eats all memory+swap, stays like this for a moment and then >> frees memory. >> >> mds_beacon_grace was already set to 1800 >> >> Also on other it is seen this message: Map has assigned me to become a >> standby. >> >> Does it matter, which MDS we stop and which we leave running? >> >> Anton >> >> >> On 04.12.2020 11:53, Dan van der Ster wrote: >>> How many active MDS's did you have? (max_mds == 1, right?) >>> >>> Stop the other two MDS's so you can focus on getting exactly one running. >>> Tail the log file and see what it is reporting. >>> Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS >>> while it is rejoining. >>> >>> Is that single MDS running out of memory during the rejoin phase? >>> >>> -- dan >>> >>> On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >>>> Hello community, >>>> >>>> we are on ceph 13.2.8 - today something happenned with one MDS and cephs >>>> status tells, that filesystem is degraded. It won't mount either. I have >>>> take server with MDS, that was not working down. There are 2 more MDS >>>> servers, but they stay in "rejoin" state. Also only 1 is shown in >>>> "services", even though there are 2. >>>> >>>> Both running MDS servers have these lines in their logs: >>>> >>>> heartbeat_map is_healthy 'MDSRank' had timed out after 15 >>>> mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked >>>> 28.8979s ago); MDS internal heartbeat is not healthy! >>>> >>>> On one of MDS nodes I enabled more detailed debug, so I am getting there >>>> also: >>>> >>>> mds.beacon.mds3 Sending beacon up:standby seq 178 >>>> mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 >>>> >>>> Makes no sense and too much stress in my head... Anyone could help please? >>>> >>>> Anton. >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Dan van der Ster

1:39 p.m.

Excellent! For the record, this PR is the plan to fix this: https://github.com/ceph/ceph/pull/36089 (nautilus, octopus PRs here: https://github.com/ceph/ceph/pull/37382 https://github.com/ceph/ceph/pull/37383) Cheers, Dan On Fri, Dec 4, 2020 at 11:35 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > > Thank you very much! This solution helped: > > Stop all MDS, then: > # rados -p cephfs_metadata_pool rm mds0_openfiles.0 > then start one MDS. > > We are back online. Amazing!!! :) > > > On 04.12.2020 12:20, Dan van der Ster wrote: > > Please also make sure the mds_beacon_grace is high on the mon's too. > > > > it doesn't matter which mds you select to be the running one. > > > > Is the processing getting killed, restarted? > > If you're confident that the mds is getting OOM killed during rejoin > > step, then you might find this useful: > > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028964.html > > > > Stop all MDS, then: > > # rados -p cephfs_metadata_pool rm mds0_openfiles.0 > > then start one MDS. > > > > -- Dan > > > > On Fri, Dec 4, 2020 at 11:05 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > >> Yes, MDS eats all memory+swap, stays like this for a moment and then > >> frees memory. > >> > >> mds_beacon_grace was already set to 1800 > >> > >> Also on other it is seen this message: Map has assigned me to become a > >> standby. > >> > >> Does it matter, which MDS we stop and which we leave running? > >> > >> Anton > >> > >> > >> On 04.12.2020 11:53, Dan van der Ster wrote: > >>> How many active MDS's did you have? (max_mds == 1, right?) > >>> > >>> Stop the other two MDS's so you can focus on getting exactly one running. > >>> Tail the log file and see what it is reporting. > >>> Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS > >>> while it is rejoining. > >>> > >>> Is that single MDS running out of memory during the rejoin phase? > >>> > >>> -- dan > >>> > >>> On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > >>>> Hello community, > >>>> > >>>> we are on ceph 13.2.8 - today something happenned with one MDS and cephs > >>>> status tells, that filesystem is degraded. It won't mount either. I have > >>>> take server with MDS, that was not working down. There are 2 more MDS > >>>> servers, but they stay in "rejoin" state. Also only 1 is shown in > >>>> "services", even though there are 2. > >>>> > >>>> Both running MDS servers have these lines in their logs: > >>>> > >>>> heartbeat_map is_healthy 'MDSRank' had timed out after 15 > >>>> mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked > >>>> 28.8979s ago); MDS internal heartbeat is not healthy! > >>>> > >>>> On one of MDS nodes I enabled more detailed debug, so I am getting there > >>>> also: > >>>> > >>>> mds.beacon.mds3 Sending beacon up:standby seq 178 > >>>> mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 > >>>> > >>>> Makes no sense and too much stress in my head... Anyone could help please? > >>>> > >>>> Anton. > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Janek Bevendorff

10:20 p.m.

This is very common issue. Deleting mdsX_openfiles.Y has become part of my standard maintenance repertoire. As soon as you have a few more clients and one of them starts opening and closing files in rapid succession (or does other metadata-heavy things), it becomes very likely that the MDS crashes and is unable to recover. There have been numerous fixes in the past, which improved the overall stability, but it is far from perfect. I am happy to see another patch in that direction, but I believe more effort needs to be spent here. It is way too easy to DoS the MDS from a single client. Our 78-node CephFS beats our old NFS RAID server in terms of throughput, but latency and stability are way behind. Janek On 04/12/2020 11:39, Dan van der Ster wrote: > Excellent! > > For the record, this PR is the plan to fix this: > https://github.com/ceph/ceph/pull/36089 > (nautilus, octopus PRs here: https://github.com/ceph/ceph/pull/37382 > https://github.com/ceph/ceph/pull/37383) > > Cheers, Dan > > On Fri, Dec 4, 2020 at 11:35 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >> Thank you very much! This solution helped: >> >> Stop all MDS, then: >> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 >> then start one MDS. >> >> We are back online. Amazing!!! :) >> >> >> On 04.12.2020 12:20, Dan van der Ster wrote: >>> Please also make sure the mds_beacon_grace is high on the mon's too. >>> >>> it doesn't matter which mds you select to be the running one. >>> >>> Is the processing getting killed, restarted? >>> If you're confident that the mds is getting OOM killed during rejoin >>> step, then you might find this useful: >>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028964.html >>> >>> Stop all MDS, then: >>> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 >>> then start one MDS. >>> >>> -- Dan >>> >>> On Fri, Dec 4, 2020 at 11:05 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >>>> Yes, MDS eats all memory+swap, stays like this for a moment and then >>>> frees memory. >>>> >>>> mds_beacon_grace was already set to 1800 >>>> >>>> Also on other it is seen this message: Map has assigned me to become a >>>> standby. >>>> >>>> Does it matter, which MDS we stop and which we leave running? >>>> >>>> Anton >>>> >>>> >>>> On 04.12.2020 11:53, Dan van der Ster wrote: >>>>> How many active MDS's did you have? (max_mds == 1, right?) >>>>> >>>>> Stop the other two MDS's so you can focus on getting exactly one running. >>>>> Tail the log file and see what it is reporting. >>>>> Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS >>>>> while it is rejoining. >>>>> >>>>> Is that single MDS running out of memory during the rejoin phase? >>>>> >>>>> -- dan >>>>> >>>>> On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >>>>>> Hello community, >>>>>> >>>>>> we are on ceph 13.2.8 - today something happenned with one MDS and cephs >>>>>> status tells, that filesystem is degraded. It won't mount either. I have >>>>>> take server with MDS, that was not working down. There are 2 more MDS >>>>>> servers, but they stay in "rejoin" state. Also only 1 is shown in >>>>>> "services", even though there are 2. >>>>>> >>>>>> Both running MDS servers have these lines in their logs: >>>>>> >>>>>> heartbeat_map is_healthy 'MDSRank' had timed out after 15 >>>>>> mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked >>>>>> 28.8979s ago); MDS internal heartbeat is not healthy! >>>>>> >>>>>> On one of MDS nodes I enabled more detailed debug, so I am getting there >>>>>> also: >>>>>> >>>>>> mds.beacon.mds3 Sending beacon up:standby seq 178 >>>>>> mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 >>>>>> >>>>>> Makes no sense and too much stress in my head... Anyone could help please? >>>>>> >>>>>> Anton. >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Dan van der Ster

5 Dec 5 Dec

11:26 a.m.

Hi Janek, I'd love to hear your standard maintenance procedures. Are you cleaning up those open files outside of "rejoin" OOMs ? I guess we're pretty lucky with our CephFS's because we have more than 1k clients and it is pretty solid (though the last upgrade had a hiccup decreasing down to single active MDS). -- Dan On Fri, Dec 4, 2020 at 8:20 PM Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: > > This is very common issue. Deleting mdsX_openfiles.Y has become part of > my standard maintenance repertoire. As soon as you have a few more > clients and one of them starts opening and closing files in rapid > succession (or does other metadata-heavy things), it becomes very likely > that the MDS crashes and is unable to recover. > > There have been numerous fixes in the past, which improved the overall > stability, but it is far from perfect. I am happy to see another patch > in that direction, but I believe more effort needs to be spent here. It > is way too easy to DoS the MDS from a single client. Our 78-node CephFS > beats our old NFS RAID server in terms of throughput, but latency and > stability are way behind. > > Janek > > On 04/12/2020 11:39, Dan van der Ster wrote: > > Excellent! > > > > For the record, this PR is the plan to fix this: > > https://github.com/ceph/ceph/pull/36089 > > (nautilus, octopus PRs here: https://github.com/ceph/ceph/pull/37382 > > https://github.com/ceph/ceph/pull/37383) > > > > Cheers, Dan > > > > On Fri, Dec 4, 2020 at 11:35 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > >> Thank you very much! This solution helped: > >> > >> Stop all MDS, then: > >> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 > >> then start one MDS. > >> > >> We are back online. Amazing!!! :) > >> > >> > >> On 04.12.2020 12:20, Dan van der Ster wrote: > >>> Please also make sure the mds_beacon_grace is high on the mon's too. > >>> > >>> it doesn't matter which mds you select to be the running one. > >>> > >>> Is the processing getting killed, restarted? > >>> If you're confident that the mds is getting OOM killed during rejoin > >>> step, then you might find this useful: > >>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028964.html > >>> > >>> Stop all MDS, then: > >>> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 > >>> then start one MDS. > >>> > >>> -- Dan > >>> > >>> On Fri, Dec 4, 2020 at 11:05 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > >>>> Yes, MDS eats all memory+swap, stays like this for a moment and then > >>>> frees memory. > >>>> > >>>> mds_beacon_grace was already set to 1800 > >>>> > >>>> Also on other it is seen this message: Map has assigned me to become a > >>>> standby. > >>>> > >>>> Does it matter, which MDS we stop and which we leave running? > >>>> > >>>> Anton > >>>> > >>>> > >>>> On 04.12.2020 11:53, Dan van der Ster wrote: > >>>>> How many active MDS's did you have? (max_mds == 1, right?) > >>>>> > >>>>> Stop the other two MDS's so you can focus on getting exactly one running. > >>>>> Tail the log file and see what it is reporting. > >>>>> Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS > >>>>> while it is rejoining. > >>>>> > >>>>> Is that single MDS running out of memory during the rejoin phase? > >>>>> > >>>>> -- dan > >>>>> > >>>>> On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > >>>>>> Hello community, > >>>>>> > >>>>>> we are on ceph 13.2.8 - today something happenned with one MDS and cephs > >>>>>> status tells, that filesystem is degraded. It won't mount either. I have > >>>>>> take server with MDS, that was not working down. There are 2 more MDS > >>>>>> servers, but they stay in "rejoin" state. Also only 1 is shown in > >>>>>> "services", even though there are 2. > >>>>>> > >>>>>> Both running MDS servers have these lines in their logs: > >>>>>> > >>>>>> heartbeat_map is_healthy 'MDSRank' had timed out after 15 > >>>>>> mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked > >>>>>> 28.8979s ago); MDS internal heartbeat is not healthy! > >>>>>> > >>>>>> On one of MDS nodes I enabled more detailed debug, so I am getting there > >>>>>> also: > >>>>>> > >>>>>> mds.beacon.mds3 Sending beacon up:standby seq 178 > >>>>>> mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 > >>>>>> > >>>>>> Makes no sense and too much stress in my head... Anyone could help please? > >>>>>> > >>>>>> Anton. > >>>>>> _______________________________________________ > >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Janek Bevendorff

4:41 p.m.

On 05/12/2020 09:26, Dan van der Ster wrote:

...

Hi Janek, I'd love to hear your standard maintenance procedures. Are you cleaning up those open files outside of "rejoin" OOMs ?

No, of course not. But those rejoin problems happen more often than I'd like them to. It has become much better with recent releases, but if one of the clients trains a Tensorflow model from files in the CephFS or when our Hadoop cluster starts reading from it, the MDS will almost certainly crash or at least degrade massively in performance. S3 doesn't have these problems at all, obviously. That said, our metadata pool resides on rotating platters at the moment and we plan to move it to SSDs, but that should only fix latency issues and not the crash and rejoin problems (btw it doesn't matter how long you set the heartbeat interval, the rejoining MDS will always be replaced by a standby before it's finished). > > I guess we're pretty lucky with our CephFS's because we have more than > 1k clients and it is pretty solid (though the last upgrade had a > hiccup decreasing down to single active MDS). > > -- Dan > > > > On Fri, Dec 4, 2020 at 8:20 PM Janek Bevendorff > <janek.bevendorff(a)uni-weimar.de> wrote: >> This is very common issue. Deleting mdsX_openfiles.Y has become part of >> my standard maintenance repertoire. As soon as you have a few more >> clients and one of them starts opening and closing files in rapid >> succession (or does other metadata-heavy things), it becomes very likely >> that the MDS crashes and is unable to recover. >> >> There have been numerous fixes in the past, which improved the overall >> stability, but it is far from perfect. I am happy to see another patch >> in that direction, but I believe more effort needs to be spent here. It >> is way too easy to DoS the MDS from a single client. Our 78-node CephFS >> beats our old NFS RAID server in terms of throughput, but latency and >> stability are way behind. >> >> Janek >> >> On 04/12/2020 11:39, Dan van der Ster wrote: >>> Excellent! >>> >>> For the record, this PR is the plan to fix this: >>> https://github.com/ceph/ceph/pull/36089 >>> (nautilus, octopus PRs here: https://github.com/ceph/ceph/pull/37382 >>> https://github.com/ceph/ceph/pull/37383) >>> >>> Cheers, Dan >>> >>> On Fri, Dec 4, 2020 at 11:35 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >>>> Thank you very much! This solution helped: >>>> >>>> Stop all MDS, then: >>>> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 >>>> then start one MDS. >>>> >>>> We are back online. Amazing!!! :) >>>> >>>> >>>> On 04.12.2020 12:20, Dan van der Ster wrote: >>>>> Please also make sure the mds_beacon_grace is high on the mon's too. >>>>> >>>>> it doesn't matter which mds you select to be the running one. >>>>> >>>>> Is the processing getting killed, restarted? >>>>> If you're confident that the mds is getting OOM killed during rejoin >>>>> step, then you might find this useful: >>>>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028964.html >>>>> >>>>> Stop all MDS, then: >>>>> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 >>>>> then start one MDS. >>>>> >>>>> -- Dan >>>>> >>>>> On Fri, Dec 4, 2020 at 11:05 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >>>>>> Yes, MDS eats all memory+swap, stays like this for a moment and then >>>>>> frees memory. >>>>>> >>>>>> mds_beacon_grace was already set to 1800 >>>>>> >>>>>> Also on other it is seen this message: Map has assigned me to become a >>>>>> standby. >>>>>> >>>>>> Does it matter, which MDS we stop and which we leave running? >>>>>> >>>>>> Anton >>>>>> >>>>>> >>>>>> On 04.12.2020 11:53, Dan van der Ster wrote: >>>>>>> How many active MDS's did you have? (max_mds == 1, right?) >>>>>>> >>>>>>> Stop the other two MDS's so you can focus on getting exactly one running. >>>>>>> Tail the log file and see what it is reporting. >>>>>>> Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS >>>>>>> while it is rejoining. >>>>>>> >>>>>>> Is that single MDS running out of memory during the rejoin phase? >>>>>>> >>>>>>> -- dan >>>>>>> >>>>>>> On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >>>>>>>> Hello community, >>>>>>>> >>>>>>>> we are on ceph 13.2.8 - today something happenned with one MDS and cephs >>>>>>>> status tells, that filesystem is degraded. It won't mount either. I have >>>>>>>> take server with MDS, that was not working down. There are 2 more MDS >>>>>>>> servers, but they stay in "rejoin" state. Also only 1 is shown in >>>>>>>> "services", even though there are 2. >>>>>>>> >>>>>>>> Both running MDS servers have these lines in their logs: >>>>>>>> >>>>>>>> heartbeat_map is_healthy 'MDSRank' had timed out after 15 >>>>>>>> mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked >>>>>>>> 28.8979s ago); MDS internal heartbeat is not healthy! >>>>>>>> >>>>>>>> On one of MDS nodes I enabled more detailed debug, so I am getting there >>>>>>>> also: >>>>>>>> >>>>>>>> mds.beacon.mds3 Sending beacon up:standby seq 178 >>>>>>>> mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 >>>>>>>> >>>>>>>> Makes no sense and too much stress in my head... Anyone could help please? >>>>>>>> >>>>>>>> Anton. >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Dan van der Ster

5:04 p.m.

On Sat, Dec 5, 2020 at 2:41 PM Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote:

...

On 05/12/2020 09:26, Dan van der Ster wrote:

Hi Janek, I'd love to hear your standard maintenance procedures. Are you cleaning up those open files outside of "rejoin" OOMs ?

No, of course not. But those rejoin problems happen more often than I'd like them to. It has become much better with recent releases,

OK sounds good. We very rarely restart MDSs, and only recently upgraded to nautilus (from L) so haven't yet had many opportunities to see how often we'll suffer this rejoin issue on our most active cluster. (Only one of our test clusters saw this happen so far, during mimic days, and this provoked us to move all MDSs to 64GB VMs, with mds cache mem limit = 4GB, so there is a large amount of RAM available in case it's needed. BTW the PR introduces a new option `mds_oft_prefetch_dirfrags`; it remains to be seen exactly how true/false impacts memory consumption.

...

but if one of the clients trains a Tensorflow model from files in the CephFS or when our Hadoop cluster starts reading from it, the MDS will almost certainly crash or at least degrade massively in performance. S3 doesn't have these problems at all, obviously.

What do the crashes look like with the TF training? Do you have a tracker? How many client sessions do need to crash an MDS?

...

That said, our metadata pool resides on rotating platters at the moment and we plan to move it to SSDs, but that should only fix latency issues and not the crash and rejoin problems (btw it doesn't matter how long you set the heartbeat interval, the rejoining MDS will always be replaced by a standby before it's finished).

It works afaict. We needed mds beacon grace = several thousand seconds to survive a 30 minute MDS outage, without any failover to standby. This happened during our L->N upgrade, due to some MDS export extreme slowness. Cheers, Dan P.S. I've seen an older thread of yours about tuning caps recall. I've recently done some testing there and will reply with my input. -- Dan > > > > > I guess we're pretty lucky with our CephFS's because we have more than > > 1k clients and it is pretty solid (though the last upgrade had a > > hiccup decreasing down to single active MDS). > > > > -- Dan > > > > > > > > On Fri, Dec 4, 2020 at 8:20 PM Janek Bevendorff > > <janek.bevendorff(a)uni-weimar.de> wrote: > >> This is very common issue. Deleting mdsX_openfiles.Y has become part of > >> my standard maintenance repertoire. As soon as you have a few more > >> clients and one of them starts opening and closing files in rapid > >> succession (or does other metadata-heavy things), it becomes very likely > >> that the MDS crashes and is unable to recover. > >> > >> There have been numerous fixes in the past, which improved the overall > >> stability, but it is far from perfect. I am happy to see another patch > >> in that direction, but I believe more effort needs to be spent here. It > >> is way too easy to DoS the MDS from a single client. Our 78-node CephFS > >> beats our old NFS RAID server in terms of throughput, but latency and > >> stability are way behind. > >> > >> Janek > >> > >> On 04/12/2020 11:39, Dan van der Ster wrote: > >>> Excellent! > >>> > >>> For the record, this PR is the plan to fix this: > >>> https://github.com/ceph/ceph/pull/36089 > >>> (nautilus, octopus PRs here: https://github.com/ceph/ceph/pull/37382 > >>> https://github.com/ceph/ceph/pull/37383) > >>> > >>> Cheers, Dan > >>> > >>> On Fri, Dec 4, 2020 at 11:35 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > >>>> Thank you very much! This solution helped: > >>>> > >>>> Stop all MDS, then: > >>>> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 > >>>> then start one MDS. > >>>> > >>>> We are back online. Amazing!!! :) > >>>> > >>>> > >>>> On 04.12.2020 12:20, Dan van der Ster wrote: > >>>>> Please also make sure the mds_beacon_grace is high on the mon's too. > >>>>> > >>>>> it doesn't matter which mds you select to be the running one. > >>>>> > >>>>> Is the processing getting killed, restarted? > >>>>> If you're confident that the mds is getting OOM killed during rejoin > >>>>> step, then you might find this useful: > >>>>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028964.html > >>>>> > >>>>> Stop all MDS, then: > >>>>> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 > >>>>> then start one MDS. > >>>>> > >>>>> -- Dan > >>>>> > >>>>> On Fri, Dec 4, 2020 at 11:05 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > >>>>>> Yes, MDS eats all memory+swap, stays like this for a moment and then > >>>>>> frees memory. > >>>>>> > >>>>>> mds_beacon_grace was already set to 1800 > >>>>>> > >>>>>> Also on other it is seen this message: Map has assigned me to become a > >>>>>> standby. > >>>>>> > >>>>>> Does it matter, which MDS we stop and which we leave running? > >>>>>> > >>>>>> Anton > >>>>>> > >>>>>> > >>>>>> On 04.12.2020 11:53, Dan van der Ster wrote: > >>>>>>> How many active MDS's did you have? (max_mds == 1, right?) > >>>>>>> > >>>>>>> Stop the other two MDS's so you can focus on getting exactly one running. > >>>>>>> Tail the log file and see what it is reporting. > >>>>>>> Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS > >>>>>>> while it is rejoining. > >>>>>>> > >>>>>>> Is that single MDS running out of memory during the rejoin phase? > >>>>>>> > >>>>>>> -- dan > >>>>>>> > >>>>>>> On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: > >>>>>>>> Hello community, > >>>>>>>> > >>>>>>>> we are on ceph 13.2.8 - today something happenned with one MDS and cephs > >>>>>>>> status tells, that filesystem is degraded. It won't mount either. I have > >>>>>>>> take server with MDS, that was not working down. There are 2 more MDS > >>>>>>>> servers, but they stay in "rejoin" state. Also only 1 is shown in > >>>>>>>> "services", even though there are 2. > >>>>>>>> > >>>>>>>> Both running MDS servers have these lines in their logs: > >>>>>>>> > >>>>>>>> heartbeat_map is_healthy 'MDSRank' had timed out after 15 > >>>>>>>> mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked > >>>>>>>> 28.8979s ago); MDS internal heartbeat is not healthy! > >>>>>>>> > >>>>>>>> On one of MDS nodes I enabled more detailed debug, so I am getting there > >>>>>>>> also: > >>>>>>>> > >>>>>>>> mds.beacon.mds3 Sending beacon up:standby seq 178 > >>>>>>>> mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 > >>>>>>>> > >>>>>>>> Makes no sense and too much stress in my head... Anyone could help please? > >>>>>>>> > >>>>>>>> Anton. > >>>>>>>> _______________________________________________ > >>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Janek Bevendorff

6 Dec 6 Dec

4:28 p.m.

...

(Only one of our test clusters saw this happen so far, during mimic days, and this provoked us to move all MDSs to 64GB VMs, with mds cache mem limit = 4GB, so there is a large amount of RAM available in case it's needed.

Ours are running on machines with 128GB RAM. I tried limits between 4 and 40GB. But the higher the limit, the higher the fall after a crash. We used to have three MDSs, now I am testing out one to see if that's more stable. At the moment, it runs fine, but we have also outsourced all the heavy lifting to S3.

...

What do the crashes look like with the TF training? Do you have a tracker?

At some point the MDS becomes laggy and is killed and not even the hot standby is able to resume. There is nothing special going on. You only notice that the FS is suddenly degraded and MDS daemons are playing Russian roulette until systemd pulls the plug due to too many daemon failures. At that point I have to fail the remaining ones, run systemctl reset-failed, delete the openfiles objects and restart the daemons.

...

How many client sessions do need to crash an MDS?

Depends. Surprisingly, it can be as little as one big node with 1.5TB of RAM and a few hungry GPUs. > -- Dan > > >>> I guess we're pretty lucky with our CephFS's because we have more than >>> 1k clients and it is pretty solid (though the last upgrade had a >>> hiccup decreasing down to single active MDS). >>> >>> -- Dan >>> >>> >>> >>> On Fri, Dec 4, 2020 at 8:20 PM Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de> wrote: >>>> This is very common issue. Deleting mdsX_openfiles.Y has become part of >>>> my standard maintenance repertoire. As soon as you have a few more >>>> clients and one of them starts opening and closing files in rapid >>>> succession (or does other metadata-heavy things), it becomes very likely >>>> that the MDS crashes and is unable to recover. >>>> >>>> There have been numerous fixes in the past, which improved the overall >>>> stability, but it is far from perfect. I am happy to see another patch >>>> in that direction, but I believe more effort needs to be spent here. It >>>> is way too easy to DoS the MDS from a single client. Our 78-node CephFS >>>> beats our old NFS RAID server in terms of throughput, but latency and >>>> stability are way behind. >>>> >>>> Janek >>>> >>>> On 04/12/2020 11:39, Dan van der Ster wrote: >>>>> Excellent! >>>>> >>>>> For the record, this PR is the plan to fix this: >>>>> https://github.com/ceph/ceph/pull/36089 >>>>> (nautilus, octopus PRs here: https://github.com/ceph/ceph/pull/37382 >>>>> https://github.com/ceph/ceph/pull/37383) >>>>> >>>>> Cheers, Dan >>>>> >>>>> On Fri, Dec 4, 2020 at 11:35 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >>>>>> Thank you very much! This solution helped: >>>>>> >>>>>> Stop all MDS, then: >>>>>> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 >>>>>> then start one MDS. >>>>>> >>>>>> We are back online. Amazing!!! :) >>>>>> >>>>>> >>>>>> On 04.12.2020 12:20, Dan van der Ster wrote: >>>>>>> Please also make sure the mds_beacon_grace is high on the mon's too. >>>>>>> >>>>>>> it doesn't matter which mds you select to be the running one. >>>>>>> >>>>>>> Is the processing getting killed, restarted? >>>>>>> If you're confident that the mds is getting OOM killed during rejoin >>>>>>> step, then you might find this useful: >>>>>>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028964.html >>>>>>> >>>>>>> Stop all MDS, then: >>>>>>> # rados -p cephfs_metadata_pool rm mds0_openfiles.0 >>>>>>> then start one MDS. >>>>>>> >>>>>>> -- Dan >>>>>>> >>>>>>> On Fri, Dec 4, 2020 at 11:05 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >>>>>>>> Yes, MDS eats all memory+swap, stays like this for a moment and then >>>>>>>> frees memory. >>>>>>>> >>>>>>>> mds_beacon_grace was already set to 1800 >>>>>>>> >>>>>>>> Also on other it is seen this message: Map has assigned me to become a >>>>>>>> standby. >>>>>>>> >>>>>>>> Does it matter, which MDS we stop and which we leave running? >>>>>>>> >>>>>>>> Anton >>>>>>>> >>>>>>>> >>>>>>>> On 04.12.2020 11:53, Dan van der Ster wrote: >>>>>>>>> How many active MDS's did you have? (max_mds == 1, right?) >>>>>>>>> >>>>>>>>> Stop the other two MDS's so you can focus on getting exactly one running. >>>>>>>>> Tail the log file and see what it is reporting. >>>>>>>>> Increase mds_beacon_grace to 600 so that the mon doesn't fail this MDS >>>>>>>>> while it is rejoining. >>>>>>>>> >>>>>>>>> Is that single MDS running out of memory during the rejoin phase? >>>>>>>>> >>>>>>>>> -- dan >>>>>>>>> >>>>>>>>> On Fri, Dec 4, 2020 at 10:49 AM Anton Aleksandrov <anton(a)aleksandrov.eu> wrote: >>>>>>>>>> Hello community, >>>>>>>>>> >>>>>>>>>> we are on ceph 13.2.8 - today something happenned with one MDS and cephs >>>>>>>>>> status tells, that filesystem is degraded. It won't mount either. I have >>>>>>>>>> take server with MDS, that was not working down. There are 2 more MDS >>>>>>>>>> servers, but they stay in "rejoin" state. Also only 1 is shown in >>>>>>>>>> "services", even though there are 2. >>>>>>>>>> >>>>>>>>>> Both running MDS servers have these lines in their logs: >>>>>>>>>> >>>>>>>>>> heartbeat_map is_healthy 'MDSRank' had timed out after 15 >>>>>>>>>> mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked >>>>>>>>>> 28.8979s ago); MDS internal heartbeat is not healthy! >>>>>>>>>> >>>>>>>>>> On one of MDS nodes I enabled more detailed debug, so I am getting there >>>>>>>>>> also: >>>>>>>>>> >>>>>>>>>> mds.beacon.mds3 Sending beacon up:standby seq 178 >>>>>>>>>> mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 >>>>>>>>>> >>>>>>>>>> Makes no sense and too much stress in my head... Anyone could help please? >>>>>>>>>> >>>>>>>>>> Anton. >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Patrick Donnelly

7 Dec 7 Dec

10:51 p.m.

On Sat, Dec 5, 2020 at 5:41 AM Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote:

...

On 05/12/2020 09:26, Dan van der Ster wrote:

Hi Janek, I'd love to hear your standard maintenance procedures. Are you cleaning up those open files outside of "rejoin" OOMs ?

This sounds like there is one or a few clients acquiring too many caps. Have you checked this? Are there any messages about the OOM killer? What config changes for the MDS have you made? I'm hopeful your problems will be addressed by: https://tracker.ceph.com/issues/47307 -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

Janek Bevendorff

8 Dec 8 Dec

12:28 a.m.

...

This sounds like there is one or a few clients acquiring too many caps. Have you checked this? Are there any messages about the OOM killer? What config changes for the MDS have you made?

Yes, it's individual clients acquiring too my caps. I first ran the adjusted recall settings you suggested after we had gone through several bugs. Right now I am trying distributed ephemeral pinning with 3 MDS Dan's suggestion of 6x the default values for recall from the MDS documentation thread. So far, it's working quite well.

...

I'm hopeful your problems will be addressed by: https://tracker.ceph.com/issues/47307

That does indeed sound a bit like it might fix these kind of issues.

Patrick Donnelly

1:34 a.m.

On Mon, Dec 7, 2020 at 1:28 PM Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote:

...

This sounds like there is one or a few clients acquiring too many caps. Have you checked this? Are there any messages about the OOM killer? What config changes for the MDS have you made?

Wow! Distributed epins :) Thanks for trying it. How many sub-directories under the distributed epin'd directory? (There's a lot of stability problems that are to be fixed in Pacific associated with lots of subtrees so if you have too large of a directory, things could get ugly!)

...

I'm hopeful your problems will be addressed by: https://tracker.ceph.com/issues/47307

That does indeed sound a bit like it might fix these kind of issues.

-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

Janek Bevendorff

11:10 a.m.

...

Yay, beta testing in production! ^^ We are talking millions, but the three is very deep, not very wide. That's why it's so hard to maintain manual pins. I enabled it on a few levels of the tree, where the largest one has 117 direct descendants (but several million files below). So far, it's working all right, but it is very hard to see if the settings is actually effective. I enabled it for testing purposes on a directory that was (at that time) rather busy with 3k MDS op/s and I could see a handful of new pins come and go in ceph tell mds.0 get subtrees, but most of our directories are rather idle most of the time and manually browsing the tree isn't enough to trigger any new observable epins it seems. So for the main directories where it actually matters, I can only assume that it's working.

1239

days inactive

1243

days old

ceph-users@ceph.io

Manage subscription

14 comments

4 participants

tags (0)

participants (4)

Anton Aleksandrov
Dan van der Ster
Janek Bevendorff
Patrick Donnelly