Alright, I did some further research and found this topic which seems to be about the same problem: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/024944.ht… We have many small files (as I said, the file list is 23GB) and since we are only copying them but we are not accessing them afterwards, the clients start piling up capabilities, which at least explains the growing cache sizes and the MDSs' failure to keep up (there should definitely be a solution to this, batch-copying many files to a CephFS is a pretty standard use case). After I increased the beacon grace period, I experienced few MDS crashes (although I still see them flapping occasionally), but now I have another problem. After too many MDS failures (?) the client starts locking up and the mount becomes unresponsive. Sometimes it becomes so unresponsive, I cannot even unmount it with umount -lf and have to force-reboot the server. While the client is locked up, the MDSs recover and the FS is accessible again without issues from other clients. This looks like a bug to me. I tried upgrading the client from Mimic to Nautilus, but I have the same problem. I increased the MDS max cache size massively and started the copy job again, let's see how far it goes this time. On 22.07.19 15:02, Janek Bevendorff wrote: -- Bauhaus-Universität Weimar Bauhausstr. 9a, Room 308 99423 Weimar, Germany Phone: +49 (0)3643 - 58 3577

Looking at the archives, I think this list is dead. I moved the discussion to the ceph.com mailing list. No idea why there are two lists. Cheers and apologies to anybody here on this list. On 23/07/2019 18:20, Janek Bevendorff wrote: > Unfortunately, the experiment failed, so I tried increasing the number > of MDS to four (although I had a bad experience with three before). > This worked surprisingly well for some time, but the crash came > eventually and the rank-0 MDS got kicked. Now the standbys have been > playing round-robin trying to join and getting kicked again for two > hours straight without any end in sight. > > Log messages are > > 2019-07-23 17:53:35.618 7f3b165ab700  0 mds.beacon.XXX Skipping beacon > heartbeat to monitors (last acked 4.50406s ago); MDS internal > heartbeat is not healthy! > 2019-07-23 17:53:36.118 7f3b165ab700  1 heartbeat_map is_healthy > 'MDSRank' had timed out after 15 > > Followed by some of these: > > 2019-07-23 17:53:37.386 7f3b135a5700  0 mds.0.cache.ino(0x100019693f8) > have open dirfrag * but not leaf in fragtree_t(*^3): [dir > 0x100019693f8 /XXX_12_doc_ids_part7/ [2,head] auth{1=2,2=2} v=0 cv=0/0 > state=1140850688 f() n() hs=17033+0,ss=0+0 | child=1 replicated=1 > 0x5642a2ff7700] > > and finally: > > 2019-07-23 17:53:48.786 7fb02bc08700  1 mds.XXX Map has assigned me to > become a standby > > > It is impossible to migrate our storage server to CephFS if this > continues. I would immensely appreciate some help on this. Thanks a lot! > > Janek > > > On 23.07.19 14:18, Janek Bevendorff wrote: >> Alright, I did some further research and found this topic which seems >> to be about the same problem: >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/024944.ht… >> >> We have many small files (as I said, the file list is 23GB) and since >> we are only copying them but we are not accessing them afterwards, >> the clients start piling up capabilities, which at least explains the >> growing cache sizes and the MDSs' failure to keep up (there should >> definitely be a solution to this, batch-copying many files to a >> CephFS is a pretty standard use case). >> >> After I increased the beacon grace period, I experienced few MDS >> crashes (although I still see them flapping occasionally), but now I >> have another problem. After too many MDS failures (?) the client >> starts locking up and the mount becomes unresponsive. Sometimes it >> becomes so unresponsive, I cannot even unmount it with umount -lf and >> have to force-reboot the server. While the client is locked up, the >> MDSs recover and the FS is accessible again without issues from other >> clients. This looks like a bug to me. I tried upgrading the client >> from Mimic to Nautilus, but I have the same problem. >> >> I increased the MDS max cache size massively and started the copy job >> again, let's see how far it goes this time. >> >> >> On 22.07.19 15:02, Janek Bevendorff wrote: >>> Hi, >>> >>> I am trying to copy the contents of our storage server into a >>> CephFS, but am experiencing stability issues with my MDSs. The >>> CephFS sits on top of an erasure-coded pool with 5 MONs, 5 MDSs and >>> a max_mds setting of two. My Ceph cluster version is Nautilus, the >>> client is Mimic and uses the kernel module to mount the FS. >>> >>> The index of filenames to copy is about 23GB and I am using 16 >>> parallel rsync processes over a 10G link to copy the files over to >>> Ceph. This works perfectly for a while, but then the MDSs start >>> reporting oversized caches (between 20 and 50GB, sometimes more) and >>> an inode count between 1 and 4 million. Particularly the Inode count >>> seems quite high to me. Each rsync job has 25k files to work with, >>> so if all 16 processes open all their files at the same time, I >>> should not exceed 400k. Even if I double this number to account for >>> the client's page cache, I should get nowhere near that number of >>> inodes (a sync flush takes about 1 second). >>> >>> Then after a few hours, my MDSs start failing with messages like this: >>> >>>    -21> 2019-07-22 14:00:05.877 7f67eacec700  1 heartbeat_map >>> is_healthy 'MDSRank' had timed out after 15 >>>    -20> 2019-07-22 14:00:05.877 7f67eacec700  0 mds.beacon.XXX >>> Skipping beacon heartbeat to monitors (last acked 24.0042s ago); MDS >>> internal heartbeat is not healthy! >>> >>> The standby nodes try to take over, but take forever to become >>> active and will fail as well eventually. >>> >>> During my research, I found this related topic: >>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html, >>> but I tried everything in there from increasing to lowering my cache >>> size, the number of segments etc. I also played around with the >>> number of active MDSs and two appears to work the best, whereas one >>> cannot keep up with the load and three seems to be the worst of all >>> choices. >>> >>> Do you have any ideas how I can improve the stability of my MDS >>> damons to handle the load properly? single 10G link is a toy and we >>> could query the cluster with a lot more requests per second, but >>> it's already yielding to 16 rsync processes. >>> >>> Thanks >>> >>

There are two mailing lists announced on the website. If you go to https://ceph.com/resources/ you will find the subscribe/unsubscribe/archive links for the (much more active) ceph.com MLs. But if you click on "Mailing Lists & IRC page" you will get to a page where you can subscribe to this list, which is different. Very confusing. I actually did not think I'd get a reply here. We are a bit further than this on the other mailing list. This is the thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/036095.html To sum it up: the ceph client prevents the MDS from freeing its cache, so inodes keep piling up until either the MDS becomes too slow (fixable by increasing the beacon grace time) or runs out of memory. The latter will happen eventually. In the end, my MDSs couldn't even rejoin because they hit the host's 128GB memory limit and crashed. The same happens with only one MDS. I tried it with a fresh CephFS and after two minutes of rsyncing stuff into it, I hit 900k inodes before I stopped the process. The point was to have multiple for the transfer, because one couldn't keep up. But now it seems like the problem wasn't really the single-MDS bottleneck. The problem was rather the unbounded growth of runaway inodes, which I do not have a solution for. Yeah, same thing.

Thank you. I was looking for config directives that do exactly this all week. Why are they not documented anywhere outside that blog post? I added them as you described and the MDS seems to have stabilized and stays just under 1M inos now. I will continue to monitor it and see if it is working in the long run. Settings like these should be the default IMHO. Clients should never be able to crash the server just by holding onto their capabilities. If a server decides to drop things from its cache, clients must deal with it. Everything else threatens the stability of the system (and may even prevent the MDS from ever starting again, as we saw). No, I am not. I tried it at some point to see if it made things better, but just like the memory cache limit, it seemed to have no effect whatsoever except for delaying the health warning. I will try that, although I directory tree is quite imbalanced.

On Thu, Jul 25, 2019 at 3:08 AM Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: Based on that message, it would appear you still have an inode limit in place ("mds_cache_size"). Please unset that config option. Your mds_cache_memory_limit is apparently ~19GB. There is another limit mds_max_caps_per_client (default 1M) which the client is hitting. That's why the MDS is recalling caps from the client and not because any cache memory limit is hit. It is not recommend you increase this. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

On Thu, Jul 25, 2019 at 12:49 PM Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: Can you share two captures of `ceph daemon mds.X perf dump` about 1 second apart. You can also try increasing the aggressiveness of the MDS recall but I'm surprised it's still a problem with the settings I gave you: ceph config set mds mds_recall_max_caps 15000 ceph config set mds mds_recall_max_decay_rate 0.75 -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

Hi, I finally had the chance to try the more aggressive recall settings, but they did not change anything. As soon as the client starts copying files again, the numbers go up an I get a health message that the client is failing to respond to cache pressure. After this week of idle time, the dns/inos numbers (what does dns stand for anyway?) settled at around 8000k. That's basically that "idle" number that it goes back to when the client stops copying files. Though, for some weird reason, this number gets (quite) a bit higher every time (last time it was around 960k). Of course, I wouldn't expect it to go back all the way to zero, because that would mean dropping the entire cache for no reason, but it's still quite high and the same after restarting the MDS and all clients, which doesn't make a lot of sense to me. After resuming the copy job, the number went up to 20M in just the time it takes to write this email. There must be a bug somewhere. I attached the requested perf dumps. Thanks!

That did help. Somewhat. I removed the aggressive recall settings I set before and only set this option instead. The cache size seems to be quite stable now, although still increasing in the long run (but at least not strictly monotonically). However, now my client processes are basically in constant I/O wait state and the CephFS is slow for everybody. After I restarted the copy job, I got around 4k reqs/s and then it went down to 100 reqs/s with everybody waiting their turn. So yes, it does seem to help, but it increases latency by a magnitude. As always, it would be great if these options were documented somewhere. Google has like five results, one of them being this thread. ;-) > Increase it further if it's not aggressive enough. Please let us know > if that helps. > > It shouldn't be necessary to do this so I'll make a tracker ticket > once we confirm that's the issue.

On Tue, Aug 6, 2019 at 12:48 AM Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: 4k req/s is too fast for a create workload on one MDS. That must include other operations like getattr. I wouldn't expect such extreme latency issues. Please share: ceph config dump ceph daemon mds.X cache status and the two perf dumps one second apart again please. Also, you said you removed the aggressive recall changes. I assume you didn't reset them to the defaults, right? Just the first suggested change (10k/1.0)? -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

On Tue, Aug 6, 2019 at 7:57 AM Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: Your parallel rsync job is only getting 150 creates per second? What was the previous throughput? The cache size looks correct here. Okay, so you're getting a more normal throughput for parallel creates on a single MDS. Try pinning if possible in each parallel rsync job. Here are tracker tickets to resolve the issues you encountered: https://tracker.ceph.com/issues/41140 https://tracker.ceph.com/issues/41141 -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

I've been copying happily for days now (not very fast, but the MDS were stable), but eventually the MDSs started flapping again due to large cache sizes (they are being killed after 11M inodes). I could solve the problem by temporarily increasing the cache size in order to allow them to rejoin, but it tells me that my settings do not fully solve the problem yet (unless perhaps I increase the trim threshold even further. On 06.08.19 19:52, Janek Bevendorff wrote: >> Your parallel rsync job is only getting 150 creates per second? What >> was the previous throughput? > I am actually not quite sure what the exact throughput was or is or what > I can expect. It varies so much. I am copying from a 23GB file list that > is split into 3000 chunks which are then processed by 16-24 parallel > rsync processes. I have copied 27 of 64TB so far (according to df -h) > and to my taste it's taking a lot longer than it should be doing. The > main problem here is not that I'm trying to copy 64TB (drop in the > bucket), the problem is that it's 64TB in tiny, small, and medium-sized > files. > > This whole MDS mess and several pauses and restarts in between have > completely distorted my sense of how far in the process I actually am or > how fast I would expect it to go. Right now it's starting again from the > beginning, so I expect it'll be another day or so until it starts moving > some real data again. > >> The cache size looks correct here. > Yeah. Cache appears to be constant-size now. I am still getting > occasional "client failing to respond to cache pressure", but that goes > away as fast as it came. > > >> Try pinning if possible in each parallel rsync job. > I was considering that, but couldn't come up with a feasible pinning > strategy. We have all those files of very different sizes spread very > unevenly across a handful of top-level directories. I get the impression > that I couldn't do much (or any) better than the automatic balancer. > > >> Here are tracker tickets to resolve the issues you encountered: >> >> https://tracker.ceph.com/issues/41140 >> https://tracker.ceph.com/issues/41141 > Thanks a lot! > _______________________________________________ > Ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Amazing, thanks! > I've > also done testing in a large cluster to confirm the issue you found. > Using multiple processes to create files as fast as possible in a > single client reliably reproduced the issue. The MDS cannot recall > capabilities fast enough when the internal upkeep thread ran every 5 > seconds. Moving the cache trimming and capability recall to a separate > thread running every second resolved the issue. > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Senior Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

On Thu, Dec 5, 2019 at 10:31 AM Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: You set mds_beacon_grace ? Yes, this optimization is having some struggles with large cache sizes (ironically). Luckily, nuking the open file objects is harmless... 14.2.5: https://tracker.ceph.com/issues/41467 MDS will soon be more aggressive about recalling caps from idle sessions, which may help: https://tracker.ceph.com/issues/41865 That'll make 14.2.6 probably. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

Hey Patrick, I just wanted to give you some feedback about how 14.2.5 is working for me. I've had the chance to test it for a day now and overall, the experience is much better, although not perfect (perhaps far from it). I have two active MDS (I figured that'd spread the meta data load a little and seems to work pretty well for me). After the upgrade to the new release, I removed all special recall settings, so my MDS config is basically on default. The only thing I set is a mds_max_caps_per_client of 200k, a mds_cache_reservation of 0.1 and 40G of mds_cache_memory_limit. Right now, everything seems to be running smoothly, although I notice that the max cap setting isn't fully honoured. The overall cache size seems fairly constant at 15M (for mds.0, mds.1 a little less), but the client cap count can easily exceed 10M if I run something like `find` on a large directory. We have one particularly problematic folder containing about 400 sub folders holding a total of about 35M files among them. My first attempts at running `find -type d` on those had the weird effect that after pretty much exactly 2M caps, mds.1 got killed and replaced by a standby. Fortunately, the standby managed to take over in a matter of seconds (sometimes up to a few minutes) resetting the cap count to about 5k. The same thing then happened again once the new MDS reached the magical 2M caps. I would suppose that this is still the same problem as before, but with the huge improvement that the take-over standby MDS can actually recover. Previously, it would just die the same way after a minute or two of futile recovery attempts and the FS would be down indefinitely until I delete the openfiles object. Right now, I cannot reproduce the crash any more---the caps to surge to 10-15M, but no crash. However, I keep seeing the dreaded "client failing to respond to cache pressure" message occasionally. So far, though, the MDS have been able to keep up and reduce the number of caps after about 15M, though, so that the message disappears after a while and the cap count growth isn't entirely unbounded. I ran a `find -type d` on the most problematic folder and attached two perf dumps for you (current cap count on the client: 14660568): https://pastebin.com/W2dVJiW0 https://pastebin.com/pzQ5uQQ3 Cheers Janek P.S. Just as I was finishing this email, the rank 0 MDS actually crashed. Unfortunately, I didn't have increased debug levels enabled, so its death note is rather uninformative: 2019-12-17 09:42:12.325 7f7633dde700  1 mds.deltaweb011 Updating MDS map to version 103112 from mon.3 2019-12-17 09:43:27.774 7f7633dde700  1 mds.deltaweb011 Updating MDS map to version 103113 from mon.3 2019-12-17 09:43:40.086 7f7633dde700  1 mds.deltaweb011 Updating MDS map to version 103114 from mon.3 2019-12-17 09:44:46.203 7f7633dde700 -1 *** Caught signal (Aborted) **  in thread 7f7633dde700 thread_name:ms_dispatch Also, this time around the recovery appears to be a lot more problematic, so I'm afraid I have to apply the previous procedure again of deleting the openfiles object to get it back up. I don't think my `find` alone would have crashed the MDS, but if another client is doing similar things at the same time, it overloads the MDS.

I played with the cache size, yeah. I kind of need a large cache, otherwise everything is just slow and I'm constantly getting cache size warnings. ceph tell mds.* doesn't work when the FS is degraded.

Update: turns out I just had to wait for an hour. The MDSs were sending Beacons regularly, so the MONs didn't try to kill them and instead let them finish doing whatever they were doing. Unlike the other bug where the number of open files outgrows what the MDS can handle, this incident allowed "self-healing", but I still consider this a severe bug. On 06/01/2020 12:05, Janek Bevendorff wrote: > Hi, my MDS failed again, but this time I cannot recover it by deleting > the mds*_openfiles .0 object. The startup behaviour is also different. > Both inode count and cache size stay at zero while the MDS is replaying. > > When I set the MDS log level to 7, I get tons of these messages: > > 2020-01-06 11:59:49.303 7f30149e4700  7 mds.1.cache  current root is > [dir 0x10000073682 /XXX/XXX/ [2,head] auth v=5527265 cv=0/0 dir_auth=1 > state=1073741824 f(v0 m2019-08-14 16:39:17.790395 4=1+3) n(v84855 > rc2019-09-17 08:54:57.569803 b3226894326662 5255834=4707755+548079) > hs=1+0,ss=0+0 | child=1 subtree=1 0x5608a02e7900] > 2020-01-06 11:59:49.323 7f30149e4700  7 mds.1.cache adjust_subtree_auth > -1,-2 -> -2,-2 on [dir 0x1000ae4a784 /XXX/XXX/ [2,head] auth v=114 > cv=0/0 state=1073741824 f(v0 m2019-08-23 05:07:32.658490 9=9+0) n(v1 > rc2019-09-16 15:51:58.418555 b21646377 9=9+0) hs=0+0,ss=0+0 0x5608c602cd00] > 2020-01-06 11:59:49.323 7f30149e4700  7 mds.1.cache  current root is > [dir 0x10000073682 /XXX/XXX/ [2,head] auth v=5527265 cv=0/0 dir_auth=1 > state=1073741824 f(v0 m2019-08-14 16:39:17.790395 4=1+3) n(v84855 > rc2019-09-17 08:54:57.569803 b3226894326662 5255834=4707755+548079) > hs=1+0,ss=0+0 | child=1 subtree=1 0x5608a02e7900] > 2020-01-06 11:59:49.343 7f30149e4700  7 mds.1.cache adjust_subtree_auth > -1,-2 -> -2,-2 on [dir 0x1000ae4a78b /XXX/XXX/ [2,head] auth v=102 > cv=0/0 state=1073741824 f(v0 m2019-08-23 05:07:35.046498 9=9+0) n(v1 > rc2019-09-16 15:51:58.478556 b1430317 9=9+0) hs=0+0,ss=0+0 0x5608c602d200] > 2020-01-06 11:59:49.343 7f30149e4700  7 mds.1.cache  current root is > [dir 0x10000073682 /XXX/XXX/ [2,head] auth v=5527265 cv=0/0 dir_auth=1 > state=1073741824 f(v0 m2019-08-14 16:39:17.790395 4=1+3) n(v84855 > rc2019-09-17 08:54:57.569803 b3226894326662 5255834=4707755+548079) > hs=1+0,ss=0+0 | child=1 subtree=1 0x5608a02e7900] > 2020-01-06 11:59:49.363 7f30149e4700  7 mds.1.cache adjust_subtree_auth > -1,-2 -> -2,-2 on [dir 0x1000ae4a78e /XXX/XXX/ [2,head] auth v=91 cv=0/0 > state=1073741824 f(v0 m2019-08-23 05:07:38.986513 8=8+0) n(v1 > rc2019-09-16 15:51:58.498556 b1932614 8=8+0) hs=0+0,ss=0+0 0x5608c602d700] > 2020-01-06 11:59:49.363 7f30149e4700  7 mds.1.cache  current root is > [dir 0x10000073682 /XXX/XXX/ [2,head] auth v=5527265 cv=0/0 dir_auth=1 > state=1073741824 f(v0 m2019-08-14 16:39:17.790395 4=1+3) n(v84855 > rc2019-09-17 08:54:57.569803 b3226894326662 5255834=4707755+548079) > hs=1+0,ss=0+0 | child=1 subtree=1 0x5608a02e7900] > > Is there any way I can recover the MDS? I tried wiping sessions on > startup etc., but nothing worked. > > Thanks > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io