Hello Eugen and all, Thanks for the reply. We’ve checked the SuSE doc before raising it twice. From 100k to 125k, then to 150k. We are a bit worried about the continuous growth of strays at 50K a day and would like to find an effective to reduce the strays. Last night another 30K increase in the strays. Thanks Jinfeng From: Eugen Block <eblock@nde.ag> Date: Sunday, 16 February 2025 at 7:32 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: How to reduce CephFS num_strays effectively? ⚠ EXTERNAL EMAIL: Do not click links or open any attachments unless you trust the sender and know the content is safe. ⚠ Hi, this SUSE article [0] covers that, it helped us with a customer a few years ago. The recommendation was to double the mds_bal_fragment_size_max (default 100k) to 200k, which worked nicely for them. Also note the mentioned correlation between mds_bal_fragment_size_max and mds_cache_memory_limit. Regards, Eugen [0] https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.suse.com%2Fde-de%2Fsupport%2Fkb%2Fdoc%2F%3Fid%3D000020569&data=05%7C02%7Cjinfeng.biao%40cba.com.au%7Cd2558d1e23384b02c9f308dd4e6cc82a%7Cdddffba06c174f3497483fa5e08cc366%7C0%7C0%7C638752951318780733%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GXn4vSVpmchKLA3IWFZx0HTTRlo%2FHHimg82inRUnHy4%3D&reserved=0<https://www.suse.com/de-de/support/kb/doc/?id=000020569> Zitat von jinfeng.biao@cba.com.au:

Hello folks,

We had an issue with the num_strays hit 1 million recently. As a workaround, max bal was increased to 125,000.

The stray_num keeps growing at 25k per day. After a recent observation of 10TiB file deletion, the relevant application was stopped.

Then we increased purging options to below values

mds advanced filer_max_purge_ops 40 mds advanced mds_max_purge_files 1024 mds advanced mds_max_purge_ops 32768 mds advanced mds_max_purge_ops_per_pg 3

And run "du -hsx" to the top level directory mounted to the app that does massive deletion.

Despite all above, strays still growing at 60K per day.

There are a lot more applications using this CephFS filesystem and only this app is observed perform deletion at this scale.

I'm wondering what would be the effective way to cleanup the strays in this situation while making the least impact to production.

Note: We are on 14.2.6

thanks James Biao _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ************** IMPORTANT MESSAGE ************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124 AFSL and Australian credit licence 234945) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Opt Out in the subject line. **************************************************

Hi, On 18.02.25 01:00, Jinfeng Biao wrote:

Hello Eugen and all,

Thanks for the reply. We’ve checked the SuSE doc before raising it twice. From 100k to 125k, then to 150k.

We are a bit worried about the continuous growth of strays at 50K a day and would like to find an effective to reduce the strays.

Last night another 30K increase in the strays.

Just two comments: 1. Your ceph release is ancient and EOL. You should upgrade as soon as possible. Newer release also support more than 1 million strays. 2. The main source for strays are pending deleted files. If your workload is creating and deleting files over and over, you should adjust your workload (e.g. use local disk space for temporary files). But strays can also be created by _hardlinks_. And a number of widely used scientific workflow systems tend to use hardlinks for a lot of operations. You can try to reintegrate these strays e.g. by running a recursive find on the filesystem or scrub it (https://docs.ceph.com/en/latest/cephfs/scrub/). I would recommend to upgrade before running any of these checks. Best regards, Burkhard Linke