cephfs: massive drop in MDS requests per second with increasing number of caps
Hi all, we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec. When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps. See the attached screenshots. When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again. When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client. Can someone comment on this and explain it? How can this be solved, so that the performance remains stable? We are running ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) on all ceph cluster nodes and on all clients. The OS on all ceph cluster nodes and client nodes is CentOS 7.9. The filesystem is mounted via CentOS kernel client (latest official version). Thanks in advance. ~Best Dietmar -- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
Hi, On 1/18/21 5:46 PM, Dietmar Rieder wrote:
Hi all,
we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec.
When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps.
See the attached screenshots.
When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again.
When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client.
Can someone comment on this and explain it? How can this be solved, so that the performance remains stable?
The MDS has a (soft) limit for number of caps per client. If a clients starts to requests more caps, the MDS will ask it to release caps. This will add an extra network round trip, thus increasing processing time. The setting is 'mds_max_caps_per_client'. The default value is 1 million caps per client, but maybe this setting was changed in our configuration or the overall cap limit for the MDS is restricting it. Since each assigned cap increases the memory consumption of the MDS, setting an upper limit helps to control the overall amount of memory the MDS is using. So the memory target also affects the number of active caps an MDS can manage. You need to adjust both values to your use case. I would also recommend to monitor the cap usafe of the MDS, e.g. by running 'ceph daemonperf mds.<mds name>' in a shell on the mds server. Other methods using the various monitoring interfaces provided by ceph are also possible. There are also settings that control how fast a client is releasing caps for files; maybe tweaking these settings on the client side may also help in your case. Regards, Burkhard
Hi Burkhard, thanks so much for the quick reply and the explanation and suggestions. I'll check these settings and eventually change them and report back. Best Dietmar On 1/18/21 6:00 PM, Burkhard Linke wrote:
Hi,
On 1/18/21 5:46 PM, Dietmar Rieder wrote:
Hi all,
we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec.
When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps.
See the attached screenshots.
When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again.
When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client.
Can someone comment on this and explain it? How can this be solved, so that the performance remains stable?
The MDS has a (soft) limit for number of caps per client. If a clients starts to requests more caps, the MDS will ask it to release caps. This will add an extra network round trip, thus increasing processing time. The setting is 'mds_max_caps_per_client'. The default value is 1 million caps per client, but maybe this setting was changed in our configuration or the overall cap limit for the MDS is restricting it.
Since each assigned cap increases the memory consumption of the MDS, setting an upper limit helps to control the overall amount of memory the MDS is using. So the memory target also affects the number of active caps an MDS can manage. You need to adjust both values to your use case.
I would also recommend to monitor the cap usafe of the MDS, e.g. by running 'ceph daemonperf mds.<mds name>' in a shell on the mds server. Other methods using the various monitoring interfaces provided by ceph are also possible.
There are also settings that control how fast a client is releasing caps for files; maybe tweaking these settings on the client side may also help in your case.
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
Hi Burkhard, hi list, I checked the 'mds_max_caps_per_client' setting and it turned out that it was set to the default value of 1 million. The 'mds_cache_memory_limit' setting, however I had previously set to 40GB. Given this, I now started to play around with the max_caps and set 'mds_max_caps_per_client' 64k: # ceph config set mds mds_max_caps_per_client 65536 And this resulted in to a much better an stable performance of ~1.4k req/seq from one client and ~2.9k req/sec when running 2 clients in parallel. Remember it was max ~660 req/sec before using the 1M default and it gradually decreased to ~60 req/sec after some minute never getting higher again unless manually dropping the dentries and inodes from the vm cache on the client. (I guess this is because only 5000 caps are recalled after reaching the mds_max_caps_per_client) I'll keep this for now and observe if it has any impact on other operations or situations. Still I wonder why a higher number (i.e. >64k) of caps on the client destroys the performance completely. Thanks again Dietmar On 1/18/21 6:20 PM, Dietmar Rieder wrote:
Hi Burkhard,
thanks so much for the quick reply and the explanation and suggestions. I'll check these settings and eventually change them and report back.
Best Dietmar
On 1/18/21 6:00 PM, Burkhard Linke wrote:
Hi,
On 1/18/21 5:46 PM, Dietmar Rieder wrote:
Hi all,
we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec.
When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps.
See the attached screenshots.
When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again.
When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client.
Can someone comment on this and explain it? How can this be solved, so that the performance remains stable?
The MDS has a (soft) limit for number of caps per client. If a clients starts to requests more caps, the MDS will ask it to release caps. This will add an extra network round trip, thus increasing processing time. The setting is 'mds_max_caps_per_client'. The default value is 1 million caps per client, but maybe this setting was changed in our configuration or the overall cap limit for the MDS is restricting it.
Since each assigned cap increases the memory consumption of the MDS, setting an upper limit helps to control the overall amount of memory the MDS is using. So the memory target also affects the number of active caps an MDS can manage. You need to adjust both values to your use case.
I would also recommend to monitor the cap usafe of the MDS, e.g. by running 'ceph daemonperf mds.<mds name>' in a shell on the mds server. Other methods using the various monitoring interfaces provided by ceph are also possible.
There are also settings that control how fast a client is releasing caps for files; maybe tweaking these settings on the client side may also help in your case.
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
Hi Dietmar, thanks for discovering this. I also observed in the past that clients can become unbearably slow for no apparent reason. I never managed to reproduce this and, therefore, didn't report it here. A question about setting these flags on an existing mount. Will a "mount -o remount /mnt/cephfs" update client settings from the cluster without interrupting I/O? I couldn't find anything regarding updating config settings in the manual pages. I would be most interested in further updates in this matter and also if you find other flags with positive performance impact. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dietmar Rieder <dietmar.rieder@i-med.ac.at> Sent: 18 January 2021 21:01:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: cephfs: massive drop in MDS requests per second with increasing number of caps Hi Burkhard, hi list, I checked the 'mds_max_caps_per_client' setting and it turned out that it was set to the default value of 1 million. The 'mds_cache_memory_limit' setting, however I had previously set to 40GB. Given this, I now started to play around with the max_caps and set 'mds_max_caps_per_client' 64k: # ceph config set mds mds_max_caps_per_client 65536 And this resulted in to a much better an stable performance of ~1.4k req/seq from one client and ~2.9k req/sec when running 2 clients in parallel. Remember it was max ~660 req/sec before using the 1M default and it gradually decreased to ~60 req/sec after some minute never getting higher again unless manually dropping the dentries and inodes from the vm cache on the client. (I guess this is because only 5000 caps are recalled after reaching the mds_max_caps_per_client) I'll keep this for now and observe if it has any impact on other operations or situations. Still I wonder why a higher number (i.e. >64k) of caps on the client destroys the performance completely. Thanks again Dietmar On 1/18/21 6:20 PM, Dietmar Rieder wrote:
Hi Burkhard,
thanks so much for the quick reply and the explanation and suggestions. I'll check these settings and eventually change them and report back.
Best Dietmar
On 1/18/21 6:00 PM, Burkhard Linke wrote:
Hi,
On 1/18/21 5:46 PM, Dietmar Rieder wrote:
Hi all,
we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec.
When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps.
See the attached screenshots.
When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again.
When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client.
Can someone comment on this and explain it? How can this be solved, so that the performance remains stable?
The MDS has a (soft) limit for number of caps per client. If a clients starts to requests more caps, the MDS will ask it to release caps. This will add an extra network round trip, thus increasing processing time. The setting is 'mds_max_caps_per_client'. The default value is 1 million caps per client, but maybe this setting was changed in our configuration or the overall cap limit for the MDS is restricting it.
Since each assigned cap increases the memory consumption of the MDS, setting an upper limit helps to control the overall amount of memory the MDS is using. So the memory target also affects the number of active caps an MDS can manage. You need to adjust both values to your use case.
I would also recommend to monitor the cap usafe of the MDS, e.g. by running 'ceph daemonperf mds.<mds name>' in a shell on the mds server. Other methods using the various monitoring interfaces provided by ceph are also possible.
There are also settings that control how fast a client is releasing caps for files; maybe tweaking these settings on the client side may also help in your case.
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
Hi Frank, you don't need to remount the fs. The kernel driver should react to the change on the MDS. Best Dietmar On 1/19/21 9:07 AM, Frank Schilder wrote:
Hi Dietmar,
thanks for discovering this. I also observed in the past that clients can become unbearably slow for no apparent reason. I never managed to reproduce this and, therefore, didn't report it here.
A question about setting these flags on an existing mount. Will a "mount -o remount /mnt/cephfs" update client settings from the cluster without interrupting I/O? I couldn't find anything regarding updating config settings in the manual pages.
I would be most interested in further updates in this matter and also if you find other flags with positive performance impact.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dietmar Rieder <dietmar.rieder@i-med.ac.at> Sent: 18 January 2021 21:01:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: cephfs: massive drop in MDS requests per second with increasing number of caps
Hi Burkhard, hi list,
I checked the 'mds_max_caps_per_client' setting and it turned out that it was set to the default value of 1 million. The 'mds_cache_memory_limit' setting, however I had previously set to 40GB.
Given this, I now started to play around with the max_caps and set 'mds_max_caps_per_client' 64k:
# ceph config set mds mds_max_caps_per_client 65536
And this resulted in to a much better an stable performance of ~1.4k req/seq from one client and ~2.9k req/sec when running 2 clients in parallel. Remember it was max ~660 req/sec before using the 1M default and it gradually decreased to ~60 req/sec after some minute never getting higher again unless manually dropping the dentries and inodes from the vm cache on the client. (I guess this is because only 5000 caps are recalled after reaching the mds_max_caps_per_client)
I'll keep this for now and observe if it has any impact on other operations or situations.
Still I wonder why a higher number (i.e. >64k) of caps on the client destroys the performance completely.
Thanks again Dietmar
On 1/18/21 6:20 PM, Dietmar Rieder wrote:
Hi Burkhard,
thanks so much for the quick reply and the explanation and suggestions. I'll check these settings and eventually change them and report back.
Best Dietmar
On 1/18/21 6:00 PM, Burkhard Linke wrote:
Hi,
On 1/18/21 5:46 PM, Dietmar Rieder wrote:
Hi all,
we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec.
When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps.
See the attached screenshots.
When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again.
When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client.
Can someone comment on this and explain it? How can this be solved, so that the performance remains stable?
The MDS has a (soft) limit for number of caps per client. If a clients starts to requests more caps, the MDS will ask it to release caps. This will add an extra network round trip, thus increasing processing time. The setting is 'mds_max_caps_per_client'. The default value is 1 million caps per client, but maybe this setting was changed in our configuration or the overall cap limit for the MDS is restricting it.
Since each assigned cap increases the memory consumption of the MDS, setting an upper limit helps to control the overall amount of memory the MDS is using. So the memory target also affects the number of active caps an MDS can manage. You need to adjust both values to your use case.
I would also recommend to monitor the cap usafe of the MDS, e.g. by running 'ceph daemonperf mds.<mds name>' in a shell on the mds server. Other methods using the various monitoring interfaces provided by ceph are also possible.
There are also settings that control how fast a client is releasing caps for files; maybe tweaking these settings on the client side may also help in your case.
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
Hi Dietmar, thanks for that. I reduced the value and, indeed, the number of caps clients were holding started going down. A question about the particular value of 64K. Did you run several tests and found this one to be optimal, or was it just a lucky guess? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dietmar Rieder <dietmar.rieder@i-med.ac.at> Sent: 19 January 2021 13:24:15 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: cephfs: massive drop in MDS requests per second with increasing number of caps Hi Frank, you don't need to remount the fs. The kernel driver should react to the change on the MDS. Best Dietmar On 1/19/21 9:07 AM, Frank Schilder wrote:
Hi Dietmar,
thanks for discovering this. I also observed in the past that clients can become unbearably slow for no apparent reason. I never managed to reproduce this and, therefore, didn't report it here.
A question about setting these flags on an existing mount. Will a "mount -o remount /mnt/cephfs" update client settings from the cluster without interrupting I/O? I couldn't find anything regarding updating config settings in the manual pages.
I would be most interested in further updates in this matter and also if you find other flags with positive performance impact.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dietmar Rieder <dietmar.rieder@i-med.ac.at> Sent: 18 January 2021 21:01:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: cephfs: massive drop in MDS requests per second with increasing number of caps
Hi Burkhard, hi list,
I checked the 'mds_max_caps_per_client' setting and it turned out that it was set to the default value of 1 million. The 'mds_cache_memory_limit' setting, however I had previously set to 40GB.
Given this, I now started to play around with the max_caps and set 'mds_max_caps_per_client' 64k:
# ceph config set mds mds_max_caps_per_client 65536
And this resulted in to a much better an stable performance of ~1.4k req/seq from one client and ~2.9k req/sec when running 2 clients in parallel. Remember it was max ~660 req/sec before using the 1M default and it gradually decreased to ~60 req/sec after some minute never getting higher again unless manually dropping the dentries and inodes from the vm cache on the client. (I guess this is because only 5000 caps are recalled after reaching the mds_max_caps_per_client)
I'll keep this for now and observe if it has any impact on other operations or situations.
Still I wonder why a higher number (i.e. >64k) of caps on the client destroys the performance completely.
Thanks again Dietmar
On 1/18/21 6:20 PM, Dietmar Rieder wrote:
Hi Burkhard,
thanks so much for the quick reply and the explanation and suggestions. I'll check these settings and eventually change them and report back.
Best Dietmar
On 1/18/21 6:00 PM, Burkhard Linke wrote:
Hi,
On 1/18/21 5:46 PM, Dietmar Rieder wrote:
Hi all,
we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec.
When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps.
See the attached screenshots.
When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again.
When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client.
Can someone comment on this and explain it? How can this be solved, so that the performance remains stable?
The MDS has a (soft) limit for number of caps per client. If a clients starts to requests more caps, the MDS will ask it to release caps. This will add an extra network round trip, thus increasing processing time. The setting is 'mds_max_caps_per_client'. The default value is 1 million caps per client, but maybe this setting was changed in our configuration or the overall cap limit for the MDS is restricting it.
Since each assigned cap increases the memory consumption of the MDS, setting an upper limit helps to control the overall amount of memory the MDS is using. So the memory target also affects the number of active caps an MDS can manage. You need to adjust both values to your use case.
I would also recommend to monitor the cap usafe of the MDS, e.g. by running 'ceph daemonperf mds.<mds name>' in a shell on the mds server. Other methods using the various monitoring interfaces provided by ceph are also possible.
There are also settings that control how fast a client is releasing caps for files; maybe tweaking these settings on the client side may also help in your case.
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
Hi Frank, yes I ran several tests going down from the default 1M, 256k, 128k, 96k to 64k which seemed to be opimal in our case. ~Best Dietmar On 1/20/21 1:01 PM, Frank Schilder wrote:
Hi Dietmar,
thanks for that. I reduced the value and, indeed, the number of caps clients were holding started going down.
A question about the particular value of 64K. Did you run several tests and found this one to be optimal, or was it just a lucky guess?
Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dietmar Rieder <dietmar.rieder@i-med.ac.at> Sent: 19 January 2021 13:24:15 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: cephfs: massive drop in MDS requests per second with increasing number of caps
Hi Frank,
you don't need to remount the fs. The kernel driver should react to the change on the MDS.
Best Dietmar
On 1/19/21 9:07 AM, Frank Schilder wrote:
Hi Dietmar,
thanks for discovering this. I also observed in the past that clients can become unbearably slow for no apparent reason. I never managed to reproduce this and, therefore, didn't report it here.
A question about setting these flags on an existing mount. Will a "mount -o remount /mnt/cephfs" update client settings from the cluster without interrupting I/O? I couldn't find anything regarding updating config settings in the manual pages.
I would be most interested in further updates in this matter and also if you find other flags with positive performance impact.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dietmar Rieder <dietmar.rieder@i-med.ac.at> Sent: 18 January 2021 21:01:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: cephfs: massive drop in MDS requests per second with increasing number of caps
Hi Burkhard, hi list,
I checked the 'mds_max_caps_per_client' setting and it turned out that it was set to the default value of 1 million. The 'mds_cache_memory_limit' setting, however I had previously set to 40GB.
Given this, I now started to play around with the max_caps and set 'mds_max_caps_per_client' 64k:
# ceph config set mds mds_max_caps_per_client 65536
And this resulted in to a much better an stable performance of ~1.4k req/seq from one client and ~2.9k req/sec when running 2 clients in parallel. Remember it was max ~660 req/sec before using the 1M default and it gradually decreased to ~60 req/sec after some minute never getting higher again unless manually dropping the dentries and inodes from the vm cache on the client. (I guess this is because only 5000 caps are recalled after reaching the mds_max_caps_per_client)
I'll keep this for now and observe if it has any impact on other operations or situations.
Still I wonder why a higher number (i.e. >64k) of caps on the client destroys the performance completely.
Thanks again Dietmar
On 1/18/21 6:20 PM, Dietmar Rieder wrote:
Hi Burkhard,
thanks so much for the quick reply and the explanation and suggestions. I'll check these settings and eventually change them and report back.
Best Dietmar
On 1/18/21 6:00 PM, Burkhard Linke wrote:
Hi,
On 1/18/21 5:46 PM, Dietmar Rieder wrote:
Hi all,
we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec.
When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps.
See the attached screenshots.
When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again.
When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client.
Can someone comment on this and explain it? How can this be solved, so that the performance remains stable?
The MDS has a (soft) limit for number of caps per client. If a clients starts to requests more caps, the MDS will ask it to release caps. This will add an extra network round trip, thus increasing processing time. The setting is 'mds_max_caps_per_client'. The default value is 1 million caps per client, but maybe this setting was changed in our configuration or the overall cap limit for the MDS is restricting it.
Since each assigned cap increases the memory consumption of the MDS, setting an upper limit helps to control the overall amount of memory the MDS is using. So the memory target also affects the number of active caps an MDS can manage. You need to adjust both values to your use case.
I would also recommend to monitor the cap usafe of the MDS, e.g. by running 'ceph daemonperf mds.<mds name>' in a shell on the mds server. Other methods using the various monitoring interfaces provided by ceph are also possible.
There are also settings that control how fast a client is releasing caps for files; maybe tweaking these settings on the client side may also help in your case.
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
Hi Dietmar, thanks for that info. I repeated a benchmark test we were using when trying to find out what the problem is. Its un-taring an anaconda2 archive, which produces a high mixed load on a file system. I remember that it used to take ca. 4 minutes on a freshly mounted client. After reducing the cache value, it takes only 2 minutes. The reason we used this as a benchmark is, that a user observed that the extraction stalls after a while. Back then we couldn't reliably reproduce the problem, so we didn't really know what to look for. It was really strange. When I did it as myself, it was fast. When I did it as the other user on the same server, it stalled. It usually happened around the same place when hard-links were created. The ceph documentation mentioned that hard links are a challenge and we didn't have time to dig more into it. I don't have any observations about possible slow-downs as seen before, but its only a day in production for now. I hope the reduction doesn't have any negative side effects. So far, I didn't receive any complaints. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dietmar Rieder <dietmar.rieder@i-med.ac.at> Sent: 20 January 2021 14:18:19 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: cephfs: massive drop in MDS requests per second with increasing number of caps Hi Frank, yes I ran several tests going down from the default 1M, 256k, 128k, 96k to 64k which seemed to be opimal in our case. ~Best Dietmar On 1/20/21 1:01 PM, Frank Schilder wrote:
Hi Dietmar,
thanks for that. I reduced the value and, indeed, the number of caps clients were holding started going down.
A question about the particular value of 64K. Did you run several tests and found this one to be optimal, or was it just a lucky guess?
Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dietmar Rieder <dietmar.rieder@i-med.ac.at> Sent: 19 January 2021 13:24:15 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: cephfs: massive drop in MDS requests per second with increasing number of caps
Hi Frank,
you don't need to remount the fs. The kernel driver should react to the change on the MDS.
Best Dietmar
On 1/19/21 9:07 AM, Frank Schilder wrote:
Hi Dietmar,
thanks for discovering this. I also observed in the past that clients can become unbearably slow for no apparent reason. I never managed to reproduce this and, therefore, didn't report it here.
A question about setting these flags on an existing mount. Will a "mount -o remount /mnt/cephfs" update client settings from the cluster without interrupting I/O? I couldn't find anything regarding updating config settings in the manual pages.
I would be most interested in further updates in this matter and also if you find other flags with positive performance impact.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dietmar Rieder <dietmar.rieder@i-med.ac.at> Sent: 18 January 2021 21:01:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: cephfs: massive drop in MDS requests per second with increasing number of caps
Hi Burkhard, hi list,
I checked the 'mds_max_caps_per_client' setting and it turned out that it was set to the default value of 1 million. The 'mds_cache_memory_limit' setting, however I had previously set to 40GB.
Given this, I now started to play around with the max_caps and set 'mds_max_caps_per_client' 64k:
# ceph config set mds mds_max_caps_per_client 65536
And this resulted in to a much better an stable performance of ~1.4k req/seq from one client and ~2.9k req/sec when running 2 clients in parallel. Remember it was max ~660 req/sec before using the 1M default and it gradually decreased to ~60 req/sec after some minute never getting higher again unless manually dropping the dentries and inodes from the vm cache on the client. (I guess this is because only 5000 caps are recalled after reaching the mds_max_caps_per_client)
I'll keep this for now and observe if it has any impact on other operations or situations.
Still I wonder why a higher number (i.e. >64k) of caps on the client destroys the performance completely.
Thanks again Dietmar
On 1/18/21 6:20 PM, Dietmar Rieder wrote:
Hi Burkhard,
thanks so much for the quick reply and the explanation and suggestions. I'll check these settings and eventually change them and report back.
Best Dietmar
On 1/18/21 6:00 PM, Burkhard Linke wrote:
Hi,
On 1/18/21 5:46 PM, Dietmar Rieder wrote:
Hi all,
we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec.
When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps.
See the attached screenshots.
When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again.
When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client.
Can someone comment on this and explain it? How can this be solved, so that the performance remains stable?
The MDS has a (soft) limit for number of caps per client. If a clients starts to requests more caps, the MDS will ask it to release caps. This will add an extra network round trip, thus increasing processing time. The setting is 'mds_max_caps_per_client'. The default value is 1 million caps per client, but maybe this setting was changed in our configuration or the overall cap limit for the MDS is restricting it.
Since each assigned cap increases the memory consumption of the MDS, setting an upper limit helps to control the overall amount of memory the MDS is using. So the memory target also affects the number of active caps an MDS can manage. You need to adjust both values to your use case.
I would also recommend to monitor the cap usafe of the MDS, e.g. by running 'ceph daemonperf mds.<mds name>' in a shell on the mds server. Other methods using the various monitoring interfaces provided by ceph are also possible.
There are also settings that control how fast a client is releasing caps for files; maybe tweaking these settings on the client side may also help in your case.
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
-- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
participants (3)
-
Burkhard Linke
-
Dietmar Rieder
-
Frank Schilder