1 clients failing to respond to cache pressure (quincy:17.2.6)
Hello. I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning. I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool. My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data) --------------------------------------------------------------------------------- --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB --------------------------------------------------------------------------------- root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) ----------------------------------------------------------------------------------- My MDS settings are below: mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144 I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ? - Thanks
Hi, I have dealt with this topic multiple times, the SUSE team helped understanding what's going on under the hood. The summary can be found in this thread [1]. What helped in our case was to reduce the mds_recall_max_caps from 30k (default) to 3k. We tried it in steps of 1k IIRC. So I suggest to reduce that value step by step (maybe start with 20k or something) to find the optimal value. Regards, Eugen [1] https://www.spinics.net/lists/ceph-users/msg73188.html Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello.
I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning.
I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool.
My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data)
--------------------------------------------------------------------------------- --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB --------------------------------------------------------------------------------- root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
----------------------------------------------------------------------------------- My MDS settings are below:
mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144
I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ?
- Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Eugen. Thank you for the answer. According to knowledge and test results at this issue: https://github.com/ceph/ceph/pull/38574 I've tried their advice and I've applied the following changes. max_mds = 4 standby_mds = 1 mds_cache_memory_limit = 16GB mds_recall_max_caps = 40000 When I set these parameters, 1 day later I saw this log: [8531248.982954] Out of memory: Killed process 1580586 (ceph-mds) total-vm:70577592kB, anon-rss:70244236kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:137832kB oom_score_adj:0 All the MDS services created memory leak and killed by kernel. Because of this I changed it as below and it is stable now but performance is very poor and I still get cache pressure alerts. max_mds = 1 standby_mds = 5 mds_cache_memory_limit = 8GB mds_recall_max_caps = 30000 I'm very surprised that you are advising to decrease "mds_recall_max_caps" because it is the opposite of what developers advised in the issue I've sended. It is very hard to play around with MDS parameters without expert level of understanding what these parameters stands for and how it will effect the behavior. Because of this I'm trying to understand the MDS code flow and I'm very interested with learning more and tuning my system by debugging and understanding my own data flow and MDS usage. I have a very unique data flow and I think I need to configure the system for this case. I have 80+ clients and via all of these clients my users are requesting Read a range of objects and compare them in GPU, they generate new data and Write the new data back in the cluster. So it means my clients usually reads objects only one time and do not read the same object again. Sometimes same user runs multiple service in multiple clients and these services can read the same data from different clients. So having a large cache is useless for my use case. I need to setup MDS and Cephfs Client for this data flow. When I debug the MDS ram usage, I see high allocation all the time and I wonder why? If any of my client does not read any object why MDS does not remove that data from ram allocation? I need to configure MDS for reading the data and removing it very fast if the data is constantly requested from clients. In this case ofc I want a ram cache tier. I'm little confused and I need to learn more about how MDS works and how should I make multiple active MDS faster for my subvolumes and client data flow. Best regards. Eugen Block <eblock@nde.ag>, 16 Oca 2024 Sal, 11:36 tarihinde şunu yazdı:
Hi,
I have dealt with this topic multiple times, the SUSE team helped understanding what's going on under the hood. The summary can be found in this thread [1].
What helped in our case was to reduce the mds_recall_max_caps from 30k (default) to 3k. We tried it in steps of 1k IIRC. So I suggest to reduce that value step by step (maybe start with 20k or something) to find the optimal value.
Regards, Eugen
[1] https://www.spinics.net/lists/ceph-users/msg73188.html
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello.
I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning.
I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool.
My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data)
---------------------------------------------------------------------------------
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB
---------------------------------------------------------------------------------
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
-----------------------------------------------------------------------------------
My MDS settings are below:
mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144
I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ?
- Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Let me share some outputs about my cluster. root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 31 /s 3022k 3021k 52.6k 385k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.4T cephfs.ud-data.data data 45.2T 44.4T STANDBY MDS ud-data.ud-03.lhwkml ud-data.ud-05.rnhcfe ud-data.ud-01.uatjle ud-data.ud-04.seggyv -------------------------------------------------------------------------- This is "ceph tell mds.ud-data.ud-02.xcoojt session ls" output for the reported client for cache pressure warning. { "id": 1282205, "entity": { "name": { "type": "client", "num": 1282205 }, "addr": { "type": "v1", "addr": "172.16.3.48:0", "nonce": 2169935642 } }, "state": "open", "num_leases": 0, "num_caps": 52092, "request_load_avg": 1, "uptime": 75754.745608647994, "requests_in_flight": 0, "num_completed_requests": 0, "num_completed_flushes": 1, "reconnecting": false, "recall_caps": { "value": 2577232.0049106553, "halflife": 60 }, "release_caps": { "value": 1.4093491463510395, "halflife": 60 }, "recall_caps_throttle": { "value": 63733.985544098425, "halflife": 1.5 }, "recall_caps_throttle2o": { "value": 19452.428409271757, "halflife": 0.5 }, "session_cache_liveness": { "value": 14.100272208890081, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "delegated_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 } ], "inst": "client.1282205 v1:172.16.3.48:0/2169935642", "completed_requests": [], "prealloc_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 }, { "start": "0x1000fa86e5f", "length": 54 }, { "start": "0x1000faa069c", "length": 501 } ], "client_metadata": { "client_features": { "feature_bits": "0x0000000000007bff" }, "metric_spec": { "metric_flags": { "feature_bits": "0x00000000000003ff" } }, "entity_id": "admin", "hostname": "bennevis-2", "kernel_version": "5.15.0-91-generic", "root": "/volumes/babblians" } } Özkan Göksu <ozkangksu@gmail.com>, 17 Oca 2024 Çar, 07:22 tarihinde şunu yazdı:
Hello Eugen.
Thank you for the answer. According to knowledge and test results at this issue: https://github.com/ceph/ceph/pull/38574 I've tried their advice and I've applied the following changes.
max_mds = 4 standby_mds = 1 mds_cache_memory_limit = 16GB mds_recall_max_caps = 40000
When I set these parameters, 1 day later I saw this log: [8531248.982954] Out of memory: Killed process 1580586 (ceph-mds) total-vm:70577592kB, anon-rss:70244236kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:137832kB oom_score_adj:0
All the MDS services created memory leak and killed by kernel. Because of this I changed it as below and it is stable now but performance is very poor and I still get cache pressure alerts.
max_mds = 1 standby_mds = 5 mds_cache_memory_limit = 8GB mds_recall_max_caps = 30000
I'm very surprised that you are advising to decrease "mds_recall_max_caps" because it is the opposite of what developers advised in the issue I've sended. It is very hard to play around with MDS parameters without expert level of understanding what these parameters stands for and how it will effect the behavior. Because of this I'm trying to understand the MDS code flow and I'm very interested with learning more and tuning my system by debugging and understanding my own data flow and MDS usage.
I have a very unique data flow and I think I need to configure the system for this case. I have 80+ clients and via all of these clients my users are requesting Read a range of objects and compare them in GPU, they generate new data and Write the new data back in the cluster. So it means my clients usually reads objects only one time and do not read the same object again. Sometimes same user runs multiple service in multiple clients and these services can read the same data from different clients.
So having a large cache is useless for my use case. I need to setup MDS and Cephfs Client for this data flow. When I debug the MDS ram usage, I see high allocation all the time and I wonder why? If any of my client does not read any object why MDS does not remove that data from ram allocation? I need to configure MDS for reading the data and removing it very fast if the data is constantly requested from clients. In this case ofc I want a ram cache tier.
I'm little confused and I need to learn more about how MDS works and how should I make multiple active MDS faster for my subvolumes and client data flow.
Best regards.
Eugen Block <eblock@nde.ag>, 16 Oca 2024 Sal, 11:36 tarihinde şunu yazdı:
Hi,
I have dealt with this topic multiple times, the SUSE team helped understanding what's going on under the hood. The summary can be found in this thread [1].
What helped in our case was to reduce the mds_recall_max_caps from 30k (default) to 3k. We tried it in steps of 1k IIRC. So I suggest to reduce that value step by step (maybe start with 20k or something) to find the optimal value.
Regards, Eugen
[1] https://www.spinics.net/lists/ceph-users/msg73188.html
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello.
I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning.
I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool.
My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data)
---------------------------------------------------------------------------------
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB
---------------------------------------------------------------------------------
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
-----------------------------------------------------------------------------------
My MDS settings are below:
mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144
I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ?
- Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
All of my clients are servers located at 2 hop away with 10Gbit network and 2x Xeon CPU/16++ cores and minimum 64GB ram with SSD OS drive + 8GB spare. I use ceph kernel mount only and this is the command: - mount.ceph admin@$fsid.ud-data=/volumes/subvolumegroup ${MOUNT_DIR} -o name=admin,secret=XXX==,mon_addr=XXX I think all of my clients have enough resources to answer MDS requests very fast. The only possibility that any of my clients fails to respond to cache pressure is the default settings at cephfs client or MDS server. I have some problem with understanding how cephfs client works and why it needs communication with MDS server for managing local cache. And even at the beggining I didn't understand why MDS server needs direct control over clients and tell them what to do. My mind does not understand the concept and its logic. To me, clients must be independent and they must manage their data flow without any server side control. The client must send read and write request to the remote server and return answer to the kernel. Client can have read cache management future but it does not need communicate with remote server. When a client detects multiple read for the same object it should cache it with a set of protocols and release it when it needed. I don't understand why MDS needs to tell clients to release the allocation and why client needs to report the release status back... The logical answer for me is I think I'm looking from the wrong angle and this is not the cache that I know from block filesystems. With my use case, clients reads 50-100GB of data (10.000++ objects) only one or two times with each runtime in few hours. ------------------------------------------------------------------------------------ While I was researching, I saw that some users recommends decreasing "mds_max_caps_per_client" from 1M to 64K # ceph config set mds mds_max_caps_per_client 65536 But if you check the reported client ls at previous mail you will see "num_caps": 52092, for a failing client for cache pressure. So its even under 64K and I'm not sure changing this value can help or not. I want to repeat my main goal. I'm not trying to solve cache pressure warning. The ceph random read and write performance is not good and a lot of reads from 80+ clients creates latency. I'm trying to increase the speed by creating multiple MDS even maybe binding subvolumes to specific MDS servers and decrease the latency. Also when I check MDS CPU usage I see %120++ usage time to time. But when I check the server CPU load at MDS location, I see MDS only uses 2-4 cores and other CPU cores are almost at idle. I think MDS has a CPU core limitation and I need to increase the value to decrease the latency. How can I do that? Özkan Göksu <ozkangksu@gmail.com>, 17 Oca 2024 Çar, 07:44 tarihinde şunu yazdı:
Let me share some outputs about my cluster.
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 31 /s 3022k 3021k 52.6k 385k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.4T cephfs.ud-data.data data 45.2T 44.4T STANDBY MDS ud-data.ud-03.lhwkml ud-data.ud-05.rnhcfe ud-data.ud-01.uatjle ud-data.ud-04.seggyv
-------------------------------------------------------------------------- This is "ceph tell mds.ud-data.ud-02.xcoojt session ls" output for the reported client for cache pressure warning.
{ "id": 1282205, "entity": { "name": { "type": "client", "num": 1282205 }, "addr": { "type": "v1", "addr": "172.16.3.48:0", "nonce": 2169935642 } }, "state": "open", "num_leases": 0, "num_caps": 52092, "request_load_avg": 1, "uptime": 75754.745608647994, "requests_in_flight": 0, "num_completed_requests": 0, "num_completed_flushes": 1, "reconnecting": false, "recall_caps": { "value": 2577232.0049106553, "halflife": 60 }, "release_caps": { "value": 1.4093491463510395, "halflife": 60 }, "recall_caps_throttle": { "value": 63733.985544098425, "halflife": 1.5 }, "recall_caps_throttle2o": { "value": 19452.428409271757, "halflife": 0.5 }, "session_cache_liveness": { "value": 14.100272208890081, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "delegated_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 } ], "inst": "client.1282205 v1:172.16.3.48:0/2169935642", "completed_requests": [], "prealloc_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 }, { "start": "0x1000fa86e5f", "length": 54 }, { "start": "0x1000faa069c", "length": 501 } ], "client_metadata": { "client_features": { "feature_bits": "0x0000000000007bff" }, "metric_spec": { "metric_flags": { "feature_bits": "0x00000000000003ff" } }, "entity_id": "admin", "hostname": "bennevis-2", "kernel_version": "5.15.0-91-generic", "root": "/volumes/babblians" } }
Özkan Göksu <ozkangksu@gmail.com>, 17 Oca 2024 Çar, 07:22 tarihinde şunu yazdı:
Hello Eugen.
Thank you for the answer. According to knowledge and test results at this issue: https://github.com/ceph/ceph/pull/38574 I've tried their advice and I've applied the following changes.
max_mds = 4 standby_mds = 1 mds_cache_memory_limit = 16GB mds_recall_max_caps = 40000
When I set these parameters, 1 day later I saw this log: [8531248.982954] Out of memory: Killed process 1580586 (ceph-mds) total-vm:70577592kB, anon-rss:70244236kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:137832kB oom_score_adj:0
All the MDS services created memory leak and killed by kernel. Because of this I changed it as below and it is stable now but performance is very poor and I still get cache pressure alerts.
max_mds = 1 standby_mds = 5 mds_cache_memory_limit = 8GB mds_recall_max_caps = 30000
I'm very surprised that you are advising to decrease "mds_recall_max_caps" because it is the opposite of what developers advised in the issue I've sended. It is very hard to play around with MDS parameters without expert level of understanding what these parameters stands for and how it will effect the behavior. Because of this I'm trying to understand the MDS code flow and I'm very interested with learning more and tuning my system by debugging and understanding my own data flow and MDS usage.
I have a very unique data flow and I think I need to configure the system for this case. I have 80+ clients and via all of these clients my users are requesting Read a range of objects and compare them in GPU, they generate new data and Write the new data back in the cluster. So it means my clients usually reads objects only one time and do not read the same object again. Sometimes same user runs multiple service in multiple clients and these services can read the same data from different clients.
So having a large cache is useless for my use case. I need to setup MDS and Cephfs Client for this data flow. When I debug the MDS ram usage, I see high allocation all the time and I wonder why? If any of my client does not read any object why MDS does not remove that data from ram allocation? I need to configure MDS for reading the data and removing it very fast if the data is constantly requested from clients. In this case ofc I want a ram cache tier.
I'm little confused and I need to learn more about how MDS works and how should I make multiple active MDS faster for my subvolumes and client data flow.
Best regards.
Eugen Block <eblock@nde.ag>, 16 Oca 2024 Sal, 11:36 tarihinde şunu yazdı:
Hi,
I have dealt with this topic multiple times, the SUSE team helped understanding what's going on under the hood. The summary can be found in this thread [1].
What helped in our case was to reduce the mds_recall_max_caps from 30k (default) to 3k. We tried it in steps of 1k IIRC. So I suggest to reduce that value step by step (maybe start with 20k or something) to find the optimal value.
Regards, Eugen
[1] https://www.spinics.net/lists/ceph-users/msg73188.html
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello.
I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning.
I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool.
My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data)
---------------------------------------------------------------------------------
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB
---------------------------------------------------------------------------------
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
-----------------------------------------------------------------------------------
My MDS settings are below:
mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144
I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ?
- Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This my active MDS perf dump output: root@ud-01:~# ceph tell mds.ud-data.ud-02.xcoojt perf dump { "AsyncMessenger::Worker-0": { "msgr_recv_messages": 17179307, "msgr_send_messages": 15867134, "msgr_recv_bytes": 445239812294, "msgr_send_bytes": 42003529245, "msgr_created_connections": 96, "msgr_active_connections": 54, "msgr_running_total_time": 2640.694486182, "msgr_running_send_time": 716.588978014, "msgr_running_recv_time": 45572.840723683, "msgr_running_fast_dispatch_time": 898.208871568, "msgr_send_messages_queue_lat": { "avgcount": 15867133, "sum": 4104.943819792, "avgtime": 0.000258707 }, "msgr_handle_ack_lat": { "avgcount": 7464879, "sum": 4.939409831, "avgtime": 0.000000661 } }, "AsyncMessenger::Worker-1": { "msgr_recv_messages": 18116812, "msgr_send_messages": 17095685, "msgr_recv_bytes": 506282885037, "msgr_send_bytes": 55228114586, "msgr_created_connections": 197, "msgr_active_connections": 56, "msgr_running_total_time": 3012.176186383, "msgr_running_send_time": 814.143975507, "msgr_running_recv_time": 57814.469683537, "msgr_running_fast_dispatch_time": 993.179349828, "msgr_send_messages_queue_lat": { "avgcount": 17095673, "sum": 5137.869051226, "avgtime": 0.000300536 }, "msgr_handle_ack_lat": { "avgcount": 8803334, "sum": 6.183353371, "avgtime": 0.000000702 } }, "AsyncMessenger::Worker-2": { "msgr_recv_messages": 23067443, "msgr_send_messages": 20150693, "msgr_recv_bytes": 524358461533, "msgr_send_bytes": 124188927975, "msgr_created_connections": 136, "msgr_active_connections": 57, "msgr_running_total_time": 3732.249144012, "msgr_running_send_time": 1160.515966671, "msgr_running_recv_time": 4777.558043971, "msgr_running_fast_dispatch_time": 1037.227270644, "msgr_send_messages_queue_lat": { "avgcount": 20150689, "sum": 5507.000605725, "avgtime": 0.000273290 }, "msgr_handle_ack_lat": { "avgcount": 12133428, "sum": 10.398089696, "avgtime": 0.000000856 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "queue_len": 0, "complete_latency": { "avgcount": 7983999, "sum": 42368.199433194, "avgtime": 0.005306638 } }, "finisher-PurgeQueue": { "queue_len": 0, "complete_latency": { "avgcount": 1681586, "sum": 1814.673700573, "avgtime": 0.001079144 } }, "mds": { "request": 22046242, "reply": 22046200, "reply_latency": { "avgcount": 22046200, "sum": 485668.718167318, "avgtime": 0.022029588 }, "slow_reply": 0, "forward": 0, "dir_fetch": 8048670, "dir_commit": 162554, "dir_split": 1, "dir_merge": 5, "inodes": 3007665, "inodes_top": 1851907, "inodes_bottom": 1075947, "inodes_pin_tail": 79811, "inodes_pinned": 362083, "inodes_expired": 2390089349, "inodes_with_caps": 359488, "caps": 367045, "subtrees": 2, "traverse": 30835852, "traverse_hit": 28292209, "traverse_forward": 0, "traverse_discover": 0, "traverse_dir_fetch": 997199, "traverse_remote_ino": 4936, "traverse_lock": 14640, "load_cent": 15904796, "q": 0, "exported": 0, "exported_inodes": 0, "imported": 0, "imported_inodes": 0, "openino_dir_fetch": 392728, "openino_backtrace_fetch": 118146, "openino_peer_discover": 0, "root_rfiles": 96255545, "root_rbytes": 28616263717441, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_set_tag": 0, "scrub_backtrace_repaired": 0, "scrub_inotable_repaired": 0, "scrub_dir_inodes": 0, "scrub_dir_base_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "handle_inode_file_caps": 0, "ceph_cap_op_revoke": 1377821, "ceph_cap_op_grant": 10915803, "ceph_cap_op_trunc": 14858, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_flush_ack": 0, "handle_client_caps": 13064247, "handle_client_caps_dirty": 1447210, "handle_client_cap_release": 2377564, "process_request_cap_release": 9609997 }, "mds_cache": { "num_strays": 7, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 1132660, "strays_enqueued": 1129983, "strays_reintegrated": 10807, "strays_migrated": 0, "num_recovering_processing": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "recovery_started": 1, "recovery_completed": 1, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 6, "ireq_fragstats": 0, "ireq_inodestats": 0 }, "mds_log": { "evadd": 9429876, "evex": 9437748, "evtrm": 9437748, "ev": 88984, "evexg": 0, "evexd": 25, "segadd": 12165, "segex": 12165, "segtrm": 12165, "seg": 129, "segexg": 0, "segexd": 1, "expos": 12708408729837, "wrpos": 12708585242899, "rdpos": 12690823784258, "jlat": { "avgcount": 697759, "sum": 18206.880493144, "avgtime": 0.026093365 }, "replayed": 96856 }, "mds_mem": { "ino": 3006191, "ino+": 2391671062, "ino-": 2388664871, "dir": 56731, "dir+": 7146231, "dir-": 7089500, "dn": 3006389, "dn+": 2395253847, "dn-": 2392247458, "cap": 366021, "cap+": 394343412, "cap-": 393977391, "rss": 9974008, "heap": 223516 }, "mds_server": { "dispatch_client_request": 33305573, "dispatch_server_request": 0, "handle_client_request": 22046242, "handle_client_session": 546679, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1265353, "sum": 8092.372232781, "avgtime": 0.006395347 }, "req_getattr_latency": { "avgcount": 742761, "sum": 7466.177333377, "avgtime": 0.010051924 }, "req_getfilelock_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_link_latency": { "avgcount": 7159, "sum": 14.511870538, "avgtime": 0.002027080 }, "req_lookup_latency": { "avgcount": 6453655, "sum": 112979.615492227, "avgtime": 0.017506299 }, "req_lookuphash_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupino_latency": { "avgcount": 2, "sum": 0.000204966, "avgtime": 0.000102483 }, "req_lookupname_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupparent_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupsnap_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lssnap_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_mkdir_latency": { "avgcount": 105563, "sum": 576.465681718, "avgtime": 0.005460868 }, "req_mknod_latency": { "avgcount": 12, "sum": 0.019379782, "avgtime": 0.001614981 }, "req_mksnap_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_open_latency": { "avgcount": 1515084, "sum": 12869.581748984, "avgtime": 0.008494302 }, "req_readdir_latency": { "avgcount": 10525466, "sum": 336757.025480678, "avgtime": 0.031994500 }, "req_rename_latency": { "avgcount": 364696, "sum": 1200.921138094, "avgtime": 0.003292937 }, "req_renamesnap_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_rmdir_latency": { "avgcount": 98241, "sum": 2455.598234225, "avgtime": 0.024995655 }, "req_rmsnap_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_rmxattr_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setattr_latency": { "avgcount": 18921, "sum": 761.713863946, "avgtime": 0.040257590 }, "req_setdirlayout_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setfilelock_latency": { "avgcount": 26396, "sum": 72.242011848, "avgtime": 0.002736854 }, "req_setlayout_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setxattr_latency": { "avgcount": 17, "sum": 0.006095368, "avgtime": 0.000358551 }, "req_symlink_latency": { "avgcount": 5534, "sum": 19.856199425, "avgtime": 0.003588037 }, "req_unlink_latency": { "avgcount": 917340, "sum": 2402.611199361, "avgtime": 0.002619106 }, "cap_revoke_eviction": 0, "cap_acquisition_throttle": 0, "req_getvxattr_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "mds_sessions": { "session_count": 84, "session_add": 93, "session_remove": 9, "sessions_open": 84, "sessions_stale": 0, "total_load": 2949, "average_load": 35, "avg_session_uptime": 9327000 }, "mempool": { "bloom_filter_bytes": 83341685, "bloom_filter_items": 83341685, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_items": 0, "bluefs_bytes": 0, "bluefs_items": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "buffer_anon_bytes": 2114708, "buffer_anon_items": 825, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 25728, "osdmap_items": 946, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "mds_co_bytes": 8173443932, "mds_co_items": 109004579, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "op_active": 0, "op_laggy": 0, "op_send": 13563810, "op_send_bytes": 21613887606, "op_resend": 1, "op_reply": 13563809, "oplen_avg": { "avgcount": 13563809, "sum": 31377549 }, "op": 13563809, "op_r": 10213362, "op_w": 3350447, "op_rmw": 0, "op_pg": 0, "osdop_stat": 75945, "osdop_create": 1139381, "osdop_read": 15848, "osdop_write": 713549, "osdop_writefull": 20267, "osdop_writesame": 0, "osdop_append": 0, "osdop_zero": 2, "osdop_truncate": 0, "osdop_delete": 1226688, "osdop_mapext": 0, "osdop_sparse_read": 0, "osdop_clonerange": 0, "osdop_getxattr": 7321546, "osdop_setxattr": 2283499, "osdop_cmpxattr": 0, "osdop_rmxattr": 0, "osdop_resetxattrs": 0, "osdop_call": 0, "osdop_watch": 0, "osdop_notify": 0, "osdop_src_cmpxattr": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_other": 49342, "linger_active": 0, "linger_send": 0, "linger_resend": 0, "linger_ping": 0, "poolop_active": 0, "poolop_send": 0, "poolop_resend": 0, "poolstat_active": 0, "poolstat_send": 0, "poolstat_resend": 0, "statfs_active": 0, "statfs_send": 0, "statfs_resend": 0, "command_active": 0, "command_send": 0, "command_resend": 0, "map_epoch": 13646, "map_full": 0, "map_inc": 97, "osd_sessions": 80, "osd_session_open": 176, "osd_session_close": 96, "osd_laggy": 0, "omap_wr": 354624, "omap_rd": 18128035, "omap_del": 48823 }, "oft": { "omap_total_objs": 3, "omap_total_kv_pairs": 31549, "omap_total_updates": 9972364, "omap_total_removes": 7080093 }, "purge_queue": { "pq_executing_ops": 0, "pq_executing_ops_high_water": 1126, "pq_executing": 0, "pq_executing_high_water": 64, "pq_executed": 1129983, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "val": 0, "max": 104857600, "get_started": 0, "get": 58363558, "get_sum": 1471342573618, "get_or_fail_fail": 0, "get_or_fail_success": 58363558, "take": 0, "take_sum": 0, "put": 58363558, "put_sum": 1471342573618, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-objecter_bytes": { "val": 0, "max": 104857600, "get_started": 0, "get": 0, "get_sum": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "take": 13563809, "take_sum": 21700796870, "put": 13563809, "put_sum": 21700796870, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-objecter_ops": { "val": 0, "max": 1024, "get_started": 0, "get": 0, "get_sum": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "take": 13563809, "take_sum": 13563809, "put": 13563809, "put_sum": 13563809, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-write_buf_throttle": { "val": 0, "max": 3758096384, "get_started": 0, "get": 1129983, "get_sum": 114128283, "get_or_fail_fail": 0, "get_or_fail_success": 1129983, "take": 0, "take_sum": 0, "put": 15790, "put_sum": 114128283, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-write_buf_throttle-0x5569b6836500": { "val": 0, "max": 3758096384, "get_started": 0, "get": 9429876, "get_sum": 17761457237, "get_or_fail_fail": 0, "get_or_fail_success": 9429876, "take": 0, "take_sum": 0, "put": 697759, "put_sum": 17761457237, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } } } Özkan Göksu <ozkangksu@gmail.com>, 17 Oca 2024 Çar, 08:29 tarihinde şunu yazdı:
All of my clients are servers located at 2 hop away with 10Gbit network and 2x Xeon CPU/16++ cores and minimum 64GB ram with SSD OS drive + 8GB spare. I use ceph kernel mount only and this is the command: - mount.ceph admin@$fsid.ud-data=/volumes/subvolumegroup ${MOUNT_DIR} -o name=admin,secret=XXX==,mon_addr=XXX
I think all of my clients have enough resources to answer MDS requests very fast. The only possibility that any of my clients fails to respond to cache pressure is the default settings at cephfs client or MDS server.
I have some problem with understanding how cephfs client works and why it needs communication with MDS server for managing local cache. And even at the beggining I didn't understand why MDS server needs direct control over clients and tell them what to do. My mind does not understand the concept and its logic. To me, clients must be independent and they must manage their data flow without any server side control. The client must send read and write request to the remote server and return answer to the kernel. Client can have read cache management future but it does not need communicate with remote server. When a client detects multiple read for the same object it should cache it with a set of protocols and release it when it needed. I don't understand why MDS needs to tell clients to release the allocation and why client needs to report the release status back...
The logical answer for me is I think I'm looking from the wrong angle and this is not the cache that I know from block filesystems.
With my use case, clients reads 50-100GB of data (10.000++ objects) only one or two times with each runtime in few hours.
------------------------------------------------------------------------------------ While I was researching, I saw that some users recommends decreasing "mds_max_caps_per_client" from 1M to 64K # ceph config set mds mds_max_caps_per_client 65536
But if you check the reported client ls at previous mail you will see "num_caps": 52092, for a failing client for cache pressure. So its even under 64K and I'm not sure changing this value can help or not.
I want to repeat my main goal. I'm not trying to solve cache pressure warning. The ceph random read and write performance is not good and a lot of reads from 80+ clients creates latency. I'm trying to increase the speed by creating multiple MDS even maybe binding subvolumes to specific MDS servers and decrease the latency.
Also when I check MDS CPU usage I see %120++ usage time to time. But when I check the server CPU load at MDS location, I see MDS only uses 2-4 cores and other CPU cores are almost at idle. I think MDS has a CPU core limitation and I need to increase the value to decrease the latency. How can I do that?
Özkan Göksu <ozkangksu@gmail.com>, 17 Oca 2024 Çar, 07:44 tarihinde şunu yazdı:
Let me share some outputs about my cluster.
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 31 /s 3022k 3021k 52.6k 385k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.4T cephfs.ud-data.data data 45.2T 44.4T STANDBY MDS ud-data.ud-03.lhwkml ud-data.ud-05.rnhcfe ud-data.ud-01.uatjle ud-data.ud-04.seggyv
-------------------------------------------------------------------------- This is "ceph tell mds.ud-data.ud-02.xcoojt session ls" output for the reported client for cache pressure warning.
{ "id": 1282205, "entity": { "name": { "type": "client", "num": 1282205 }, "addr": { "type": "v1", "addr": "172.16.3.48:0", "nonce": 2169935642 } }, "state": "open", "num_leases": 0, "num_caps": 52092, "request_load_avg": 1, "uptime": 75754.745608647994, "requests_in_flight": 0, "num_completed_requests": 0, "num_completed_flushes": 1, "reconnecting": false, "recall_caps": { "value": 2577232.0049106553, "halflife": 60 }, "release_caps": { "value": 1.4093491463510395, "halflife": 60 }, "recall_caps_throttle": { "value": 63733.985544098425, "halflife": 1.5 }, "recall_caps_throttle2o": { "value": 19452.428409271757, "halflife": 0.5 }, "session_cache_liveness": { "value": 14.100272208890081, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "delegated_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 } ], "inst": "client.1282205 v1:172.16.3.48:0/2169935642", "completed_requests": [], "prealloc_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 }, { "start": "0x1000fa86e5f", "length": 54 }, { "start": "0x1000faa069c", "length": 501 } ], "client_metadata": { "client_features": { "feature_bits": "0x0000000000007bff" }, "metric_spec": { "metric_flags": { "feature_bits": "0x00000000000003ff" } }, "entity_id": "admin", "hostname": "bennevis-2", "kernel_version": "5.15.0-91-generic", "root": "/volumes/babblians" } }
Özkan Göksu <ozkangksu@gmail.com>, 17 Oca 2024 Çar, 07:22 tarihinde şunu yazdı:
Hello Eugen.
Thank you for the answer. According to knowledge and test results at this issue: https://github.com/ceph/ceph/pull/38574 I've tried their advice and I've applied the following changes.
max_mds = 4 standby_mds = 1 mds_cache_memory_limit = 16GB mds_recall_max_caps = 40000
When I set these parameters, 1 day later I saw this log: [8531248.982954] Out of memory: Killed process 1580586 (ceph-mds) total-vm:70577592kB, anon-rss:70244236kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:137832kB oom_score_adj:0
All the MDS services created memory leak and killed by kernel. Because of this I changed it as below and it is stable now but performance is very poor and I still get cache pressure alerts.
max_mds = 1 standby_mds = 5 mds_cache_memory_limit = 8GB mds_recall_max_caps = 30000
I'm very surprised that you are advising to decrease "mds_recall_max_caps" because it is the opposite of what developers advised in the issue I've sended. It is very hard to play around with MDS parameters without expert level of understanding what these parameters stands for and how it will effect the behavior. Because of this I'm trying to understand the MDS code flow and I'm very interested with learning more and tuning my system by debugging and understanding my own data flow and MDS usage.
I have a very unique data flow and I think I need to configure the system for this case. I have 80+ clients and via all of these clients my users are requesting Read a range of objects and compare them in GPU, they generate new data and Write the new data back in the cluster. So it means my clients usually reads objects only one time and do not read the same object again. Sometimes same user runs multiple service in multiple clients and these services can read the same data from different clients.
So having a large cache is useless for my use case. I need to setup MDS and Cephfs Client for this data flow. When I debug the MDS ram usage, I see high allocation all the time and I wonder why? If any of my client does not read any object why MDS does not remove that data from ram allocation? I need to configure MDS for reading the data and removing it very fast if the data is constantly requested from clients. In this case ofc I want a ram cache tier.
I'm little confused and I need to learn more about how MDS works and how should I make multiple active MDS faster for my subvolumes and client data flow.
Best regards.
Eugen Block <eblock@nde.ag>, 16 Oca 2024 Sal, 11:36 tarihinde şunu yazdı:
Hi,
I have dealt with this topic multiple times, the SUSE team helped understanding what's going on under the hood. The summary can be found in this thread [1].
What helped in our case was to reduce the mds_recall_max_caps from 30k (default) to 3k. We tried it in steps of 1k IIRC. So I suggest to reduce that value step by step (maybe start with 20k or something) to find the optimal value.
Regards, Eugen
[1] https://www.spinics.net/lists/ceph-users/msg73188.html
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello.
I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning.
I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool.
My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data)
---------------------------------------------------------------------------------
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB
---------------------------------------------------------------------------------
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
-----------------------------------------------------------------------------------
My MDS settings are below:
mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144
I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ?
- Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Eugen. I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning. Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow? While searching, I find this: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E... quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release." Because of this I think I also need to play around with the client side too. My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep the free cache size in the desired range. I need to make it aggressive but this should not reduce speed and increase latency. mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128 What do you think?
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not the usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services? Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E... quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep the free cache size in the desired range. I need to make it aggressive but this should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?
I will try my best to explain my situation. I don't have a separate mds server. I have 5 identical nodes, 3 of them mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4) root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M) data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr ------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T ------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577 ------------------------------ When I check the failing client with session ls I see only "num_caps: 12298" ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3 clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 ------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low. avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60 Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20 ------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used directly by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better. I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it. https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E... quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release." ------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90 Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not the usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services?
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this:
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep
free cache size in the desired range. I need to make it aggressive but
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E... the this
should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?
I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is. Zitat von Özkan Göksu <ozkangksu@gmail.com>:
I will try my best to explain my situation.
I don't have a separate mds server. I have 5 identical nodes, 3 of them mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4)
root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure
services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M)
data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep
io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr
------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T
------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577
------------------------------ When I check the failing client with session ls I see only "num_caps: 12298"
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1
------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low.
avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20
------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used directly by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better.
I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E... quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not the usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services?
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this:
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep
free cache size in the desired range. I need to make it aggressive but
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E... the this
should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?
Every user has a 1x subvolume and I only have 1 pool. At the beginning we were using each subvolume for ldap home directory + user data. When a user logins any docker on any host, it was using the cluster for home and the for user related data, we was have second directory in the same subvolume. Time to time users were feeling a very slow home environment and after a month it became almost impossible to use home. VNC sessions became unresponsive and slow etc. 2 weeks ago, I had to migrate home to a ZFS storage and now the overall performance is better for only user_data without home. But still the performance is not good enough as I expected because of the problems related to MDS. The usage is low but allocation is high and Cpu usage is high. You saw the IO Op/s, it's nothing but allocation is high. I develop a fio benchmark script and I run the script on 4x test server at the same time, the results are below: Script: https://github.com/ozkangoksu/benchmark/blob/8f5df87997864c25ef32447e02fcd41... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... While running benchmark, I take sample values for each type of iobench run. Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 70 MiB/s rd, 762 MiB/s wr, 337 op/s rd, 24.41k op/s wr client: 60 MiB/s rd, 551 MiB/s wr, 303 op/s rd, 35.12k op/s wr client: 13 MiB/s rd, 161 MiB/s wr, 101 op/s rd, 41.30k op/s wr Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 1.6 GiB/s rd, 219 KiB/s wr, 28.76k op/s rd, 89 op/s wr client: 370 MiB/s rd, 475 KiB/s wr, 90.38k op/s rd, 89 op/s wr Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 63 MiB/s rd, 1.5 GiB/s wr, 8.77k op/s rd, 5.50k op/s wr client: 14 MiB/s rd, 1.8 GiB/s wr, 81 op/s rd, 13.86k op/s wr client: 6.6 MiB/s rd, 1.2 GiB/s wr, 61 op/s rd, 30.13k op/s wr Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 317 MiB/s rd, 841 MiB/s wr, 426 op/s rd, 10.98k op/s wr client: 2.8 GiB/s rd, 882 MiB/s wr, 25.68k op/s rd, 291 op/s wr client: 4.0 GiB/s rd, 226 MiB/s wr, 89.63k op/s rd, 124 op/s wr client: 2.4 GiB/s rd, 295 KiB/s wr, 197.86k op/s rd, 20 op/s wr It seems I only have problems with the 4K,8K,16K other sector sizes. Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 19:06 tarihinde şunu yazdı:
I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
I will try my best to explain my situation.
I don't have a separate mds server. I have 5 identical nodes, 3 of them mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4)
root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure
services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M)
data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep
io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr
------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T
------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577
------------------------------ When I check the failing client with session ls I see only "num_caps: 12298"
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1
------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low.
avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20
------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used directly by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better.
I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not the usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services?
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep the free cache size in the desired range. I need to make it aggressive but this should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?
This is client side metrics from a "failing to respond to cache pressure" warned client. root@datagen-27:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1282187# cat bdi/stats BdiWriteback: 0 kB BdiReclaimable: 0 kB BdiDirtyThresh: 0 kB DirtyThresh: 35979376 kB BackgroundThresh: 17967720 kB BdiDirtied: 3071616 kB BdiWritten: 3036864 kB BdiWriteBandwidth: 20 kBps b_dirty: 0 b_io: 0 b_more_io: 0 b_dirty_time: 0 bdi_list: 1 state: 1 ------------------------------------------------ root@d27:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1282187# cat metrics item total ------------------------------------------ opened files / total inodes 4 / 14129 pinned i_caps / total inodes 14129 / 14129 opened inodes / total inodes 2 / 14129 item total avg_lat(us) min_lat(us) max_lat(us) stdev(us) ----------------------------------------------------------------------------------- read 1218753 3116 208 8741271 2154 write 34945 24003 3017 2191493 16156 metadata 1703642 8395 127 17936115 1497 item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes) ---------------------------------------------------------------------------------------- read 1218753 227009 1 4194304 276668475618 write 34945 85860 1 4194304 3000382055 item total miss hit ------------------------------------------------- d_lease 306 19110 3317071969 caps 14129 145404 3761682333 Özkan Göksu <ozkangksu@gmail.com>, 25 Oca 2024 Per, 20:25 tarihinde şunu yazdı:
Every user has a 1x subvolume and I only have 1 pool. At the beginning we were using each subvolume for ldap home directory + user data. When a user logins any docker on any host, it was using the cluster for home and the for user related data, we was have second directory in the same subvolume. Time to time users were feeling a very slow home environment and after a month it became almost impossible to use home. VNC sessions became unresponsive and slow etc.
2 weeks ago, I had to migrate home to a ZFS storage and now the overall performance is better for only user_data without home. But still the performance is not good enough as I expected because of the problems related to MDS. The usage is low but allocation is high and Cpu usage is high. You saw the IO Op/s, it's nothing but allocation is high.
I develop a fio benchmark script and I run the script on 4x test server at the same time, the results are below: Script: https://github.com/ozkangoksu/benchmark/blob/8f5df87997864c25ef32447e02fcd41...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
While running benchmark, I take sample values for each type of iobench run.
Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 70 MiB/s rd, 762 MiB/s wr, 337 op/s rd, 24.41k op/s wr client: 60 MiB/s rd, 551 MiB/s wr, 303 op/s rd, 35.12k op/s wr client: 13 MiB/s rd, 161 MiB/s wr, 101 op/s rd, 41.30k op/s wr
Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 1.6 GiB/s rd, 219 KiB/s wr, 28.76k op/s rd, 89 op/s wr client: 370 MiB/s rd, 475 KiB/s wr, 90.38k op/s rd, 89 op/s wr
Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 63 MiB/s rd, 1.5 GiB/s wr, 8.77k op/s rd, 5.50k op/s wr client: 14 MiB/s rd, 1.8 GiB/s wr, 81 op/s rd, 13.86k op/s wr client: 6.6 MiB/s rd, 1.2 GiB/s wr, 61 op/s rd, 30.13k op/s wr
Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 317 MiB/s rd, 841 MiB/s wr, 426 op/s rd, 10.98k op/s wr client: 2.8 GiB/s rd, 882 MiB/s wr, 25.68k op/s rd, 291 op/s wr client: 4.0 GiB/s rd, 226 MiB/s wr, 89.63k op/s rd, 124 op/s wr client: 2.4 GiB/s rd, 295 KiB/s wr, 197.86k op/s rd, 20 op/s wr
It seems I only have problems with the 4K,8K,16K other sector sizes.
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 19:06 tarihinde şunu yazdı:
I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
I will try my best to explain my situation.
I don't have a separate mds server. I have 5 identical nodes, 3 of them mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4)
root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure
services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M)
data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep
io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr
------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T
------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577
------------------------------ When I check the failing client with session ls I see only "num_caps: 12298"
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1
------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low.
avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20
------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used directly by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better.
I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not the usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services?
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep the free cache size in the desired range. I need to make it aggressive but this should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?
Performance for small files is more about IOPS rather than throughput, and the IOPS in your fio tests look okay to me. What you could try is to split the PGs to get around 150 or 200 PGs per OSD. You're currently at around 60 according to the ceph osd df output. Before you do that, can you share 'ceph pg ls-by-pool cephfs.ud-data.data | head'? I don't need the whole output, just to see how many objects each PG has. We had a case once where that helped, but it was an older cluster and the pool was backed by HDDs and separate rocksDB on SSDs. So this might not be the solution here, but it could improve things as well. Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Every user has a 1x subvolume and I only have 1 pool. At the beginning we were using each subvolume for ldap home directory + user data. When a user logins any docker on any host, it was using the cluster for home and the for user related data, we was have second directory in the same subvolume. Time to time users were feeling a very slow home environment and after a month it became almost impossible to use home. VNC sessions became unresponsive and slow etc.
2 weeks ago, I had to migrate home to a ZFS storage and now the overall performance is better for only user_data without home. But still the performance is not good enough as I expected because of the problems related to MDS. The usage is low but allocation is high and Cpu usage is high. You saw the IO Op/s, it's nothing but allocation is high.
I develop a fio benchmark script and I run the script on 4x test server at the same time, the results are below: Script: https://github.com/ozkangoksu/benchmark/blob/8f5df87997864c25ef32447e02fcd41...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
While running benchmark, I take sample values for each type of iobench run.
Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 70 MiB/s rd, 762 MiB/s wr, 337 op/s rd, 24.41k op/s wr client: 60 MiB/s rd, 551 MiB/s wr, 303 op/s rd, 35.12k op/s wr client: 13 MiB/s rd, 161 MiB/s wr, 101 op/s rd, 41.30k op/s wr
Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 1.6 GiB/s rd, 219 KiB/s wr, 28.76k op/s rd, 89 op/s wr client: 370 MiB/s rd, 475 KiB/s wr, 90.38k op/s rd, 89 op/s wr
Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 63 MiB/s rd, 1.5 GiB/s wr, 8.77k op/s rd, 5.50k op/s wr client: 14 MiB/s rd, 1.8 GiB/s wr, 81 op/s rd, 13.86k op/s wr client: 6.6 MiB/s rd, 1.2 GiB/s wr, 61 op/s rd, 30.13k op/s wr
Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 317 MiB/s rd, 841 MiB/s wr, 426 op/s rd, 10.98k op/s wr client: 2.8 GiB/s rd, 882 MiB/s wr, 25.68k op/s rd, 291 op/s wr client: 4.0 GiB/s rd, 226 MiB/s wr, 89.63k op/s rd, 124 op/s wr client: 2.4 GiB/s rd, 295 KiB/s wr, 197.86k op/s rd, 20 op/s wr
It seems I only have problems with the 4K,8K,16K other sector sizes.
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 19:06 tarihinde şunu yazdı:
I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
I will try my best to explain my situation.
I don't have a separate mds server. I have 5 identical nodes, 3 of them mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4)
root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure
services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M)
data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep
io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr
------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T
------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577
------------------------------ When I check the failing client with session ls I see only "num_caps: 12298"
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1
------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low.
avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20
------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used directly by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better.
I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not the usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services?
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep the free cache size in the desired range. I need to make it aggressive but this should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?
Hi, this message is one of those that are often spurious. I don't recall in which thread/PR/tracker I read it, but the story was something like that: If an MDS gets under memory pressure it will request dentry items back from *all* clients, not just the active ones or the ones holding many of them. If you have a client that's below the min-threshold for dentries (its one of the client/mds tuning options), it will not respond. This client will be flagged as not responding, which is a false positive. I believe the devs are working on a fix to get rid of these spurious warnings. There is a "bug/feature" in the MDS that does not clear this warning flag for inactive clients. Hence, the message hangs and never disappears. I usually clear it with a "echo 3 > /proc/sys/vm/drop_caches" on the client. However, except for being annoying in the dashboard, it has no performance or otherwise negative impact. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Friday, January 26, 2024 10:05 AM To: Özkan Göksu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: 1 clients failing to respond to cache pressure (quincy:17.2.6) Performance for small files is more about IOPS rather than throughput, and the IOPS in your fio tests look okay to me. What you could try is to split the PGs to get around 150 or 200 PGs per OSD. You're currently at around 60 according to the ceph osd df output. Before you do that, can you share 'ceph pg ls-by-pool cephfs.ud-data.data | head'? I don't need the whole output, just to see how many objects each PG has. We had a case once where that helped, but it was an older cluster and the pool was backed by HDDs and separate rocksDB on SSDs. So this might not be the solution here, but it could improve things as well. Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Every user has a 1x subvolume and I only have 1 pool. At the beginning we were using each subvolume for ldap home directory + user data. When a user logins any docker on any host, it was using the cluster for home and the for user related data, we was have second directory in the same subvolume. Time to time users were feeling a very slow home environment and after a month it became almost impossible to use home. VNC sessions became unresponsive and slow etc.
2 weeks ago, I had to migrate home to a ZFS storage and now the overall performance is better for only user_data without home. But still the performance is not good enough as I expected because of the problems related to MDS. The usage is low but allocation is high and Cpu usage is high. You saw the IO Op/s, it's nothing but allocation is high.
I develop a fio benchmark script and I run the script on 4x test server at the same time, the results are below: Script: https://github.com/ozkangoksu/benchmark/blob/8f5df87997864c25ef32447e02fcd41...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-... https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
While running benchmark, I take sample values for each type of iobench run.
Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 70 MiB/s rd, 762 MiB/s wr, 337 op/s rd, 24.41k op/s wr client: 60 MiB/s rd, 551 MiB/s wr, 303 op/s rd, 35.12k op/s wr client: 13 MiB/s rd, 161 MiB/s wr, 101 op/s rd, 41.30k op/s wr
Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 1.6 GiB/s rd, 219 KiB/s wr, 28.76k op/s rd, 89 op/s wr client: 370 MiB/s rd, 475 KiB/s wr, 90.38k op/s rd, 89 op/s wr
Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 63 MiB/s rd, 1.5 GiB/s wr, 8.77k op/s rd, 5.50k op/s wr client: 14 MiB/s rd, 1.8 GiB/s wr, 81 op/s rd, 13.86k op/s wr client: 6.6 MiB/s rd, 1.2 GiB/s wr, 61 op/s rd, 30.13k op/s wr
Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 317 MiB/s rd, 841 MiB/s wr, 426 op/s rd, 10.98k op/s wr client: 2.8 GiB/s rd, 882 MiB/s wr, 25.68k op/s rd, 291 op/s wr client: 4.0 GiB/s rd, 226 MiB/s wr, 89.63k op/s rd, 124 op/s wr client: 2.4 GiB/s rd, 295 KiB/s wr, 197.86k op/s rd, 20 op/s wr
It seems I only have problems with the 4K,8K,16K other sector sizes.
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 19:06 tarihinde şunu yazdı:
I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
I will try my best to explain my situation.
I don't have a separate mds server. I have 5 identical nodes, 3 of them mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4)
root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure
services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M)
data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep
io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr
------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T
------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577
------------------------------ When I check the failing client with session ls I see only "num_caps: 12298"
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1
------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low.
avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20
------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used directly by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better.
I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not the usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services?
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep the free cache size in the desired range. I need to make it aggressive but this should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Frank. I have 84 clients (high-end servers) with: Ubuntu 20.04.5 LTS - Kernel: Linux 5.4.0-125-generic My cluster 17.2.6 quincy. I have some client nodes with "ceph-common/stable,now 17.2.7-1focal" I wonder using new version clients is the main problem? Maybe I have a communication error. For example I hit this problem and I can not collect client stats "https://github.com/ceph/ceph/pull/52127/files" Best regards. Frank Schilder <frans@dtu.dk>, 26 Oca 2024 Cum, 14:53 tarihinde şunu yazdı:
Hi, this message is one of those that are often spurious. I don't recall in which thread/PR/tracker I read it, but the story was something like that:
If an MDS gets under memory pressure it will request dentry items back from *all* clients, not just the active ones or the ones holding many of them. If you have a client that's below the min-threshold for dentries (its one of the client/mds tuning options), it will not respond. This client will be flagged as not responding, which is a false positive.
I believe the devs are working on a fix to get rid of these spurious warnings. There is a "bug/feature" in the MDS that does not clear this warning flag for inactive clients. Hence, the message hangs and never disappears. I usually clear it with a "echo 3 > /proc/sys/vm/drop_caches" on the client. However, except for being annoying in the dashboard, it has no performance or otherwise negative impact.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Friday, January 26, 2024 10:05 AM To: Özkan Göksu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: 1 clients failing to respond to cache pressure (quincy:17.2.6)
Performance for small files is more about IOPS rather than throughput, and the IOPS in your fio tests look okay to me. What you could try is to split the PGs to get around 150 or 200 PGs per OSD. You're currently at around 60 according to the ceph osd df output. Before you do that, can you share 'ceph pg ls-by-pool cephfs.ud-data.data | head'? I don't need the whole output, just to see how many objects each PG has. We had a case once where that helped, but it was an older cluster and the pool was backed by HDDs and separate rocksDB on SSDs. So this might not be the solution here, but it could improve things as well.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Every user has a 1x subvolume and I only have 1 pool. At the beginning we were using each subvolume for ldap home directory + user data. When a user logins any docker on any host, it was using the cluster for home and the for user related data, we was have second directory in the same subvolume. Time to time users were feeling a very slow home environment and after a month it became almost impossible to use home. VNC sessions became unresponsive and slow etc.
2 weeks ago, I had to migrate home to a ZFS storage and now the overall performance is better for only user_data without home. But still the performance is not good enough as I expected because of the problems related to MDS. The usage is low but allocation is high and Cpu usage is high. You saw the IO Op/s, it's nothing but allocation is high.
I develop a fio benchmark script and I run the script on 4x test server at the same time, the results are below: Script:
https://github.com/ozkangoksu/benchmark/blob/8f5df87997864c25ef32447e02fcd41...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
While running benchmark, I take sample values for each type of iobench
run.
Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 70 MiB/s rd, 762 MiB/s wr, 337 op/s rd, 24.41k op/s wr client: 60 MiB/s rd, 551 MiB/s wr, 303 op/s rd, 35.12k op/s wr client: 13 MiB/s rd, 161 MiB/s wr, 101 op/s rd, 41.30k op/s wr
Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 1.6 GiB/s rd, 219 KiB/s wr, 28.76k op/s rd, 89 op/s wr client: 370 MiB/s rd, 475 KiB/s wr, 90.38k op/s rd, 89 op/s wr
Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 63 MiB/s rd, 1.5 GiB/s wr, 8.77k op/s rd, 5.50k op/s wr client: 14 MiB/s rd, 1.8 GiB/s wr, 81 op/s rd, 13.86k op/s wr client: 6.6 MiB/s rd, 1.2 GiB/s wr, 61 op/s rd, 30.13k op/s wr
Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 317 MiB/s rd, 841 MiB/s wr, 426 op/s rd, 10.98k op/s wr client: 2.8 GiB/s rd, 882 MiB/s wr, 25.68k op/s rd, 291 op/s wr client: 4.0 GiB/s rd, 226 MiB/s wr, 89.63k op/s rd, 124 op/s wr client: 2.4 GiB/s rd, 295 KiB/s wr, 197.86k op/s rd, 20 op/s wr
It seems I only have problems with the 4K,8K,16K other sector sizes.
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 19:06 tarihinde şunu
yazdı:
I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
I will try my best to explain my situation.
I don't have a separate mds server. I have 5 identical nodes, 3 of
mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4)
root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure
services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M)
data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep
io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr
------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T
------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache
mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond
to
cache pressure client_id: 1275577
------------------------------ When I check the failing client with session ls I see only "num_caps: 12298"
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1
------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low.
avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20
------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used
by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better.
I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not
them pressure directly the
usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services?
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep the free cache size in the desired range. I need to make it aggressive but this should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I started to investigate my clients. for example: root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577 root@ud-01:~# ceph fs status ud-data - 86 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 34 /s 2926k 2827k 155k 1157k ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3 clientid: *1275577*= num_caps: 12312, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275571= num_caps: 16307, num_leases: 1, request_load_avg: 2101, num_completed_requests: 0, num_completed_flushes: 3 clientid: 1282130= num_caps: 26337, num_leases: 3, request_load_avg: 116, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1191789= num_caps: 32784, num_leases: 0, request_load_avg: 1846, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275535= num_caps: 79825, num_leases: 2, request_load_avg: 133, num_completed_requests: 8, num_completed_flushes: 8 clientid: 1282142= num_caps: 80581, num_leases: 6, request_load_avg: 125, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1275532= num_caps: 87836, num_leases: 3, request_load_avg: 190, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1275547= num_caps: 94129, num_leases: 4, request_load_avg: 149, num_completed_requests: 2, num_completed_flushes: 4 clientid: 1275553= num_caps: 96460, num_leases: 4, request_load_avg: 155, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282139= num_caps: 108882, num_leases: 25, request_load_avg: 99, num_completed_requests: 2, num_completed_flushes: 4 clientid: 1275538= num_caps: 437162, num_leases: 0, request_load_avg: 101, num_completed_requests: 2, num_completed_flushes: 0 -------------------------------------- *MY CLIENT:* The client is actually at idle mode and there is no reason to fail at all. root@bmw-m4:~# apt list --installed |grep ceph ceph-common/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed] libcephfs2/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed,automatic] python3-ceph-argparse/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed,automatic] python3-ceph-common/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 all [installed,automatic] python3-cephfs/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed,automatic] Let's check metrics and stats: root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# cat metrics item total ------------------------------------------ opened files / total inodes 2 / 12312 pinned i_caps / total inodes 12312 / 12312 opened inodes / total inodes 1 / 12312 item total avg_lat(us) min_lat(us) max_lat(us) stdev(us) ----------------------------------------------------------------------------------- read 22283 44409 430 1804853 15619 write 112702 419725 3658 8879541 6008 metadata 353322 5712 154 917903 5357 item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes) ---------------------------------------------------------------------------------------- read 22283 1701940 1 4194304 37924318602 write 112702 246211 1 4194304 27748469309 item total miss hit ------------------------------------------------- d_lease 62 63627 28564698 caps 12312 36658 44568261 root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# cat bdi/stats BdiWriteback: 0 kB BdiReclaimable: 800 kB BdiDirtyThresh: 0 kB DirtyThresh: 5795340 kB BackgroundThresh: 2894132 kB BdiDirtied: 27316320 kB BdiWritten: 27316320 kB BdiWriteBandwidth: 1472 kBps b_dirty: 0 b_io: 0 b_more_io: 0 b_dirty_time: 0 bdi_list: 1 state: 1 Last 3 days dmesg output: [Wed Jan 24 16:45:13 2024] xfsettingsd[653036]: segfault at 18 ip 00007fbd12f5d337 sp 00007ffd254332a0 error 4 in libxklavier.so.16.4.0[7fbd12f4d000+19000] [Wed Jan 24 16:45:13 2024] Code: 4c 89 e7 e8 0b 56 ff ff 48 89 03 48 8b 5c 24 30 e9 d1 fd ff ff e8 b9 5b ff ff 66 0f 1f 84 00 00 00 00 00 41 54 55 48 89 f5 53 <48> 8b 42 18 48 89 d1 49 89 fc 48 89 d3 48 89 fa 48 89 ef 48 8b b0 [Thu Jan 25 06:51:31 2024] NVRM: GPU at PCI:0000:81:00: GPU-02efbb18-c9e4-3a16-d615-598959520b99 [Thu Jan 25 06:51:31 2024] NVRM: GPU Board Serial Number: 1321421015411 [Thu Jan 25 06:51:31 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=683281, name=python, Ch 00000008 [Thu Jan 25 06:56:49 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=683377, name=python, Ch 00000018 [Thu Jan 25 20:14:13 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=696062, name=python, Ch 00000008 [Fri Jan 26 04:05:40 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=700166, name=python, Ch 00000008 [Fri Jan 26 05:05:12 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=700320, name=python, Ch 00000008 [Fri Jan 26 05:44:50 2024] NVRM: GPU at PCI:0000:82:00: GPU-3af62a2c-e7eb-a7d5-c073-22f06dc7065f [Fri Jan 26 05:44:50 2024] NVRM: GPU Board Serial Number: 1321421010400 [Fri Jan 26 05:44:50 2024] NVRM: Xid (PCI:0000:82:00): 43, pid=700757, name=python, Ch 00000018 [Fri Jan 26 05:56:02 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=701096, name=python, Ch 00000028 [Fri Jan 26 06:34:20 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=701226, name=python, Ch 00000038 root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# free -h total used free shared buff/cache available Mem: 62Gi 34Gi 27Gi 0.0Ki 639Mi 27Gi Swap: 1.8Ti 18Gi 1.8Ti root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# cat /proc/vmstat nr_free_pages 7231171 nr_zone_inactive_anon 7924766 nr_zone_active_anon 525190 nr_zone_inactive_file 44029 nr_zone_active_file 55966 nr_zone_unevictable 13042 nr_zone_write_pending 3 nr_mlock 13042 nr_bounce 0 nr_zspages 0 nr_free_cma 0 numa_hit 6701928919 numa_miss 312628341 numa_foreign 312628341 numa_interleave 31538 numa_local 6701864751 numa_other 312692567 nr_inactive_anon 7924766 nr_active_anon 525190 nr_inactive_file 44029 nr_active_file 55966 nr_unevictable 13042 nr_slab_reclaimable 61076 nr_slab_unreclaimable 63509 nr_isolated_anon 0 nr_isolated_file 0 workingset_nodes 3934 workingset_refault_anon 30325493 workingset_refault_file 14593094 workingset_activate_anon 5376050 workingset_activate_file 3250679 workingset_restore_anon 292317 workingset_restore_file 1166673 workingset_nodereclaim 488665 nr_anon_pages 8451968 nr_mapped 35731 nr_file_pages 138824 nr_dirty 3 nr_writeback 0 nr_writeback_temp 0 nr_shmem 242 nr_shmem_hugepages 0 nr_shmem_pmdmapped 0 nr_file_hugepages 0 nr_file_pmdmapped 0 nr_anon_transparent_hugepages 3588 nr_vmscan_write 33746573 nr_vmscan_immediate_reclaim 160 nr_dirtied 48165341 nr_written 80207893 nr_kernel_misc_reclaimable 0 nr_foll_pin_acquired 174002 nr_foll_pin_released 174002 nr_kernel_stack 60032 nr_page_table_pages 46041 nr_swapcached 36166 nr_dirty_threshold 1448010 nr_dirty_background_threshold 723121 pgpgin 129904699 pgpgout 299261581 pswpin 30325493 pswpout 45158221 pgalloc_dma 1024 pgalloc_dma32 57788566 pgalloc_normal 6956384725 pgalloc_movable 0 allocstall_dma 0 allocstall_dma32 0 allocstall_normal 188 allocstall_movable 63024 pgskip_dma 0 pgskip_dma32 0 pgskip_normal 0 pgskip_movable 0 pgfree 7222273815 pgactivate 1371753960 pgdeactivate 18329381 pglazyfree 10 pgfault 7795723861 pgmajfault 4600007 pglazyfreed 0 pgrefill 18575528 pgreuse 81910383 pgsteal_kswapd 980532060 pgsteal_direct 38942066 pgdemote_kswapd 0 pgdemote_direct 0 pgscan_kswapd 1135293298 pgscan_direct 58883653 pgscan_direct_throttle 15 pgscan_anon 220939938 pgscan_file 973237013 pgsteal_anon 46538607 pgsteal_file 972935519 zone_reclaim_failed 0 pginodesteal 0 slabs_scanned 25879882 kswapd_inodesteal 2179831 kswapd_low_wmark_hit_quickly 152797 kswapd_high_wmark_hit_quickly 32025 pageoutrun 204447 pgrotated 44963935 drop_pagecache 0 drop_slab 0 oom_kill 0 numa_pte_updates 2724410955 numa_huge_pte_updates 1695890 numa_hint_faults 1739823254 numa_hint_faults_local 1222358972 numa_pages_migrated 312611639 pgmigrate_success 510846802 pgmigrate_fail 875493 thp_migration_success 156413 thp_migration_fail 2 thp_migration_split 0 compact_migrate_scanned 1274073243 compact_free_scanned 8430842597 compact_isolated 400278352 compact_stall 145300 compact_fail 128562 compact_success 16738 compact_daemon_wake 170247 compact_daemon_migrate_scanned 35486283 compact_daemon_free_scanned 369870412 htlb_buddy_alloc_success 0 htlb_buddy_alloc_fail 0 unevictable_pgs_culled 2774290 unevictable_pgs_scanned 0 unevictable_pgs_rescued 2675031 unevictable_pgs_mlocked 2813622 unevictable_pgs_munlocked 2674972 unevictable_pgs_cleared 84231 unevictable_pgs_stranded 84225 thp_fault_alloc 416468 thp_fault_fallback 19181 thp_fault_fallback_charge 0 thp_collapse_alloc 17931 thp_collapse_alloc_failed 76 thp_file_alloc 0 thp_file_fallback 0 thp_file_fallback_charge 0 thp_file_mapped 0 thp_split_page 2 thp_split_page_failed 0 thp_deferred_split_page 66 thp_split_pmd 22451 thp_split_pud 0 thp_zero_page_alloc 1 thp_zero_page_alloc_failed 0 thp_swpout 22332 thp_swpout_fallback 0 balloon_inflate 0 balloon_deflate 0 balloon_migrate 0 swap_ra 25777929 swap_ra_hit 25658825 direct_map_level2_splits 1249 direct_map_level3_splits 49 nr_unstable 0 Özkan Göksu <ozkangksu@gmail.com>, 27 Oca 2024 Cmt, 02:36 tarihinde şunu yazdı:
Hello Frank.
I have 84 clients (high-end servers) with: Ubuntu 20.04.5 LTS - Kernel: Linux 5.4.0-125-generic
My cluster 17.2.6 quincy. I have some client nodes with "ceph-common/stable,now 17.2.7-1focal" I wonder using new version clients is the main problem? Maybe I have a communication error. For example I hit this problem and I can not collect client stats " https://github.com/ceph/ceph/pull/52127/files"
Best regards.
Frank Schilder <frans@dtu.dk>, 26 Oca 2024 Cum, 14:53 tarihinde şunu yazdı:
Hi, this message is one of those that are often spurious. I don't recall in which thread/PR/tracker I read it, but the story was something like that:
If an MDS gets under memory pressure it will request dentry items back from *all* clients, not just the active ones or the ones holding many of them. If you have a client that's below the min-threshold for dentries (its one of the client/mds tuning options), it will not respond. This client will be flagged as not responding, which is a false positive.
I believe the devs are working on a fix to get rid of these spurious warnings. There is a "bug/feature" in the MDS that does not clear this warning flag for inactive clients. Hence, the message hangs and never disappears. I usually clear it with a "echo 3 > /proc/sys/vm/drop_caches" on the client. However, except for being annoying in the dashboard, it has no performance or otherwise negative impact.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Friday, January 26, 2024 10:05 AM To: Özkan Göksu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: 1 clients failing to respond to cache pressure (quincy:17.2.6)
Performance for small files is more about IOPS rather than throughput, and the IOPS in your fio tests look okay to me. What you could try is to split the PGs to get around 150 or 200 PGs per OSD. You're currently at around 60 according to the ceph osd df output. Before you do that, can you share 'ceph pg ls-by-pool cephfs.ud-data.data | head'? I don't need the whole output, just to see how many objects each PG has. We had a case once where that helped, but it was an older cluster and the pool was backed by HDDs and separate rocksDB on SSDs. So this might not be the solution here, but it could improve things as well.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Every user has a 1x subvolume and I only have 1 pool. At the beginning we were using each subvolume for ldap home directory + user data. When a user logins any docker on any host, it was using the cluster for home and the for user related data, we was have second directory in the same subvolume. Time to time users were feeling a very slow home environment and after a month it became almost impossible to use home. VNC sessions became unresponsive and slow etc.
2 weeks ago, I had to migrate home to a ZFS storage and now the overall performance is better for only user_data without home. But still the performance is not good enough as I expected because of the problems related to MDS. The usage is low but allocation is high and Cpu usage is high. You saw the IO Op/s, it's nothing but allocation is high.
I develop a fio benchmark script and I run the script on 4x test server at the same time, the results are below: Script:
https://github.com/ozkangoksu/benchmark/blob/8f5df87997864c25ef32447e02fcd41...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
While running benchmark, I take sample values for each type of iobench
run.
Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 70 MiB/s rd, 762 MiB/s wr, 337 op/s rd, 24.41k op/s wr client: 60 MiB/s rd, 551 MiB/s wr, 303 op/s rd, 35.12k op/s wr client: 13 MiB/s rd, 161 MiB/s wr, 101 op/s rd, 41.30k op/s wr
Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 1.6 GiB/s rd, 219 KiB/s wr, 28.76k op/s rd, 89 op/s wr client: 370 MiB/s rd, 475 KiB/s wr, 90.38k op/s rd, 89 op/s wr
Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 63 MiB/s rd, 1.5 GiB/s wr, 8.77k op/s rd, 5.50k op/s wr client: 14 MiB/s rd, 1.8 GiB/s wr, 81 op/s rd, 13.86k op/s wr client: 6.6 MiB/s rd, 1.2 GiB/s wr, 61 op/s rd, 30.13k op/s wr
Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 317 MiB/s rd, 841 MiB/s wr, 426 op/s rd, 10.98k op/s wr client: 2.8 GiB/s rd, 882 MiB/s wr, 25.68k op/s rd, 291 op/s wr client: 4.0 GiB/s rd, 226 MiB/s wr, 89.63k op/s rd, 124 op/s wr client: 2.4 GiB/s rd, 295 KiB/s wr, 197.86k op/s rd, 20 op/s wr
It seems I only have problems with the 4K,8K,16K other sector sizes.
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 19:06 tarihinde şunu
yazdı:
I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
I will try my best to explain my situation.
I don't have a separate mds server. I have 5 identical nodes, 3 of
mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4)
root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure
services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M)
data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep
io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr
------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T
------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache
mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to
respond to
cache pressure client_id: 1275577
------------------------------ When I check the failing client with session ls I see only "num_caps: 12298"
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1
------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low.
avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20
------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used
by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better.
I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not
them pressure directly the
usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services?
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
> Hello Eugen. > > I read all of your MDS related topics and thank you so much for your effort > on this. > There is not much information and I couldn't find a MDS tuning guide at > all. It seems that you are the correct person to discuss mds debugging and > tuning. > > Do you have any documents or may I learn what is the proper way to debug > MDS and clients ? > Which debug logs will guide me to understand the limitations and will help > to tune according to the data flow? > > While searching, I find this: >
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
> quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic > caps recall eventually starts recalling those but the (el7 kernel) client > won't release them. Stopping Codium seems to be the only way to release." > > Because of this I think I also need to play around with the client side too. > > My main goal is increasing the speed and reducing the latency and I wonder > if these ideas are correct or not: > - Maybe I need to increase client side cache size because via each client, > multiple users request a lot of objects and clearly the > client_cache_size=16 default is not enough. > - Maybe I need to increase client side maximum cache limit for > object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi > to 400mi" > - The client cache cleaning threshold is not aggressive enough to keep the > free cache size in the desired range. I need to make it aggressive but this > should not reduce speed and increase latency. > > mds_cache_memory_limit=4gi to 16gi > client_oc_max_objects=1000 to 10000 > client_oc_size=200mi to 400mi > client_permissions=false #to reduce latency. > client_cache_size=16 to 128 > > > What do you think?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Wow I noticed something! To prevent ram overflow with gpu training allocations, I'm using a 2TB Samsung 870 evo for swap. As you can see below, swap usage 18Gi and server was idle, that means maybe ceph client hits latency because of the swap usage. root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# free -h total used free shared buff/cache available Mem: 62Gi 34Gi 27Gi 0.0Ki 639Mi 27Gi Swap: 1.8Ti 18Gi 1.8Ti I decided to play around kernel parameters to prevent ceph swap usage. kernel.shmmax = 60654764851 # Maximum shared segment size in bytes
kernel.shmall = 16453658 # Maximum number of shared memory segments in pages vm.nr_hugepages = 4096 # Increase Transparent Huge Pages (THP) Defrag: vm.swappiness = 0 # Set vm.swappiness to 0 to minimize swapping vm.min_free_kbytes = 1048576 # required free memory (set to 1% of physical ram)
I reboot the server and after reboot swap usage is 0 as expected. To give a try I started the iobench.sh ( https://github.com/ozkangoksu/benchmark/blob/main/iobench.sh) This client has 1G nic only. As you can see below, other then 4K block size, ceph client can saturate NIC. root@bmw-m4:~# nicstat -MUz 1 Time Int rMbps wMbps rPk/s wPk/s rAvs wAvs %rUtil %wUtil 01:04:48 ens1f0 936.9 92.90 91196.8 60126.3 1346.6 202.5 98.2 9.74 root@bmw-m4:/mounts/ud-data/benchuser1/96f13211-c37f-42db-8d05-f3255a05129e/testdir# bash iobench.sh Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M write: IOPS=112, BW=112MiB/s (118MB/s)(3072MiB/27395msec); 0 zone resets BS=128K write: IOPS=894, BW=112MiB/s (117MB/s)(3072MiB/27462msec); 0 zone resets BS=64K write: IOPS=1758, BW=110MiB/s (115MB/s)(3072MiB/27948msec); 0 zone resets BS=32K write: IOPS=3542, BW=111MiB/s (116MB/s)(3072MiB/27748msec); 0 zone resets BS=16K write: IOPS=6839, BW=107MiB/s (112MB/s)(3072MiB/28747msec); 0 zone resets BS=4K write: IOPS=8473, BW=33.1MiB/s (34.7MB/s)(3072MiB/92813msec); 0 zone resets Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M read: IOPS=112, BW=112MiB/s (118MB/s)(3072MiB/27386msec) BS=128K read: IOPS=895, BW=112MiB/s (117MB/s)(3072MiB/27431msec) BS=64K read: IOPS=1788, BW=112MiB/s (117MB/s)(3072MiB/27486msec) BS=32K read: IOPS=3561, BW=111MiB/s (117MB/s)(3072MiB/27603msec) BS=16K read: IOPS=6924, BW=108MiB/s (113MB/s)(3072MiB/28392msec) BS=4K read: IOPS=21.3k, BW=83.3MiB/s (87.3MB/s)(3072MiB/36894msec) Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M write: IOPS=112, BW=112MiB/s (118MB/s)(3072MiB/27406msec); 0 zone resets BS=128K write: IOPS=894, BW=112MiB/s (117MB/s)(3072MiB/27466msec); 0 zone resets BS=64K write: IOPS=1781, BW=111MiB/s (117MB/s)(3072MiB/27591msec); 0 zone resets BS=32K write: IOPS=3545, BW=111MiB/s (116MB/s)(3072MiB/27729msec); 0 zone resets BS=16K write: IOPS=6823, BW=107MiB/s (112MB/s)(3072MiB/28814msec); 0 zone resets BS=4K write: IOPS=12.7k, BW=49.8MiB/s (52.2MB/s)(3072MiB/61694msec); 0 zone resets Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M read: IOPS=112, BW=112MiB/s (118MB/s)(3072MiB/27388msec) BS=128K read: IOPS=894, BW=112MiB/s (117MB/s)(3072MiB/27479msec) BS=64K read: IOPS=1784, BW=112MiB/s (117MB/s)(3072MiB/27547msec) BS=32K read: IOPS=3559, BW=111MiB/s (117MB/s)(3072MiB/27614msec) BS=16K read: IOPS=7047, BW=110MiB/s (115MB/s)(3072MiB/27897msec) BS=4K read: IOPS=26.9k, BW=105MiB/s (110MB/s)(3072MiB/29199msec) root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1818702# cat metrics item total ------------------------------------------ opened files / total inodes 0 / 109 pinned i_caps / total inodes 109 / 109 opened inodes / total inodes 0 / 109 item total avg_lat(us) min_lat(us) max_lat(us) stdev(us) ----------------------------------------------------------------------------------- read 2316289 13904 221 8827984 760 write 2317824 21152 2975 9243821 2365 metadata 170 5944 225 202505 24314 item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes) ---------------------------------------------------------------------------------------- read 2316289 16688 4096 1048576 38654712361 write 2317824 19457 4096 4194304 45097156608 item total miss hit ------------------------------------------------- d_lease 112 3 858 caps 109 58 6963547 root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1818702# free -h total used free shared buff/cache available Mem: 62Gi 11Gi 50Gi 3.0Mi 1.0Gi 49Gi Swap: 1.8Ti 0B 1.8Ti I started to feel we are getting closer :) Özkan Göksu <ozkangksu@gmail.com>, 27 Oca 2024 Cmt, 02:58 tarihinde şunu yazdı:
I started to investigate my clients.
for example:
root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577
root@ud-01:~# ceph fs status ud-data - 86 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 34 /s 2926k 2827k 155k 1157k
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: *1275577*= num_caps: 12312, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275571= num_caps: 16307, num_leases: 1, request_load_avg: 2101, num_completed_requests: 0, num_completed_flushes: 3 clientid: 1282130= num_caps: 26337, num_leases: 3, request_load_avg: 116, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1191789= num_caps: 32784, num_leases: 0, request_load_avg: 1846, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275535= num_caps: 79825, num_leases: 2, request_load_avg: 133, num_completed_requests: 8, num_completed_flushes: 8 clientid: 1282142= num_caps: 80581, num_leases: 6, request_load_avg: 125, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1275532= num_caps: 87836, num_leases: 3, request_load_avg: 190, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1275547= num_caps: 94129, num_leases: 4, request_load_avg: 149, num_completed_requests: 2, num_completed_flushes: 4 clientid: 1275553= num_caps: 96460, num_leases: 4, request_load_avg: 155, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282139= num_caps: 108882, num_leases: 25, request_load_avg: 99, num_completed_requests: 2, num_completed_flushes: 4 clientid: 1275538= num_caps: 437162, num_leases: 0, request_load_avg: 101, num_completed_requests: 2, num_completed_flushes: 0
--------------------------------------
*MY CLIENT:*
The client is actually at idle mode and there is no reason to fail at all.
root@bmw-m4:~# apt list --installed |grep ceph ceph-common/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed] libcephfs2/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed,automatic] python3-ceph-argparse/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed,automatic] python3-ceph-common/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 all [installed,automatic] python3-cephfs/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed,automatic]
Let's check metrics and stats:
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# cat metrics item total ------------------------------------------ opened files / total inodes 2 / 12312 pinned i_caps / total inodes 12312 / 12312 opened inodes / total inodes 1 / 12312
item total avg_lat(us) min_lat(us) max_lat(us) stdev(us)
----------------------------------------------------------------------------------- read 22283 44409 430 1804853 15619 write 112702 419725 3658 8879541 6008 metadata 353322 5712 154 917903 5357
item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)
---------------------------------------------------------------------------------------- read 22283 1701940 1 4194304 37924318602 write 112702 246211 1 4194304 27748469309
item total miss hit ------------------------------------------------- d_lease 62 63627 28564698 caps 12312 36658 44568261
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# cat bdi/stats BdiWriteback: 0 kB BdiReclaimable: 800 kB BdiDirtyThresh: 0 kB DirtyThresh: 5795340 kB BackgroundThresh: 2894132 kB BdiDirtied: 27316320 kB BdiWritten: 27316320 kB BdiWriteBandwidth: 1472 kBps b_dirty: 0 b_io: 0 b_more_io: 0 b_dirty_time: 0 bdi_list: 1 state: 1
Last 3 days dmesg output:
[Wed Jan 24 16:45:13 2024] xfsettingsd[653036]: segfault at 18 ip 00007fbd12f5d337 sp 00007ffd254332a0 error 4 in libxklavier.so.16.4.0[7fbd12f4d000+19000] [Wed Jan 24 16:45:13 2024] Code: 4c 89 e7 e8 0b 56 ff ff 48 89 03 48 8b 5c 24 30 e9 d1 fd ff ff e8 b9 5b ff ff 66 0f 1f 84 00 00 00 00 00 41 54 55 48 89 f5 53 <48> 8b 42 18 48 89 d1 49 89 fc 48 89 d3 48 89 fa 48 89 ef 48 8b b0 [Thu Jan 25 06:51:31 2024] NVRM: GPU at PCI:0000:81:00: GPU-02efbb18-c9e4-3a16-d615-598959520b99 [Thu Jan 25 06:51:31 2024] NVRM: GPU Board Serial Number: 1321421015411 [Thu Jan 25 06:51:31 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=683281, name=python, Ch 00000008 [Thu Jan 25 06:56:49 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=683377, name=python, Ch 00000018 [Thu Jan 25 20:14:13 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=696062, name=python, Ch 00000008 [Fri Jan 26 04:05:40 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=700166, name=python, Ch 00000008 [Fri Jan 26 05:05:12 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=700320, name=python, Ch 00000008 [Fri Jan 26 05:44:50 2024] NVRM: GPU at PCI:0000:82:00: GPU-3af62a2c-e7eb-a7d5-c073-22f06dc7065f [Fri Jan 26 05:44:50 2024] NVRM: GPU Board Serial Number: 1321421010400 [Fri Jan 26 05:44:50 2024] NVRM: Xid (PCI:0000:82:00): 43, pid=700757, name=python, Ch 00000018 [Fri Jan 26 05:56:02 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=701096, name=python, Ch 00000028 [Fri Jan 26 06:34:20 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=701226, name=python, Ch 00000038
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# free -h total used free shared buff/cache available Mem: 62Gi 34Gi 27Gi 0.0Ki 639Mi 27Gi Swap: 1.8Ti 18Gi 1.8Ti
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# cat /proc/vmstat nr_free_pages 7231171 nr_zone_inactive_anon 7924766 nr_zone_active_anon 525190 nr_zone_inactive_file 44029 nr_zone_active_file 55966 nr_zone_unevictable 13042 nr_zone_write_pending 3 nr_mlock 13042 nr_bounce 0 nr_zspages 0 nr_free_cma 0 numa_hit 6701928919 numa_miss 312628341 numa_foreign 312628341 numa_interleave 31538 numa_local 6701864751 numa_other 312692567 nr_inactive_anon 7924766 nr_active_anon 525190 nr_inactive_file 44029 nr_active_file 55966 nr_unevictable 13042 nr_slab_reclaimable 61076 nr_slab_unreclaimable 63509 nr_isolated_anon 0 nr_isolated_file 0 workingset_nodes 3934 workingset_refault_anon 30325493 workingset_refault_file 14593094 workingset_activate_anon 5376050 workingset_activate_file 3250679 workingset_restore_anon 292317 workingset_restore_file 1166673 workingset_nodereclaim 488665 nr_anon_pages 8451968 nr_mapped 35731 nr_file_pages 138824 nr_dirty 3 nr_writeback 0 nr_writeback_temp 0 nr_shmem 242 nr_shmem_hugepages 0 nr_shmem_pmdmapped 0 nr_file_hugepages 0 nr_file_pmdmapped 0 nr_anon_transparent_hugepages 3588 nr_vmscan_write 33746573 nr_vmscan_immediate_reclaim 160 nr_dirtied 48165341 nr_written 80207893 nr_kernel_misc_reclaimable 0 nr_foll_pin_acquired 174002 nr_foll_pin_released 174002 nr_kernel_stack 60032 nr_page_table_pages 46041 nr_swapcached 36166 nr_dirty_threshold 1448010 nr_dirty_background_threshold 723121 pgpgin 129904699 pgpgout 299261581 pswpin 30325493 pswpout 45158221 pgalloc_dma 1024 pgalloc_dma32 57788566 pgalloc_normal 6956384725 pgalloc_movable 0 allocstall_dma 0 allocstall_dma32 0 allocstall_normal 188 allocstall_movable 63024 pgskip_dma 0 pgskip_dma32 0 pgskip_normal 0 pgskip_movable 0 pgfree 7222273815 pgactivate 1371753960 pgdeactivate 18329381 pglazyfree 10 pgfault 7795723861 pgmajfault 4600007 pglazyfreed 0 pgrefill 18575528 pgreuse 81910383 pgsteal_kswapd 980532060 pgsteal_direct 38942066 pgdemote_kswapd 0 pgdemote_direct 0 pgscan_kswapd 1135293298 pgscan_direct 58883653 pgscan_direct_throttle 15 pgscan_anon 220939938 pgscan_file 973237013 pgsteal_anon 46538607 pgsteal_file 972935519 zone_reclaim_failed 0 pginodesteal 0 slabs_scanned 25879882 kswapd_inodesteal 2179831 kswapd_low_wmark_hit_quickly 152797 kswapd_high_wmark_hit_quickly 32025 pageoutrun 204447 pgrotated 44963935 drop_pagecache 0 drop_slab 0 oom_kill 0 numa_pte_updates 2724410955 numa_huge_pte_updates 1695890 numa_hint_faults 1739823254 numa_hint_faults_local 1222358972 numa_pages_migrated 312611639 pgmigrate_success 510846802 pgmigrate_fail 875493 thp_migration_success 156413 thp_migration_fail 2 thp_migration_split 0 compact_migrate_scanned 1274073243 compact_free_scanned 8430842597 compact_isolated 400278352 compact_stall 145300 compact_fail 128562 compact_success 16738 compact_daemon_wake 170247 compact_daemon_migrate_scanned 35486283 compact_daemon_free_scanned 369870412 htlb_buddy_alloc_success 0 htlb_buddy_alloc_fail 0 unevictable_pgs_culled 2774290 unevictable_pgs_scanned 0 unevictable_pgs_rescued 2675031 unevictable_pgs_mlocked 2813622 unevictable_pgs_munlocked 2674972 unevictable_pgs_cleared 84231 unevictable_pgs_stranded 84225 thp_fault_alloc 416468 thp_fault_fallback 19181 thp_fault_fallback_charge 0 thp_collapse_alloc 17931 thp_collapse_alloc_failed 76 thp_file_alloc 0 thp_file_fallback 0 thp_file_fallback_charge 0 thp_file_mapped 0 thp_split_page 2 thp_split_page_failed 0 thp_deferred_split_page 66 thp_split_pmd 22451 thp_split_pud 0 thp_zero_page_alloc 1 thp_zero_page_alloc_failed 0 thp_swpout 22332 thp_swpout_fallback 0 balloon_inflate 0 balloon_deflate 0 balloon_migrate 0 swap_ra 25777929 swap_ra_hit 25658825 direct_map_level2_splits 1249 direct_map_level3_splits 49 nr_unstable 0
Özkan Göksu <ozkangksu@gmail.com>, 27 Oca 2024 Cmt, 02:36 tarihinde şunu yazdı:
Hello Frank.
I have 84 clients (high-end servers) with: Ubuntu 20.04.5 LTS - Kernel: Linux 5.4.0-125-generic
My cluster 17.2.6 quincy. I have some client nodes with "ceph-common/stable,now 17.2.7-1focal" I wonder using new version clients is the main problem? Maybe I have a communication error. For example I hit this problem and I can not collect client stats " https://github.com/ceph/ceph/pull/52127/files"
Best regards.
Frank Schilder <frans@dtu.dk>, 26 Oca 2024 Cum, 14:53 tarihinde şunu yazdı:
Hi, this message is one of those that are often spurious. I don't recall in which thread/PR/tracker I read it, but the story was something like that:
If an MDS gets under memory pressure it will request dentry items back from *all* clients, not just the active ones or the ones holding many of them. If you have a client that's below the min-threshold for dentries (its one of the client/mds tuning options), it will not respond. This client will be flagged as not responding, which is a false positive.
I believe the devs are working on a fix to get rid of these spurious warnings. There is a "bug/feature" in the MDS that does not clear this warning flag for inactive clients. Hence, the message hangs and never disappears. I usually clear it with a "echo 3 > /proc/sys/vm/drop_caches" on the client. However, except for being annoying in the dashboard, it has no performance or otherwise negative impact.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Friday, January 26, 2024 10:05 AM To: Özkan Göksu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: 1 clients failing to respond to cache pressure (quincy:17.2.6)
Performance for small files is more about IOPS rather than throughput, and the IOPS in your fio tests look okay to me. What you could try is to split the PGs to get around 150 or 200 PGs per OSD. You're currently at around 60 according to the ceph osd df output. Before you do that, can you share 'ceph pg ls-by-pool cephfs.ud-data.data | head'? I don't need the whole output, just to see how many objects each PG has. We had a case once where that helped, but it was an older cluster and the pool was backed by HDDs and separate rocksDB on SSDs. So this might not be the solution here, but it could improve things as well.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Every user has a 1x subvolume and I only have 1 pool. At the beginning we were using each subvolume for ldap home directory + user data. When a user logins any docker on any host, it was using the cluster for home and the for user related data, we was have second directory in the same subvolume. Time to time users were feeling a very slow home environment and after a month it became almost impossible to use home. VNC sessions became unresponsive and slow etc.
2 weeks ago, I had to migrate home to a ZFS storage and now the overall performance is better for only user_data without home. But still the performance is not good enough as I expected because of the problems related to MDS. The usage is low but allocation is high and Cpu usage is high. You saw the IO Op/s, it's nothing but allocation is high.
I develop a fio benchmark script and I run the script on 4x test server at the same time, the results are below: Script:
https://github.com/ozkangoksu/benchmark/blob/8f5df87997864c25ef32447e02fcd41...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
While running benchmark, I take sample values for each type of iobench
run.
Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 70 MiB/s rd, 762 MiB/s wr, 337 op/s rd, 24.41k op/s wr client: 60 MiB/s rd, 551 MiB/s wr, 303 op/s rd, 35.12k op/s wr client: 13 MiB/s rd, 161 MiB/s wr, 101 op/s rd, 41.30k op/s wr
Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 1.6 GiB/s rd, 219 KiB/s wr, 28.76k op/s rd, 89 op/s wr client: 370 MiB/s rd, 475 KiB/s wr, 90.38k op/s rd, 89 op/s wr
Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 63 MiB/s rd, 1.5 GiB/s wr, 8.77k op/s rd, 5.50k op/s wr client: 14 MiB/s rd, 1.8 GiB/s wr, 81 op/s rd, 13.86k op/s wr client: 6.6 MiB/s rd, 1.2 GiB/s wr, 61 op/s rd, 30.13k op/s wr
Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 317 MiB/s rd, 841 MiB/s wr, 426 op/s rd, 10.98k op/s wr client: 2.8 GiB/s rd, 882 MiB/s wr, 25.68k op/s rd, 291 op/s wr client: 4.0 GiB/s rd, 226 MiB/s wr, 89.63k op/s rd, 124 op/s wr client: 2.4 GiB/s rd, 295 KiB/s wr, 197.86k op/s rd, 20 op/s wr
It seems I only have problems with the 4K,8K,16K other sector sizes.
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 19:06 tarihinde şunu
yazdı:
I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
I will try my best to explain my situation.
I don't have a separate mds server. I have 5 identical nodes, 3 of
mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4)
root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure
services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M)
data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep
io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr
------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T
------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache
mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to
respond to
cache pressure client_id: 1275577
------------------------------ When I check the failing client with session ls I see only "num_caps: 12298"
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1
------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low.
avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20
------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used
by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better.
I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
> There is no definitive answer wrt mds tuning. As it is everywhere > mentioned, it's about finding the right setup for your specific > workload. If you can synthesize your workload (maybe scale down a bit) > try optimizing it in a test cluster without interrupting your > developers too much. > But what you haven't explained yet is what are you experiencing as a > performance issue? Do you have numbers or a detailed description? > From the fs status output you didn't seem to have too much activity > going on (around 140 requests per second), but that's probably not
them pressure directly the
> usual traffic? What does ceph report in its client IO output? > Can you paste the 'ceph osd df' output as well? > Do you have dedicated MDS servers or are they colocated with other > services? > > Zitat von Özkan Göksu <ozkangksu@gmail.com>: > > > Hello Eugen. > > > > I read all of your MDS related topics and thank you so much for your > effort > > on this. > > There is not much information and I couldn't find a MDS tuning guide at > > all. It seems that you are the correct person to discuss mds debugging > and > > tuning. > > > > Do you have any documents or may I learn what is the proper way to debug > > MDS and clients ? > > Which debug logs will guide me to understand the limitations and will > help > > to tune according to the data flow? > > > > While searching, I find this: > > >
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
> > quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic > > caps recall eventually starts recalling those but the (el7 kernel) client > > won't release them. Stopping Codium seems to be the only way to release." > > > > Because of this I think I also need to play around with the client side > too. > > > > My main goal is increasing the speed and reducing the latency and I > wonder > > if these ideas are correct or not: > > - Maybe I need to increase client side cache size because via each > client, > > multiple users request a lot of objects and clearly the > > client_cache_size=16 default is not enough. > > - Maybe I need to increase client side maximum cache limit for > > object "client_oc_max_objects=1000 to 10000" and data > "client_oc_size=200mi > > to 400mi" > > - The client cache cleaning threshold is not aggressive enough to keep > the > > free cache size in the desired range. I need to make it aggressive but > this > > should not reduce speed and increase latency. > > > > mds_cache_memory_limit=4gi to 16gi > > client_oc_max_objects=1000 to 10000 > > client_oc_size=200mi to 400mi > > client_permissions=false #to reduce latency. > > client_cache_size=16 to 128 > > > > > > What do you think? > > > >
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I decided to tune cephfs client's kernels and increase network buffers to increase speed. This time my client has 1x 10Gbit DAC cable. Client version is 1 step ahead: ceph-common/stable,now 17.2.7-1focal amd64 [installed] The kernel tunnings: root@maradona:~# cat /etc/sysctl.conf net.ipv4.tcp_syncookies = 0 # Disable syncookies (syncookies are not RFC compliant and can use too muche resources) net.ipv4.tcp_keepalive_time = 600 # Keepalive time for TCP connections (seconds) net.ipv4.tcp_synack_retries = 3 # Number of SYNACK retries before giving up net.ipv4.tcp_syn_retries = 3 # Number of SYN retries before giving up net.ipv4.tcp_rfc1337 = 1 # RFC1337 The set to 1 to enable RFC 1337 protection. net.ipv4.conf.all.log_martians = 1 # Log packets with impossible addresses to kernel log net.ipv4.inet_peer_gc_mintime = 5 # Minimum interval between garbage collection passes This interval is net.ipv4.tcp_ecn = 0 # Disable Explicit Congestion Notification in TCP net.ipv4.tcp_window_scaling = 1 # Enable window scaling as defined in RFC1323 net.ipv4.tcp_timestamps = 1 # Enable timestamps (RFC1323) net.ipv4.tcp_sack = 1 # Enable select acknowledgments net.ipv4.tcp_fack = 1 # Enable FACK congestion avoidance and fast restransmission net.ipv4.tcp_dsack = 1 # Allows TCP to send "duplicate" SACKs net.ipv4.ip_forward = 0 # Controls IP packet forwarding net.ipv4.conf.default.rp_filter = 0 # No controls source route verification (RFC1812) net.ipv4.tcp_tw_recycle = 1 # Enable fast recycling TIME-WAIT sockets net.ipv4.tcp_max_syn_backlog = 20000 # to keep TCP_SYNQ_HSIZE*16<=tcp_max_syn_backlog net.ipv4.tcp_max_orphans = 412520 # tells the kernel how many TCP sockets that are not attached to any user file handle to maintain net.ipv4.tcp_orphan_retries = 1 # How may times to retry before killing TCP connection, closed by our side net.ipv4.tcp_fin_timeout = 20 # how long to keep sockets in the state FIN-WAIT-2 if we were the one closing the socket net.ipv4.tcp_max_tw_buckets = 33001472 # maximum number of sockets in TIME-WAIT to be held simultaneously net.ipv4.tcp_no_metrics_save = 1 # don't cache ssthresh from previous connection net.ipv4.tcp_moderate_rcvbuf = 1 # don't cache ssthresh from previous connection net.ipv4.tcp_rmem = 4096 87380 16777216 # increase Linux autotuning TCP buffer limits net.ipv4.tcp_wmem = 4096 65536 16777216 # increase Linux autotuning TCP buffer limits # increase TCP max buffer size # net.core.rmem_max = 16777216 #try this if you get problems # net.core.wmem_max = 16777216 #try this if you get problems net.core.rmem_max = 67108864 net.core.wmem_max = 67108864 net.core.rmem_default = 262144 net.core.wmem_default = 262144 #net.core.netdev_max_backlog = 2500 #try this if you get problems net.core.netdev_max_backlog = 30000 net.core.somaxconn = 65000 net.ipv6.conf.all.disable_ipv6 = 1 # Disable ipv6 # You can monitor the kernel behavior with regard to the dirty # pages by using grep -A 1 dirty /proc/vmstat vm.dirty_background_ratio = 5 vm.dirty_ratio = 15 fs.file-max = 16500736 # system open file limit # Core dump kernel.core_pattern = /var/core_dumps/core.%e.%p.%h.%t fs.suid_dumpable = 2 # Kernel related tunnings kernel.printk = 4 4 1 7 kernel.core_uses_pid = 1 kernel.sysrq = 0 kernel.msgmax = 65536 kernel.msgmnb = 65536 kernel.shmmax = 243314299699 # Maximum shared segment size in bytes kernel.shmall = 66003228 # Maximum number of shared memory segments in pages vm.nr_hugepages = 4096 # Increase Transparent Huge Pages (THP) Defrag: vm.swappiness = 0 # Set vm.swappiness to 0 to minimize swapping vm.min_free_kbytes = 2640129 # required free memory (set to 1% of physical ram) iobenchmark result: Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M write: IOPS=1111, BW=1111MiB/s (1165MB/s)(3072MiB/2764msec); 0 zone resets BS=128K write: IOPS=3812, BW=477MiB/s (500MB/s)(3072MiB/6446msec); 0 zone resets BS=64K write: IOPS=5116, BW=320MiB/s (335MB/s)(3072MiB/9607msec); 0 zone resets BS=32K write: IOPS=6545, BW=205MiB/s (214MB/s)(3072MiB/15018msec); 0 zone resets BS=16K write: IOPS=8004, BW=125MiB/s (131MB/s)(3072MiB/24561msec); 0 zone resets BS=4K write: IOPS=8661, BW=33.8MiB/s (35.5MB/s)(3072MiB/90801msec); 0 zone resets Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M read: IOPS=1117, BW=1117MiB/s (1171MB/s)(3072MiB/2750msec) BS=128K read: IOPS=8353, BW=1044MiB/s (1095MB/s)(3072MiB/2942msec) BS=64K read: IOPS=11.8k, BW=739MiB/s (775MB/s)(3072MiB/4155msec) BS=32K read: IOPS=16.3k, BW=508MiB/s (533MB/s)(3072MiB/6049msec) BS=16K read: IOPS=23.0k, BW=375MiB/s (393MB/s)(3072MiB/8195msec) BS=4K read: IOPS=27.4k, BW=107MiB/s (112MB/s)(3072MiB/28740msec) Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M write: IOPS=1102, BW=1103MiB/s (1156MB/s)(3072MiB/2786msec); 0 zone resets BS=128K write: IOPS=8581, BW=1073MiB/s (1125MB/s)(3072MiB/2864msec); 0 zone resets BS=64K write: IOPS=10.9k, BW=681MiB/s (714MB/s)(3072MiB/4511msec); 0 zone resets BS=32K write: IOPS=12.1k, BW=378MiB/s (396MB/s)(3072MiB/8129msec); 0 zone resets BS=16K write: IOPS=12.7k, BW=198MiB/s (208MB/s)(3072MiB/15487msec); 0 zone resets BS=4K write: IOPS=12.7k, BW=49.7MiB/s (52.1MB/s)(3072MiB/61848msec); 0 zone resets Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M read: IOPS=1113, BW=1114MiB/s (1168MB/s)(3072MiB/2758msec) BS=128K read: IOPS=8953, BW=1119MiB/s (1173MB/s)(3072MiB/2745msec) BS=64K read: IOPS=17.9k, BW=1116MiB/s (1170MB/s)(3072MiB/2753msec) BS=32K read: IOPS=35.1k, BW=1096MiB/s (1150MB/s)(3072MiB/2802msec) BS=16K read: IOPS=69.4k, BW=1085MiB/s (1138MB/s)(3072MiB/2831msec) BS=4K read: IOPS=112k, BW=438MiB/s (459MB/s)(3072MiB/7015msec) *Everything looks good except 4K speeds:* Seq Write - BS=4K write: IOPS=8661, BW=33.8MiB/s (35.5MB/s)(3072MiB/90801msec); 0 zone resets Rand Write - BS=4K write: IOPS=12.7k, BW=49.7MiB/s (52.1MB/s)(3072MiB/61848msec); 0 zone resets What do you think? Özkan Göksu <ozkangksu@gmail.com>, 27 Oca 2024 Cmt, 04:08 tarihinde şunu yazdı:
Wow I noticed something!
To prevent ram overflow with gpu training allocations, I'm using a 2TB Samsung 870 evo for swap.
As you can see below, swap usage 18Gi and server was idle, that means maybe ceph client hits latency because of the swap usage.
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# free -h total used free shared buff/cache available Mem: 62Gi 34Gi 27Gi 0.0Ki 639Mi 27Gi Swap: 1.8Ti 18Gi 1.8Ti
I decided to play around kernel parameters to prevent ceph swap usage.
kernel.shmmax = 60654764851 # Maximum shared segment size in bytes
kernel.shmall = 16453658 # Maximum number of shared memory segments in pages vm.nr_hugepages = 4096 # Increase Transparent Huge Pages (THP) Defrag: vm.swappiness = 0 # Set vm.swappiness to 0 to minimize swapping vm.min_free_kbytes = 1048576 # required free memory (set to 1% of physical ram)
I reboot the server and after reboot swap usage is 0 as expected.
To give a try I started the iobench.sh ( https://github.com/ozkangoksu/benchmark/blob/main/iobench.sh) This client has 1G nic only. As you can see below, other then 4K block size, ceph client can saturate NIC.
root@bmw-m4:~# nicstat -MUz 1 Time Int rMbps wMbps rPk/s wPk/s rAvs wAvs %rUtil %wUtil 01:04:48 ens1f0 936.9 92.90 91196.8 60126.3 1346.6 202.5 98.2 9.74
root@bmw-m4:/mounts/ud-data/benchuser1/96f13211-c37f-42db-8d05-f3255a05129e/testdir# bash iobench.sh Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M write: IOPS=112, BW=112MiB/s (118MB/s)(3072MiB/27395msec); 0 zone resets BS=128K write: IOPS=894, BW=112MiB/s (117MB/s)(3072MiB/27462msec); 0 zone resets BS=64K write: IOPS=1758, BW=110MiB/s (115MB/s)(3072MiB/27948msec); 0 zone resets BS=32K write: IOPS=3542, BW=111MiB/s (116MB/s)(3072MiB/27748msec); 0 zone resets BS=16K write: IOPS=6839, BW=107MiB/s (112MB/s)(3072MiB/28747msec); 0 zone resets BS=4K write: IOPS=8473, BW=33.1MiB/s (34.7MB/s)(3072MiB/92813msec); 0 zone resets Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M read: IOPS=112, BW=112MiB/s (118MB/s)(3072MiB/27386msec) BS=128K read: IOPS=895, BW=112MiB/s (117MB/s)(3072MiB/27431msec) BS=64K read: IOPS=1788, BW=112MiB/s (117MB/s)(3072MiB/27486msec) BS=32K read: IOPS=3561, BW=111MiB/s (117MB/s)(3072MiB/27603msec) BS=16K read: IOPS=6924, BW=108MiB/s (113MB/s)(3072MiB/28392msec) BS=4K read: IOPS=21.3k, BW=83.3MiB/s (87.3MB/s)(3072MiB/36894msec) Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M write: IOPS=112, BW=112MiB/s (118MB/s)(3072MiB/27406msec); 0 zone resets BS=128K write: IOPS=894, BW=112MiB/s (117MB/s)(3072MiB/27466msec); 0 zone resets BS=64K write: IOPS=1781, BW=111MiB/s (117MB/s)(3072MiB/27591msec); 0 zone resets BS=32K write: IOPS=3545, BW=111MiB/s (116MB/s)(3072MiB/27729msec); 0 zone resets BS=16K write: IOPS=6823, BW=107MiB/s (112MB/s)(3072MiB/28814msec); 0 zone resets BS=4K write: IOPS=12.7k, BW=49.8MiB/s (52.2MB/s)(3072MiB/61694msec); 0 zone resets Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 BS=1M read: IOPS=112, BW=112MiB/s (118MB/s)(3072MiB/27388msec) BS=128K read: IOPS=894, BW=112MiB/s (117MB/s)(3072MiB/27479msec) BS=64K read: IOPS=1784, BW=112MiB/s (117MB/s)(3072MiB/27547msec) BS=32K read: IOPS=3559, BW=111MiB/s (117MB/s)(3072MiB/27614msec) BS=16K read: IOPS=7047, BW=110MiB/s (115MB/s)(3072MiB/27897msec) BS=4K read: IOPS=26.9k, BW=105MiB/s (110MB/s)(3072MiB/29199msec)
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1818702# cat metrics item total ------------------------------------------ opened files / total inodes 0 / 109 pinned i_caps / total inodes 109 / 109 opened inodes / total inodes 0 / 109
item total avg_lat(us) min_lat(us) max_lat(us) stdev(us)
----------------------------------------------------------------------------------- read 2316289 13904 221 8827984 760 write 2317824 21152 2975 9243821 2365 metadata 170 5944 225 202505 24314
item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)
---------------------------------------------------------------------------------------- read 2316289 16688 4096 1048576 38654712361 write 2317824 19457 4096 4194304 45097156608
item total miss hit ------------------------------------------------- d_lease 112 3 858 caps 109 58 6963547
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1818702# free -h total used free shared buff/cache available Mem: 62Gi 11Gi 50Gi 3.0Mi 1.0Gi 49Gi Swap: 1.8Ti 0B 1.8Ti
I started to feel we are getting closer :)
Özkan Göksu <ozkangksu@gmail.com>, 27 Oca 2024 Cmt, 02:58 tarihinde şunu yazdı:
I started to investigate my clients.
for example:
root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577
root@ud-01:~# ceph fs status ud-data - 86 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 34 /s 2926k 2827k 155k 1157k
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: *1275577*= num_caps: 12312, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275571= num_caps: 16307, num_leases: 1, request_load_avg: 2101, num_completed_requests: 0, num_completed_flushes: 3 clientid: 1282130= num_caps: 26337, num_leases: 3, request_load_avg: 116, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1191789= num_caps: 32784, num_leases: 0, request_load_avg: 1846, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275535= num_caps: 79825, num_leases: 2, request_load_avg: 133, num_completed_requests: 8, num_completed_flushes: 8 clientid: 1282142= num_caps: 80581, num_leases: 6, request_load_avg: 125, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1275532= num_caps: 87836, num_leases: 3, request_load_avg: 190, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1275547= num_caps: 94129, num_leases: 4, request_load_avg: 149, num_completed_requests: 2, num_completed_flushes: 4 clientid: 1275553= num_caps: 96460, num_leases: 4, request_load_avg: 155, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282139= num_caps: 108882, num_leases: 25, request_load_avg: 99, num_completed_requests: 2, num_completed_flushes: 4 clientid: 1275538= num_caps: 437162, num_leases: 0, request_load_avg: 101, num_completed_requests: 2, num_completed_flushes: 0
--------------------------------------
*MY CLIENT:*
The client is actually at idle mode and there is no reason to fail at all.
root@bmw-m4:~# apt list --installed |grep ceph ceph-common/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed] libcephfs2/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed,automatic] python3-ceph-argparse/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed,automatic] python3-ceph-common/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 all [installed,automatic] python3-cephfs/jammy-updates,now 17.2.6-0ubuntu0.22.04.2 amd64 [installed,automatic]
Let's check metrics and stats:
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# cat metrics item total ------------------------------------------ opened files / total inodes 2 / 12312 pinned i_caps / total inodes 12312 / 12312 opened inodes / total inodes 1 / 12312
item total avg_lat(us) min_lat(us) max_lat(us) stdev(us)
----------------------------------------------------------------------------------- read 22283 44409 430 1804853 15619 write 112702 419725 3658 8879541 6008 metadata 353322 5712 154 917903 5357
item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)
---------------------------------------------------------------------------------------- read 22283 1701940 1 4194304 37924318602 write 112702 246211 1 4194304 27748469309
item total miss hit ------------------------------------------------- d_lease 62 63627 28564698 caps 12312 36658 44568261
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# cat bdi/stats BdiWriteback: 0 kB BdiReclaimable: 800 kB BdiDirtyThresh: 0 kB DirtyThresh: 5795340 kB BackgroundThresh: 2894132 kB BdiDirtied: 27316320 kB BdiWritten: 27316320 kB BdiWriteBandwidth: 1472 kBps b_dirty: 0 b_io: 0 b_more_io: 0 b_dirty_time: 0 bdi_list: 1 state: 1
Last 3 days dmesg output:
[Wed Jan 24 16:45:13 2024] xfsettingsd[653036]: segfault at 18 ip 00007fbd12f5d337 sp 00007ffd254332a0 error 4 in libxklavier.so.16.4.0[7fbd12f4d000+19000] [Wed Jan 24 16:45:13 2024] Code: 4c 89 e7 e8 0b 56 ff ff 48 89 03 48 8b 5c 24 30 e9 d1 fd ff ff e8 b9 5b ff ff 66 0f 1f 84 00 00 00 00 00 41 54 55 48 89 f5 53 <48> 8b 42 18 48 89 d1 49 89 fc 48 89 d3 48 89 fa 48 89 ef 48 8b b0 [Thu Jan 25 06:51:31 2024] NVRM: GPU at PCI:0000:81:00: GPU-02efbb18-c9e4-3a16-d615-598959520b99 [Thu Jan 25 06:51:31 2024] NVRM: GPU Board Serial Number: 1321421015411 [Thu Jan 25 06:51:31 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=683281, name=python, Ch 00000008 [Thu Jan 25 06:56:49 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=683377, name=python, Ch 00000018 [Thu Jan 25 20:14:13 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=696062, name=python, Ch 00000008 [Fri Jan 26 04:05:40 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=700166, name=python, Ch 00000008 [Fri Jan 26 05:05:12 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=700320, name=python, Ch 00000008 [Fri Jan 26 05:44:50 2024] NVRM: GPU at PCI:0000:82:00: GPU-3af62a2c-e7eb-a7d5-c073-22f06dc7065f [Fri Jan 26 05:44:50 2024] NVRM: GPU Board Serial Number: 1321421010400 [Fri Jan 26 05:44:50 2024] NVRM: Xid (PCI:0000:82:00): 43, pid=700757, name=python, Ch 00000018 [Fri Jan 26 05:56:02 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=701096, name=python, Ch 00000028 [Fri Jan 26 06:34:20 2024] NVRM: Xid (PCI:0000:81:00): 43, pid=701226, name=python, Ch 00000038
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# free -h total used free shared buff/cache available Mem: 62Gi 34Gi 27Gi 0.0Ki 639Mi 27Gi Swap: 1.8Ti 18Gi 1.8Ti
root@bmw-m4:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1275577# cat /proc/vmstat nr_free_pages 7231171 nr_zone_inactive_anon 7924766 nr_zone_active_anon 525190 nr_zone_inactive_file 44029 nr_zone_active_file 55966 nr_zone_unevictable 13042 nr_zone_write_pending 3 nr_mlock 13042 nr_bounce 0 nr_zspages 0 nr_free_cma 0 numa_hit 6701928919 numa_miss 312628341 numa_foreign 312628341 numa_interleave 31538 numa_local 6701864751 numa_other 312692567 nr_inactive_anon 7924766 nr_active_anon 525190 nr_inactive_file 44029 nr_active_file 55966 nr_unevictable 13042 nr_slab_reclaimable 61076 nr_slab_unreclaimable 63509 nr_isolated_anon 0 nr_isolated_file 0 workingset_nodes 3934 workingset_refault_anon 30325493 workingset_refault_file 14593094 workingset_activate_anon 5376050 workingset_activate_file 3250679 workingset_restore_anon 292317 workingset_restore_file 1166673 workingset_nodereclaim 488665 nr_anon_pages 8451968 nr_mapped 35731 nr_file_pages 138824 nr_dirty 3 nr_writeback 0 nr_writeback_temp 0 nr_shmem 242 nr_shmem_hugepages 0 nr_shmem_pmdmapped 0 nr_file_hugepages 0 nr_file_pmdmapped 0 nr_anon_transparent_hugepages 3588 nr_vmscan_write 33746573 nr_vmscan_immediate_reclaim 160 nr_dirtied 48165341 nr_written 80207893 nr_kernel_misc_reclaimable 0 nr_foll_pin_acquired 174002 nr_foll_pin_released 174002 nr_kernel_stack 60032 nr_page_table_pages 46041 nr_swapcached 36166 nr_dirty_threshold 1448010 nr_dirty_background_threshold 723121 pgpgin 129904699 pgpgout 299261581 pswpin 30325493 pswpout 45158221 pgalloc_dma 1024 pgalloc_dma32 57788566 pgalloc_normal 6956384725 pgalloc_movable 0 allocstall_dma 0 allocstall_dma32 0 allocstall_normal 188 allocstall_movable 63024 pgskip_dma 0 pgskip_dma32 0 pgskip_normal 0 pgskip_movable 0 pgfree 7222273815 pgactivate 1371753960 pgdeactivate 18329381 pglazyfree 10 pgfault 7795723861 pgmajfault 4600007 pglazyfreed 0 pgrefill 18575528 pgreuse 81910383 pgsteal_kswapd 980532060 pgsteal_direct 38942066 pgdemote_kswapd 0 pgdemote_direct 0 pgscan_kswapd 1135293298 pgscan_direct 58883653 pgscan_direct_throttle 15 pgscan_anon 220939938 pgscan_file 973237013 pgsteal_anon 46538607 pgsteal_file 972935519 zone_reclaim_failed 0 pginodesteal 0 slabs_scanned 25879882 kswapd_inodesteal 2179831 kswapd_low_wmark_hit_quickly 152797 kswapd_high_wmark_hit_quickly 32025 pageoutrun 204447 pgrotated 44963935 drop_pagecache 0 drop_slab 0 oom_kill 0 numa_pte_updates 2724410955 numa_huge_pte_updates 1695890 numa_hint_faults 1739823254 numa_hint_faults_local 1222358972 numa_pages_migrated 312611639 pgmigrate_success 510846802 pgmigrate_fail 875493 thp_migration_success 156413 thp_migration_fail 2 thp_migration_split 0 compact_migrate_scanned 1274073243 compact_free_scanned 8430842597 compact_isolated 400278352 compact_stall 145300 compact_fail 128562 compact_success 16738 compact_daemon_wake 170247 compact_daemon_migrate_scanned 35486283 compact_daemon_free_scanned 369870412 htlb_buddy_alloc_success 0 htlb_buddy_alloc_fail 0 unevictable_pgs_culled 2774290 unevictable_pgs_scanned 0 unevictable_pgs_rescued 2675031 unevictable_pgs_mlocked 2813622 unevictable_pgs_munlocked 2674972 unevictable_pgs_cleared 84231 unevictable_pgs_stranded 84225 thp_fault_alloc 416468 thp_fault_fallback 19181 thp_fault_fallback_charge 0 thp_collapse_alloc 17931 thp_collapse_alloc_failed 76 thp_file_alloc 0 thp_file_fallback 0 thp_file_fallback_charge 0 thp_file_mapped 0 thp_split_page 2 thp_split_page_failed 0 thp_deferred_split_page 66 thp_split_pmd 22451 thp_split_pud 0 thp_zero_page_alloc 1 thp_zero_page_alloc_failed 0 thp_swpout 22332 thp_swpout_fallback 0 balloon_inflate 0 balloon_deflate 0 balloon_migrate 0 swap_ra 25777929 swap_ra_hit 25658825 direct_map_level2_splits 1249 direct_map_level3_splits 49 nr_unstable 0
Özkan Göksu <ozkangksu@gmail.com>, 27 Oca 2024 Cmt, 02:36 tarihinde şunu yazdı:
Hello Frank.
I have 84 clients (high-end servers) with: Ubuntu 20.04.5 LTS - Kernel: Linux 5.4.0-125-generic
My cluster 17.2.6 quincy. I have some client nodes with "ceph-common/stable,now 17.2.7-1focal" I wonder using new version clients is the main problem? Maybe I have a communication error. For example I hit this problem and I can not collect client stats " https://github.com/ceph/ceph/pull/52127/files"
Best regards.
Frank Schilder <frans@dtu.dk>, 26 Oca 2024 Cum, 14:53 tarihinde şunu yazdı:
Hi, this message is one of those that are often spurious. I don't recall in which thread/PR/tracker I read it, but the story was something like that:
If an MDS gets under memory pressure it will request dentry items back from *all* clients, not just the active ones or the ones holding many of them. If you have a client that's below the min-threshold for dentries (its one of the client/mds tuning options), it will not respond. This client will be flagged as not responding, which is a false positive.
I believe the devs are working on a fix to get rid of these spurious warnings. There is a "bug/feature" in the MDS that does not clear this warning flag for inactive clients. Hence, the message hangs and never disappears. I usually clear it with a "echo 3 > /proc/sys/vm/drop_caches" on the client. However, except for being annoying in the dashboard, it has no performance or otherwise negative impact.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Friday, January 26, 2024 10:05 AM To: Özkan Göksu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: 1 clients failing to respond to cache pressure (quincy:17.2.6)
Performance for small files is more about IOPS rather than throughput, and the IOPS in your fio tests look okay to me. What you could try is to split the PGs to get around 150 or 200 PGs per OSD. You're currently at around 60 according to the ceph osd df output. Before you do that, can you share 'ceph pg ls-by-pool cephfs.ud-data.data | head'? I don't need the whole output, just to see how many objects each PG has. We had a case once where that helped, but it was an older cluster and the pool was backed by HDDs and separate rocksDB on SSDs. So this might not be the solution here, but it could improve things as well.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Every user has a 1x subvolume and I only have 1 pool. At the beginning we were using each subvolume for ldap home directory
user data. When a user logins any docker on any host, it was using the cluster for home and the for user related data, we was have second directory in the same subvolume. Time to time users were feeling a very slow home environment and after a month it became almost impossible to use home. VNC sessions became unresponsive and slow etc.
2 weeks ago, I had to migrate home to a ZFS storage and now the overall performance is better for only user_data without home. But still the performance is not good enough as I expected because of the problems related to MDS. The usage is low but allocation is high and Cpu usage is high. You saw the IO Op/s, it's nothing but allocation is high.
I develop a fio benchmark script and I run the script on 4x test server at the same time, the results are below: Script:
https://github.com/ozkangoksu/benchmark/blob/8f5df87997864c25ef32447e02fcd41...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
While running benchmark, I take sample values for each type of
iobench run.
Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 70 MiB/s rd, 762 MiB/s wr, 337 op/s rd, 24.41k op/s wr client: 60 MiB/s rd, 551 MiB/s wr, 303 op/s rd, 35.12k op/s wr client: 13 MiB/s rd, 161 MiB/s wr, 101 op/s rd, 41.30k op/s wr
Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 1.6 GiB/s rd, 219 KiB/s wr, 28.76k op/s rd, 89 op/s wr client: 370 MiB/s rd, 475 KiB/s wr, 90.38k op/s rd, 89 op/s wr
Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 63 MiB/s rd, 1.5 GiB/s wr, 8.77k op/s rd, 5.50k op/s wr client: 14 MiB/s rd, 1.8 GiB/s wr, 81 op/s rd, 13.86k op/s wr client: 6.6 MiB/s rd, 1.2 GiB/s wr, 61 op/s rd, 30.13k op/s wr
Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 317 MiB/s rd, 841 MiB/s wr, 426 op/s rd, 10.98k op/s wr client: 2.8 GiB/s rd, 882 MiB/s wr, 25.68k op/s rd, 291 op/s wr client: 4.0 GiB/s rd, 226 MiB/s wr, 89.63k op/s rd, 124 op/s wr client: 2.4 GiB/s rd, 295 KiB/s wr, 197.86k op/s rd, 20 op/s wr
It seems I only have problems with the 4K,8K,16K other sector sizes.
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 19:06 tarihinde şunu
yazdı:
I understand that your MDS shows a high CPU usage, but other than
what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In
case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
> I will try my best to explain my situation. > > I don't have a separate mds server. I have 5 identical nodes, 3 of
> mons, and I use the other 2 as active and standby mds. (currently I have > left overs from max_mds 4) > > root@ud-01:~# ceph -s > cluster: > id: e42fd4b0-313b-11ee-9a00-31da71873773 > health: HEALTH_WARN > 1 clients failing to respond to cache pressure > > services: > mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) > mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd > mds: 1/1 daemons up, 4 standby > osd: 80 osds: 80 up (since 9d), 80 in (since 5M) > > data: > volumes: 1/1 healthy > pools: 3 pools, 2305 pgs > objects: 106.58M objects, 25 TiB > usage: 45 TiB used, 101 TiB / 146 TiB avail > pgs: 2303 active+clean > 2 active+clean+scrubbing+deep > > io: > client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr > > ------------------------------ > root@ud-01:~# ceph fs status > ud-data - 84 clients > ======= > RANK STATE MDS ACTIVITY DNS INOS DIRS > CAPS > 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k > 3048k > POOL TYPE USED AVAIL > cephfs.ud-data.meta metadata 136G 44.9T > cephfs.ud-data.data data 44.3T 44.9T > > ------------------------------ > root@ud-01:~# ceph health detail > HEALTH_WARN 1 clients failing to respond to cache pressure > [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache
> mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to > cache pressure client_id: 1275577 > > ------------------------------ > When I check the failing client with session ls I see only "num_caps: 12298" > > ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: > \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), > request_load_avg: \(.request_load_avg), num_completed_requests: > \(.num_completed_requests), num_completed_flushes: > \(.num_completed_flushes)"' | sort -n -t: -k3 > > clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, > num_completed_requests: 0, num_completed_flushes: 1 > clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, > num_completed_requests: 0, num_completed_flushes: 6 > clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, > num_completed_requests: 0, num_completed_flushes: 1 > clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, > num_completed_requests: 0, num_completed_flushes: 1 > clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, > num_completed_requests: 2, num_completed_flushes: 2 > clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, > num_completed_requests: 2, num_completed_flushes: 8 > clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, > num_completed_requests: 0, num_completed_flushes: 8 > clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, > num_completed_requests: 2, num_completed_flushes: 8 > clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, > num_completed_requests: 0, num_completed_flushes: 1 > clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, > num_completed_requests: 2, num_completed_flushes: 6 > clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, > num_completed_requests: 2, num_completed_flushes: 34 > clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, > num_completed_requests: 1, num_completed_flushes: 1 > clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, > num_completed_requests: 2, num_completed_flushes: 8 > clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, > num_completed_requests: 0, num_completed_flushes: 0 > clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, > num_completed_requests: 0, num_completed_flushes: 1 > clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, > num_completed_requests: 1, num_completed_flushes: 1 > clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, > num_completed_requests: 0, num_completed_flushes: 1 > > ------------------------------ > When I check the dashboard/service/mds I see %120+ CPU usage on active MDS > but on the host everything is almost idle and disk waits are very low. > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.61 0.00 0.38 0.41 0.00 98.60 > > Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s > wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm > d_await dareq-sz f/s f_await aqu-sz %util > sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 > 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 > 0.00 0.00 10.00 0.60 0.02 1.20 > sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 > 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 > 0.00 0.00 114.00 0.83 0.22 22.40 > sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 > 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 > 0.00 0.00 18.00 0.67 0.04 1.60 > sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 > 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 > 0.00 0.00 20.00 0.70 0.04 2.00 > sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 > 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 > 0.00 0.00 18.00 0.72 0.04 3.20 > sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 > 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 > 0.00 0.00 23.00 0.83 0.05 2.80 > sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 > 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 > 0.00 0.00 18.00 0.61 0.03 2.40 > sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 > 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 > 0.00 0.00 21.00 0.81 0.05 3.20 > > ------------------------------ > Other than this 5x node cluster, I also have a 3x node cluster with > identical hardware but it serves for a different purpose and data workload. > In this cluster I don't have any problem and MDS default settings seems > enough. > The only difference between two cluster is, 5x node cluster used
that that them pressure directly
> by users, 3x node cluster used heavily to read and write data via projects > not by users. So allocate and de-allocate will be better. > > I guess I just have a problematic use case on the 5x node cluster and as I > mentioned above, I might have the similar problem but I don't know how to > debug it. > >
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
> quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic > caps recall eventually starts recalling those but the (el7 kernel) client > won't release them. Stopping Codium seems to be the only way to release." > > ------------------------------ > Before reading the osd df you should know that I created 2x > OSD/per"CT4000MX500SSD1" > # ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS TYPE NAME > -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 > GiB 101 TiB 30.81 1.00 - root default > -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 > GiB 20 TiB 30.54 0.99 - host ud-01 > 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 > GiB 1.2 TiB 33.04 1.07 61 up osd.0 > 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 > GiB 1.3 TiB 28.28 0.92 53 up osd.1 > 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 > GiB 1.2 TiB 31.96 1.04 63 up osd.2 > 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 > GiB 1.3 TiB 28.30 0.92 55 up osd.3 > 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 > GiB 1.3 TiB 28.21 0.92 52 up osd.4 > 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 > GiB 1.2 TiB 31.76 1.03 61 up osd.5 > 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 > GiB 1.3 TiB 30.03 0.97 57 up osd.6 > 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 > GiB 1.2 TiB 32.32 1.05 58 up osd.7 > 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 > GiB 1.2 TiB 32.98 1.07 60 up osd.8 > 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 > GiB 1.3 TiB 30.67 1.00 61 up osd.9 > 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 > GiB 1.3 TiB 28.33 0.92 52 up osd.10 > 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 > GiB 1.3 TiB 29.57 0.96 56 up osd.11 > 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 > GiB 1.2 TiB 31.91 1.04 61 up osd.12 > 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 > GiB 1.3 TiB 30.10 0.98 55 up osd.13 > 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 > GiB 1.2 TiB 33.04 1.07 64 up osd.14 > 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 > GiB 1.3 TiB 28.20 0.92 51 up osd.15 > -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 > GiB 20 TiB 30.89 1.00 - host ud-02 > 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 > GiB 1.2 TiB 33.12 1.08 63 up osd.16 > 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 > GiB 1.3 TiB 31.26 1.01 59 up osd.17 > 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 > GiB 1.3 TiB 31.29 1.02 54 up osd.18 > 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 > GiB 1.3 TiB 29.50 0.96 56 up osd.19 > 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 > GiB 1.3 TiB 29.57 0.96 54 up osd.20 > 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 > GiB 1.2 TiB 33.04 1.07 60 up osd.21 > 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 > GiB 1.3 TiB 30.75 1.00 58 up osd.22 > 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 > GiB 1.2 TiB 33.06 1.07 60 up osd.23 > 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 > GiB 1.3 TiB 28.92 0.94 51 up osd.24 > 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 > GiB 1.3 TiB 31.27 1.02 61 up osd.25 > 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 > GiB 1.2 TiB 33.12 1.08 61 up osd.26 > 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 > GiB 1.3 TiB 28.84 0.94 53 up osd.27 > 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 > GiB 1.3 TiB 28.29 0.92 53 up osd.28 > 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 > GiB 1.2 TiB 31.91 1.04 59 up osd.29 > 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 > GiB 1.3 TiB 28.35 0.92 53 up osd.30 > 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 > GiB 1.2 TiB 31.89 1.03 61 up osd.31 > -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 > GiB 20 TiB 30.66 1.00 - host ud-03 > 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 > GiB 1.2 TiB 31.84 1.03 57 up osd.32 > 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 > GiB 1.2 TiB 33.13 1.08 63 up osd.33 > 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 > GiB 1.3 TiB 28.84 0.94 59 up osd.34 > 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 > GiB 1.3 TiB 30.16 0.98 58 up osd.35 > 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 > GiB 1.3 TiB 28.38 0.92 52 up osd.36 > 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 > GiB 1.3 TiB 28.28 0.92 55 up osd.37 > 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 > GiB 1.3 TiB 30.79 1.00 55 up osd.38 > 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 > GiB 1.2 TiB 32.48 1.05 61 up osd.39 > 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 > GiB 1.3 TiB 30.76 1.00 56 up osd.40 > 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 > GiB 1.3 TiB 28.21 0.92 54 up osd.41 > 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 > GiB 1.2 TiB 32.91 1.07 58 up osd.42 > 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 > GiB 1.2 TiB 32.51 1.06 61 up osd.43 > 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 > GiB 1.3 TiB 31.29 1.02 60 up osd.44 > 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 > GiB 1.2 TiB 33.18 1.08 62 up osd.45 > 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 > GiB 1.3 TiB 29.50 0.96 54 up osd.46 > 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 > GiB 1.3 TiB 28.25 0.92 50 up osd.47 > -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 > GiB 20 TiB 31.04 1.01 - host ud-04 > 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 > GiB 1.3 TiB 28.96 0.94 58 up osd.48 > 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 > GiB 1.2 TiB 33.11 1.07 61 up osd.49 > 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 > GiB 1.2 TiB 33.17 1.08 61 up osd.50 > 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 > GiB 1.2 TiB 33.19 1.08 61 up osd.51 > 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 > GiB 1.3 TiB 28.25 0.92 53 up osd.52 > 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 > GiB 1.2 TiB 33.17 1.08 66 up osd.53 > 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 > GiB 1.3 TiB 29.54 0.96 55 up osd.54 > 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 > GiB 1.3 TiB 28.29 0.92 52 up osd.55 > 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 > GiB 1.3 TiB 28.16 0.91 52 up osd.56 > 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 > GiB 1.2 TiB 33.03 1.07 65 up osd.57 > 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 > GiB 1.3 TiB 28.31 0.92 55 up osd.58 > 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 > GiB 1.2 TiB 33.01 1.07 60 up osd.59 > 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 > GiB 1.2 TiB 31.88 1.03 59 up osd.60 > 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 > GiB 1.2 TiB 33.04 1.07 64 up osd.61 > 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 > GiB 1.2 TiB 33.27 1.08 63 up osd.62 > 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 > GiB 1.3 TiB 28.30 0.92 53 up osd.63 > -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 > GiB 20 TiB 30.91 1.00 - host ud-05 > 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 > GiB 1.2 TiB 32.62 1.06 65 up osd.64 > 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 > GiB 1.2 TiB 32.53 1.06 57 up osd.65 > 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 > GiB 1.2 TiB 31.31 1.02 57 up osd.66 > 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 > GiB 1.3 TiB 28.82 0.94 50 up osd.67 > 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 > GiB 1.3 TiB 29.04 0.94 59 up osd.68 > 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 > GiB 1.2 TiB 32.55 1.06 59 up osd.69 > 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 > GiB 1.2 TiB 32.44 1.05 63 up osd.70 > 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 > GiB 1.2 TiB 32.53 1.06 62 up osd.71 > 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 > GiB 1.2 TiB 32.33 1.05 57 up osd.72 > 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 > GiB 1.3 TiB 30.65 0.99 58 up osd.73 > 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 > GiB 1.2 TiB 32.62 1.06 61 up osd.74 > 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 > GiB 1.3 TiB 28.80 0.93 57 up osd.75 > 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 > GiB 1.2 TiB 32.48 1.05 60 up osd.76 > 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 > GiB 1.3 TiB 28.84 0.94 52 up osd.77 > 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 > GiB 1.3 TiB 28.20 0.92 52 up osd.78 > 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 > GiB 1.3 TiB 28.76 0.93 53 up osd.79 > TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 > GiB 101 TiB 30.81 > MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90 > > > > Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı: > >> There is no definitive answer wrt mds tuning. As it is everywhere >> mentioned, it's about finding the right setup for your specific >> workload. If you can synthesize your workload (maybe scale down a bit) >> try optimizing it in a test cluster without interrupting your >> developers too much. >> But what you haven't explained yet is what are you experiencing as a >> performance issue? Do you have numbers or a detailed description? >> From the fs status output you didn't seem to have too much activity >> going on (around 140 requests per second), but that's probably not the >> usual traffic? What does ceph report in its client IO output? >> Can you paste the 'ceph osd df' output as well? >> Do you have dedicated MDS servers or are they colocated with other >> services? >> >> Zitat von Özkan Göksu <ozkangksu@gmail.com>: >> >> > Hello Eugen. >> > >> > I read all of your MDS related topics and thank you so much for your >> effort >> > on this. >> > There is not much information and I couldn't find a MDS tuning guide at >> > all. It seems that you are the correct person to discuss mds debugging >> and >> > tuning. >> > >> > Do you have any documents or may I learn what is the proper way to debug >> > MDS and clients ? >> > Which debug logs will guide me to understand the limitations and will >> help >> > to tune according to the data flow? >> > >> > While searching, I find this: >> > >>
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
>> > quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic >> > caps recall eventually starts recalling those but the (el7 kernel) client >> > won't release them. Stopping Codium seems to be the only way to release." >> > >> > Because of this I think I also need to play around with the client side >> too. >> > >> > My main goal is increasing the speed and reducing the latency and I >> wonder >> > if these ideas are correct or not: >> > - Maybe I need to increase client side cache size because via each >> client, >> > multiple users request a lot of objects and clearly the >> > client_cache_size=16 default is not enough. >> > - Maybe I need to increase client side maximum cache limit for >> > object "client_oc_max_objects=1000 to 10000" and data >> "client_oc_size=200mi >> > to 400mi" >> > - The client cache cleaning threshold is not aggressive enough to keep >> the >> > free cache size in the desired range. I need to make it aggressive but >> this >> > should not reduce speed and increase latency. >> > >> > mds_cache_memory_limit=4gi to 16gi >> > client_oc_max_objects=1000 to 10000 >> > client_oc_size=200mi to 400mi >> > client_permissions=false #to reduce latency. >> > client_cache_size=16 to 128 >> > >> > >> > What do you think? >> >> >> >>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Özkan,
... The client is actually at idle mode and there is no reason to fail at all. ...
if you re-read my message, you will notice that I wrote that - its not the client failing, its a false positive error flag that - is not cleared for idle clients. You seem to encounter exactly this situation and a simple echo 3 > /proc/sys/vm/drop_caches would probably have cleared the warning. There is nothing wrong with your client, its an issue with the client-MDS communication protocol that is probably still under review. You will encounter these warnings every now and then until its fixed. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Thank you Frank. My focus is actually performance tuning. After your mail, I started to investigate client-side. I think the kernel tunings work great now. After the tunings I didn't get any warning again. Now I will continue with performance tunings. I decided to distribute subvolumes across multiple pools instead of multi-active-mds. With this method I will have multiple MDS and [1x cephfs clients for each pool / Host] To hide subvolume uuids, I'm using "mount --bind kernel links" and I wonder is it able to create performance issues on cephfs clients? Best regards. Frank Schilder <frans@dtu.dk>, 27 Oca 2024 Cmt, 12:34 tarihinde şunu yazdı:
Hi Özkan,
... The client is actually at idle mode and there is no reason to fail at all. ...
if you re-read my message, you will notice that I wrote that
- its not the client failing, its a false positive error flag that - is not cleared for idle clients.
You seem to encounter exactly this situation and a simple
echo 3 > /proc/sys/vm/drop_caches
would probably have cleared the warning. There is nothing wrong with your client, its an issue with the client-MDS communication protocol that is probably still under review. You will encounter these warnings every now and then until its fixed.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm not sure if I understand correctly:
I decided to distribute subvolumes across multiple pools instead of multi-active-mds. With this method I will have multiple MDS and [1x cephfs clients for each pool / Host]
Those two statements contradict each other, either you have multi-active MDS or not. Great that you were able to tune your clients, that's really interesting although I haven't looked too deep into your results. But do the actual clients reflect the same improvement (if you already tested that) or was the improvement only for you fio tests? Neverthelesse, quite good IOPS! Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Thank you Frank.
My focus is actually performance tuning. After your mail, I started to investigate client-side.
I think the kernel tunings work great now. After the tunings I didn't get any warning again.
Now I will continue with performance tunings. I decided to distribute subvolumes across multiple pools instead of multi-active-mds. With this method I will have multiple MDS and [1x cephfs clients for each pool / Host]
To hide subvolume uuids, I'm using "mount --bind kernel links" and I wonder is it able to create performance issues on cephfs clients?
Best regards.
Frank Schilder <frans@dtu.dk>, 27 Oca 2024 Cmt, 12:34 tarihinde şunu yazdı:
Hi Özkan,
... The client is actually at idle mode and there is no reason to fail at all. ...
if you re-read my message, you will notice that I wrote that
- its not the client failing, its a false positive error flag that - is not cleared for idle clients.
You seem to encounter exactly this situation and a simple
echo 3 > /proc/sys/vm/drop_caches
would probably have cleared the warning. There is nothing wrong with your client, its an issue with the client-MDS communication protocol that is probably still under review. You will encounter these warnings every now and then until its fixed.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, this is not an easy topic and there is no formula that can be applied to all clusters. From my experience, it is exactly how the discussion went in the thread you mentioned, trial & error. Looking at your session ls output, this reminds of a debug session we had a few years ago: "recall_caps": { "value": 2577232.0049106553, "halflife": 60 It appears to be what we observed as well, and reducing mds_recall_max_caps helped us so the recall_caps value wouldn't pile up.
I'm not trying to solve cache pressure warning.
Your first mail proves the opposite ;-)
I have 2 questions: 1- What should I do to prevent cache pressue warning ?
I'm trying to increase the speed by creating multiple MDS even maybe binding subvolumes to specific MDS servers and decrease the latency.
And what is your result? Did it help? If you have multiple active MDS you might encounter a performance degredation if you don't use pinning because the mds balancers impact each other. We were successful with multi-active MDS and directory pinning.
Also when I check MDS CPU usage I see %120++ usage time to time. But when I check the server CPU load at MDS location, I see MDS only uses 2-4 cores and other CPU cores are almost at idle. I think MDS has a CPU core limitation and I need to increase the value to decrease the latency. How can I do that?
Yes, MDS is single-threaded. That's why we deployed multiple daemons per server (in the above mentioned cluster) and used multi-active MDS with pinning. Regards, Eugen Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Let me share some outputs about my cluster.
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 31 /s 3022k 3021k 52.6k 385k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.4T cephfs.ud-data.data data 45.2T 44.4T STANDBY MDS ud-data.ud-03.lhwkml ud-data.ud-05.rnhcfe ud-data.ud-01.uatjle ud-data.ud-04.seggyv
-------------------------------------------------------------------------- This is "ceph tell mds.ud-data.ud-02.xcoojt session ls" output for the reported client for cache pressure warning.
{ "id": 1282205, "entity": { "name": { "type": "client", "num": 1282205 }, "addr": { "type": "v1", "addr": "172.16.3.48:0", "nonce": 2169935642 } }, "state": "open", "num_leases": 0, "num_caps": 52092, "request_load_avg": 1, "uptime": 75754.745608647994, "requests_in_flight": 0, "num_completed_requests": 0, "num_completed_flushes": 1, "reconnecting": false, "recall_caps": { "value": 2577232.0049106553, "halflife": 60 }, "release_caps": { "value": 1.4093491463510395, "halflife": 60 }, "recall_caps_throttle": { "value": 63733.985544098425, "halflife": 1.5 }, "recall_caps_throttle2o": { "value": 19452.428409271757, "halflife": 0.5 }, "session_cache_liveness": { "value": 14.100272208890081, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "delegated_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 } ], "inst": "client.1282205 v1:172.16.3.48:0/2169935642", "completed_requests": [], "prealloc_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 }, { "start": "0x1000fa86e5f", "length": 54 }, { "start": "0x1000faa069c", "length": 501 } ], "client_metadata": { "client_features": { "feature_bits": "0x0000000000007bff" }, "metric_spec": { "metric_flags": { "feature_bits": "0x00000000000003ff" } }, "entity_id": "admin", "hostname": "bennevis-2", "kernel_version": "5.15.0-91-generic", "root": "/volumes/babblians" } }
Özkan Göksu <ozkangksu@gmail.com>, 17 Oca 2024 Çar, 07:22 tarihinde şunu yazdı:
Hello Eugen.
Thank you for the answer. According to knowledge and test results at this issue: https://github.com/ceph/ceph/pull/38574 I've tried their advice and I've applied the following changes.
max_mds = 4 standby_mds = 1 mds_cache_memory_limit = 16GB mds_recall_max_caps = 40000
When I set these parameters, 1 day later I saw this log: [8531248.982954] Out of memory: Killed process 1580586 (ceph-mds) total-vm:70577592kB, anon-rss:70244236kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:137832kB oom_score_adj:0
All the MDS services created memory leak and killed by kernel. Because of this I changed it as below and it is stable now but performance is very poor and I still get cache pressure alerts.
max_mds = 1 standby_mds = 5 mds_cache_memory_limit = 8GB mds_recall_max_caps = 30000
I'm very surprised that you are advising to decrease "mds_recall_max_caps" because it is the opposite of what developers advised in the issue I've sended. It is very hard to play around with MDS parameters without expert level of understanding what these parameters stands for and how it will effect the behavior. Because of this I'm trying to understand the MDS code flow and I'm very interested with learning more and tuning my system by debugging and understanding my own data flow and MDS usage.
I have a very unique data flow and I think I need to configure the system for this case. I have 80+ clients and via all of these clients my users are requesting Read a range of objects and compare them in GPU, they generate new data and Write the new data back in the cluster. So it means my clients usually reads objects only one time and do not read the same object again. Sometimes same user runs multiple service in multiple clients and these services can read the same data from different clients.
So having a large cache is useless for my use case. I need to setup MDS and Cephfs Client for this data flow. When I debug the MDS ram usage, I see high allocation all the time and I wonder why? If any of my client does not read any object why MDS does not remove that data from ram allocation? I need to configure MDS for reading the data and removing it very fast if the data is constantly requested from clients. In this case ofc I want a ram cache tier.
I'm little confused and I need to learn more about how MDS works and how should I make multiple active MDS faster for my subvolumes and client data flow.
Best regards.
Eugen Block <eblock@nde.ag>, 16 Oca 2024 Sal, 11:36 tarihinde şunu yazdı:
Hi,
I have dealt with this topic multiple times, the SUSE team helped understanding what's going on under the hood. The summary can be found in this thread [1].
What helped in our case was to reduce the mds_recall_max_caps from 30k (default) to 3k. We tried it in steps of 1k IIRC. So I suggest to reduce that value step by step (maybe start with 20k or something) to find the optimal value.
Regards, Eugen
[1] https://www.spinics.net/lists/ceph-users/msg73188.html
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello.
I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning.
I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool.
My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data)
---------------------------------------------------------------------------------
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB
---------------------------------------------------------------------------------
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
-----------------------------------------------------------------------------------
My MDS settings are below:
mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144
I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ?
- Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 1/17/24 15:57, Eugen Block wrote:
Hi,
this is not an easy topic and there is no formula that can be applied to all clusters. From my experience, it is exactly how the discussion went in the thread you mentioned, trial & error. Looking at your session ls output, this reminds of a debug session we had a few years ago:
"recall_caps": { "value": 2577232.0049106553, "halflife": 60
It appears to be what we observed as well, and reducing mds_recall_max_caps helped us so the recall_caps value wouldn't pile up.
I'm not trying to solve cache pressure warning.
Your first mail proves the opposite ;-)
I have 2 questions: 1- What should I do to prevent cache pressue warning ?
I'm trying to increase the speed by creating multiple MDS even maybe binding subvolumes to specific MDS servers and decrease the latency.
And what is your result? Did it help? If you have multiple active MDS you might encounter a performance degredation if you don't use pinning because the mds balancers impact each other. We were successful with multi-active MDS and directory pinning.
Also when I check MDS CPU usage I see %120++ usage time to time. But when I check the server CPU load at MDS location, I see MDS only uses 2-4 cores and other CPU cores are almost at idle. I think MDS has a CPU core limitation and I need to increase the value to decrease the latency. How can I do that?
Yes, MDS is single-threaded. That's why we deployed multiple daemons per server (in the above mentioned cluster) and used multi-active MDS with pinning.
IMO also could disable the balancer by setting 'mds_bal_interval' to 0. Sometimes the balancer will introduce the cache pressure for multi-active MDS. Thanks - Xiubo
Regards, Eugen
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Let me share some outputs about my cluster.
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 31 /s 3022k 3021k 52.6k 385k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.4T cephfs.ud-data.data data 45.2T 44.4T STANDBY MDS ud-data.ud-03.lhwkml ud-data.ud-05.rnhcfe ud-data.ud-01.uatjle ud-data.ud-04.seggyv
--------------------------------------------------------------------------
This is "ceph tell mds.ud-data.ud-02.xcoojt session ls" output for the reported client for cache pressure warning.
{ "id": 1282205, "entity": { "name": { "type": "client", "num": 1282205 }, "addr": { "type": "v1", "addr": "172.16.3.48:0", "nonce": 2169935642 } }, "state": "open", "num_leases": 0, "num_caps": 52092, "request_load_avg": 1, "uptime": 75754.745608647994, "requests_in_flight": 0, "num_completed_requests": 0, "num_completed_flushes": 1, "reconnecting": false, "recall_caps": { "value": 2577232.0049106553, "halflife": 60 }, "release_caps": { "value": 1.4093491463510395, "halflife": 60 }, "recall_caps_throttle": { "value": 63733.985544098425, "halflife": 1.5 }, "recall_caps_throttle2o": { "value": 19452.428409271757, "halflife": 0.5 }, "session_cache_liveness": { "value": 14.100272208890081, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "delegated_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 } ], "inst": "client.1282205 v1:172.16.3.48:0/2169935642", "completed_requests": [], "prealloc_inos": [ { "start": "0x10004a1c031", "length": 282 }, { "start": "0x10004a1c33f", "length": 207 }, { "start": "0x10004a1cdda", "length": 6 }, { "start": "0x10004a3c12e", "length": 3 }, { "start": "0x1000f9831fe", "length": 2 }, { "start": "0x1000fa86e5f", "length": 54 }, { "start": "0x1000faa069c", "length": 501 } ], "client_metadata": { "client_features": { "feature_bits": "0x0000000000007bff" }, "metric_spec": { "metric_flags": { "feature_bits": "0x00000000000003ff" } }, "entity_id": "admin", "hostname": "bennevis-2", "kernel_version": "5.15.0-91-generic", "root": "/volumes/babblians" } }
Özkan Göksu <ozkangksu@gmail.com>, 17 Oca 2024 Çar, 07:22 tarihinde şunu yazdı:
Hello Eugen.
Thank you for the answer. According to knowledge and test results at this issue: https://github.com/ceph/ceph/pull/38574 I've tried their advice and I've applied the following changes.
max_mds = 4 standby_mds = 1 mds_cache_memory_limit = 16GB mds_recall_max_caps = 40000
When I set these parameters, 1 day later I saw this log: [8531248.982954] Out of memory: Killed process 1580586 (ceph-mds) total-vm:70577592kB, anon-rss:70244236kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:137832kB oom_score_adj:0
All the MDS services created memory leak and killed by kernel. Because of this I changed it as below and it is stable now but performance is very poor and I still get cache pressure alerts.
max_mds = 1 standby_mds = 5 mds_cache_memory_limit = 8GB mds_recall_max_caps = 30000
I'm very surprised that you are advising to decrease "mds_recall_max_caps" because it is the opposite of what developers advised in the issue I've sended. It is very hard to play around with MDS parameters without expert level of understanding what these parameters stands for and how it will effect the behavior. Because of this I'm trying to understand the MDS code flow and I'm very interested with learning more and tuning my system by debugging and understanding my own data flow and MDS usage.
I have a very unique data flow and I think I need to configure the system for this case. I have 80+ clients and via all of these clients my users are requesting Read a range of objects and compare them in GPU, they generate new data and Write the new data back in the cluster. So it means my clients usually reads objects only one time and do not read the same object again. Sometimes same user runs multiple service in multiple clients and these services can read the same data from different clients.
So having a large cache is useless for my use case. I need to setup MDS and Cephfs Client for this data flow. When I debug the MDS ram usage, I see high allocation all the time and I wonder why? If any of my client does not read any object why MDS does not remove that data from ram allocation? I need to configure MDS for reading the data and removing it very fast if the data is constantly requested from clients. In this case ofc I want a ram cache tier.
I'm little confused and I need to learn more about how MDS works and how should I make multiple active MDS faster for my subvolumes and client data flow.
Best regards.
Eugen Block <eblock@nde.ag>, 16 Oca 2024 Sal, 11:36 tarihinde şunu yazdı:
Hi,
I have dealt with this topic multiple times, the SUSE team helped understanding what's going on under the hood. The summary can be found in this thread [1].
What helped in our case was to reduce the mds_recall_max_caps from 30k (default) to 3k. We tried it in steps of 1k IIRC. So I suggest to reduce that value step by step (maybe start with 20k or something) to find the optimal value.
Regards, Eugen
[1] https://www.spinics.net/lists/ceph-users/msg73188.html
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello.
I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning.
I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool.
My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data)
---------------------------------------------------------------------------------
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB
---------------------------------------------------------------------------------
root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
-----------------------------------------------------------------------------------
My MDS settings are below:
mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144
I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ?
- Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 1/13/24 07:02, Özkan Göksu wrote:
Hello.
I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning.
I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool.
My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data)
--------------------------------------------------------------------------------- --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB --------------------------------------------------------------------------------- root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
----------------------------------------------------------------------------------- My MDS settings are below:
mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144
Have you tried to set both 'mds_max_caps_per_client' and 'mds_cache_trim_threshold' to smaller sizes ? Thanks - Xiubo
I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ?
- Thanks _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
Frank Schilder
-
Xiubo Li
-
Özkan Göksu