This is client side metrics from a "failing to respond to cache pressure" warned client. root@datagen-27:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1282187# cat bdi/stats BdiWriteback: 0 kB BdiReclaimable: 0 kB BdiDirtyThresh: 0 kB DirtyThresh: 35979376 kB BackgroundThresh: 17967720 kB BdiDirtied: 3071616 kB BdiWritten: 3036864 kB BdiWriteBandwidth: 20 kBps b_dirty: 0 b_io: 0 b_more_io: 0 b_dirty_time: 0 bdi_list: 1 state: 1 ------------------------------------------------ root@d27:/sys/kernel/debug/ceph/e42fd4b0-313b-11ee-9a00-31da71873773.client1282187# cat metrics item total ------------------------------------------ opened files / total inodes 4 / 14129 pinned i_caps / total inodes 14129 / 14129 opened inodes / total inodes 2 / 14129 item total avg_lat(us) min_lat(us) max_lat(us) stdev(us) ----------------------------------------------------------------------------------- read 1218753 3116 208 8741271 2154 write 34945 24003 3017 2191493 16156 metadata 1703642 8395 127 17936115 1497 item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes) ---------------------------------------------------------------------------------------- read 1218753 227009 1 4194304 276668475618 write 34945 85860 1 4194304 3000382055 item total miss hit ------------------------------------------------- d_lease 306 19110 3317071969 caps 14129 145404 3761682333 Özkan Göksu <ozkangksu@gmail.com>, 25 Oca 2024 Per, 20:25 tarihinde şunu yazdı:
Every user has a 1x subvolume and I only have 1 pool. At the beginning we were using each subvolume for ldap home directory + user data. When a user logins any docker on any host, it was using the cluster for home and the for user related data, we was have second directory in the same subvolume. Time to time users were feeling a very slow home environment and after a month it became almost impossible to use home. VNC sessions became unresponsive and slow etc.
2 weeks ago, I had to migrate home to a ZFS storage and now the overall performance is better for only user_data without home. But still the performance is not good enough as I expected because of the problems related to MDS. The usage is low but allocation is high and Cpu usage is high. You saw the IO Op/s, it's nothing but allocation is high.
I develop a fio benchmark script and I run the script on 4x test server at the same time, the results are below: Script: https://github.com/ozkangoksu/benchmark/blob/8f5df87997864c25ef32447e02fcd41...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
https://github.com/ozkangoksu/benchmark/blob/main/benchmark-results/iobench-...
While running benchmark, I take sample values for each type of iobench run.
Seq Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 70 MiB/s rd, 762 MiB/s wr, 337 op/s rd, 24.41k op/s wr client: 60 MiB/s rd, 551 MiB/s wr, 303 op/s rd, 35.12k op/s wr client: 13 MiB/s rd, 161 MiB/s wr, 101 op/s rd, 41.30k op/s wr
Seq Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 1.6 GiB/s rd, 219 KiB/s wr, 28.76k op/s rd, 89 op/s wr client: 370 MiB/s rd, 475 KiB/s wr, 90.38k op/s rd, 89 op/s wr
Rand Write benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 63 MiB/s rd, 1.5 GiB/s wr, 8.77k op/s rd, 5.50k op/s wr client: 14 MiB/s rd, 1.8 GiB/s wr, 81 op/s rd, 13.86k op/s wr client: 6.6 MiB/s rd, 1.2 GiB/s wr, 61 op/s rd, 30.13k op/s wr
Rand Read benchmarking: size=1G,direct=1,numjobs=3,iodepth=32 client: 317 MiB/s rd, 841 MiB/s wr, 426 op/s rd, 10.98k op/s wr client: 2.8 GiB/s rd, 882 MiB/s wr, 25.68k op/s rd, 291 op/s wr client: 4.0 GiB/s rd, 226 MiB/s wr, 89.63k op/s rd, 124 op/s wr client: 2.4 GiB/s rd, 295 KiB/s wr, 197.86k op/s rd, 20 op/s wr
It seems I only have problems with the 4K,8K,16K other sector sizes.
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 19:06 tarihinde şunu yazdı:
I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks? But first I’d still like to know what the performance issue really is.
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
I will try my best to explain my situation.
I don't have a separate mds server. I have 5 identical nodes, 3 of them mons, and I use the other 2 as active and standby mds. (currently I have left overs from max_mds 4)
root@ud-01:~# ceph -s cluster: id: e42fd4b0-313b-11ee-9a00-31da71873773 health: HEALTH_WARN 1 clients failing to respond to cache pressure
services: mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d) mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd mds: 1/1 daemons up, 4 standby osd: 80 osds: 80 up (since 9d), 80 in (since 5M)
data: volumes: 1/1 healthy pools: 3 pools, 2305 pgs objects: 106.58M objects, 25 TiB usage: 45 TiB used, 101 TiB / 146 TiB avail pgs: 2303 active+clean 2 active+clean+scrubbing+deep
io: client: 16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr
------------------------------ root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-02.xcoojt Reqs: 40 /s 2579k 2578k 169k 3048k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 136G 44.9T cephfs.ud-data.data data 44.3T 44.9T
------------------------------ root@ud-01:~# ceph health detail HEALTH_WARN 1 clients failing to respond to cache pressure [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to cache pressure client_id: 1275577
------------------------------ When I check the failing client with session ls I see only "num_caps: 12298"
ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid: \(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases), request_load_avg: \(.request_load_avg), num_completed_requests: \(.num_completed_requests), num_completed_flushes: \(.num_completed_flushes)"' | sort -n -t: -k3
clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105, num_completed_requests: 0, num_completed_flushes: 6 clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57, num_completed_requests: 2, num_completed_flushes: 2 clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128, num_completed_requests: 0, num_completed_flushes: 8 clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119, num_completed_requests: 2, num_completed_flushes: 6 clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84, num_completed_requests: 2, num_completed_flushes: 34 clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115, num_completed_requests: 2, num_completed_flushes: 8 clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753, num_completed_requests: 0, num_completed_flushes: 0 clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2, num_completed_requests: 0, num_completed_flushes: 1 clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8, num_completed_requests: 1, num_completed_flushes: 1 clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0, num_completed_requests: 0, num_completed_flushes: 1
------------------------------ When I check the dashboard/service/mds I see %120+ CPU usage on active MDS but on the host everything is almost idle and disk waits are very low.
avg-cpu: %user %nice %system %iowait %steal %idle 0.61 0.00 0.38 0.41 0.00 98.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdc 2.00 0.01 0.00 0.00 0.50 6.00 20.00 0.04 0.00 0.00 0.50 2.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 0.60 0.02 1.20 sdd 3.00 0.02 0.00 0.00 0.67 8.00 285.00 1.84 77.00 21.27 0.44 6.61 0.00 0.00 0.00 0.00 0.00 0.00 114.00 0.83 0.22 22.40 sde 1.00 0.01 0.00 0.00 1.00 8.00 36.00 0.08 3.00 7.69 0.64 2.33 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.67 0.04 1.60 sdf 5.00 0.04 0.00 0.00 0.40 7.20 40.00 0.09 3.00 6.98 0.53 2.30 0.00 0.00 0.00 0.00 0.00 0.00 20.00 0.70 0.04 2.00 sdg 11.00 0.08 0.00 0.00 0.73 7.27 36.00 0.09 4.00 10.00 0.50 2.44 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.72 0.04 3.20 sdh 5.00 0.03 0.00 0.00 0.60 5.60 46.00 0.10 2.00 4.17 0.59 2.17 0.00 0.00 0.00 0.00 0.00 0.00 23.00 0.83 0.05 2.80 sdi 7.00 0.04 0.00 0.00 0.43 6.29 36.00 0.07 1.00 2.70 0.47 2.11 0.00 0.00 0.00 0.00 0.00 0.00 18.00 0.61 0.03 2.40 sdj 5.00 0.04 0.00 0.00 0.80 7.20 42.00 0.09 1.00 2.33 0.67 2.10 0.00 0.00 0.00 0.00 0.00 0.00 21.00 0.81 0.05 3.20
------------------------------ Other than this 5x node cluster, I also have a 3x node cluster with identical hardware but it serves for a different purpose and data workload. In this cluster I don't have any problem and MDS default settings seems enough. The only difference between two cluster is, 5x node cluster used directly by users, 3x node cluster used heavily to read and write data via projects not by users. So allocate and de-allocate will be better.
I guess I just have a problematic use case on the 5x node cluster and as I mentioned above, I might have the similar problem but I don't know how to debug it.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
------------------------------ Before reading the osd df you should know that I created 2x OSD/per"CT4000MX500SSD1" # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 145.54321 - 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 1.00 - root default -3 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 25 GiB 66 GiB 20 TiB 30.54 0.99 - host ud-01 0 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.04 1.07 61 up osd.0 1 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.28 0.92 53 up osd.1 2 ssd 1.81929 1.00000 1.8 TiB 595 GiB 589 GiB 2.3 GiB 4.0 GiB 1.2 TiB 31.96 1.04 63 up osd.2 3 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.8 GiB 4.2 GiB 1.3 TiB 28.30 0.92 55 up osd.3 4 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.21 0.92 52 up osd.4 5 ssd 1.81929 1.00000 1.8 TiB 592 GiB 586 GiB 1.8 GiB 3.8 GiB 1.2 TiB 31.76 1.03 61 up osd.5 6 ssd 1.81929 1.00000 1.8 TiB 559 GiB 553 GiB 1.8 GiB 4.3 GiB 1.3 TiB 30.03 0.97 57 up osd.6 7 ssd 1.81929 1.00000 1.8 TiB 602 GiB 597 GiB 836 MiB 4.4 GiB 1.2 TiB 32.32 1.05 58 up osd.7 8 ssd 1.81929 1.00000 1.8 TiB 614 GiB 609 GiB 1.2 GiB 4.5 GiB 1.2 TiB 32.98 1.07 60 up osd.8 9 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 2.2 GiB 4.2 GiB 1.3 TiB 30.67 1.00 61 up osd.9 10 ssd 1.81929 1.00000 1.8 TiB 528 GiB 522 GiB 1.3 GiB 4.1 GiB 1.3 TiB 28.33 0.92 52 up osd.10 11 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.5 GiB 3.6 GiB 1.3 TiB 29.57 0.96 56 up osd.11 12 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.8 GiB 4.4 GiB 1.2 TiB 31.91 1.04 61 up osd.12 13 ssd 1.81929 1.00000 1.8 TiB 561 GiB 555 GiB 1.1 GiB 4.3 GiB 1.3 TiB 30.10 0.98 55 up osd.13 14 ssd 1.81929 1.00000 1.8 TiB 616 GiB 609 GiB 1.9 GiB 4.2 GiB 1.2 TiB 33.04 1.07 64 up osd.14 15 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.1 GiB 4.0 GiB 1.3 TiB 28.20 0.92 51 up osd.15 -5 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 22 GiB 67 GiB 20 TiB 30.89 1.00 - host ud-02 16 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.7 GiB 4.7 GiB 1.2 TiB 33.12 1.08 63 up osd.16 17 ssd 1.81929 1.00000 1.8 TiB 582 GiB 577 GiB 1.6 GiB 4.0 GiB 1.3 TiB 31.26 1.01 59 up osd.17 18 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 418 MiB 4.0 GiB 1.3 TiB 31.29 1.02 54 up osd.18 19 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.0 GiB 1.3 TiB 29.50 0.96 56 up osd.19 20 ssd 1.81929 1.00000 1.8 TiB 551 GiB 546 GiB 1.1 GiB 4.1 GiB 1.3 TiB 29.57 0.96 54 up osd.20 21 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.4 GiB 1.2 TiB 33.04 1.07 60 up osd.21 22 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.6 GiB 4.1 GiB 1.3 TiB 30.75 1.00 58 up osd.22 23 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.3 GiB 4.3 GiB 1.2 TiB 33.06 1.07 60 up osd.23 24 ssd 1.81929 1.00000 1.8 TiB 539 GiB 534 GiB 844 MiB 3.8 GiB 1.3 TiB 28.92 0.94 51 up osd.24 25 ssd 1.81929 1.00000 1.8 TiB 583 GiB 576 GiB 2.1 GiB 4.1 GiB 1.3 TiB 31.27 1.02 61 up osd.25 26 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.3 GiB 4.6 GiB 1.2 TiB 33.12 1.08 61 up osd.26 27 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.84 0.94 53 up osd.27 28 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.2 GiB 1.3 TiB 28.29 0.92 53 up osd.28 29 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.5 GiB 4.6 GiB 1.2 TiB 31.91 1.04 59 up osd.29 30 ssd 1.81929 1.00000 1.8 TiB 528 GiB 523 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.35 0.92 53 up osd.30 31 ssd 1.81929 1.00000 1.8 TiB 594 GiB 589 GiB 1.6 GiB 3.8 GiB 1.2 TiB 31.89 1.03 61 up osd.31 -7 29.10864 - 29 TiB 8.9 TiB 8.8 TiB 23 GiB 67 GiB 20 TiB 30.66 1.00 - host ud-03 32 ssd 1.81929 1.00000 1.8 TiB 593 GiB 588 GiB 1.1 GiB 4.3 GiB 1.2 TiB 31.84 1.03 57 up osd.32 33 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.8 GiB 4.4 GiB 1.2 TiB 33.13 1.08 63 up osd.33 34 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 2.0 GiB 3.8 GiB 1.3 TiB 28.84 0.94 59 up osd.34 35 ssd 1.81929 1.00000 1.8 TiB 562 GiB 556 GiB 1.7 GiB 4.2 GiB 1.3 TiB 30.16 0.98 58 up osd.35 36 ssd 1.81929 1.00000 1.8 TiB 529 GiB 523 GiB 1.3 GiB 3.9 GiB 1.3 TiB 28.38 0.92 52 up osd.36 37 ssd 1.81929 1.00000 1.8 TiB 527 GiB 521 GiB 1.7 GiB 4.2 GiB 1.3 TiB 28.28 0.92 55 up osd.37 38 ssd 1.81929 1.00000 1.8 TiB 574 GiB 568 GiB 1.2 GiB 4.3 GiB 1.3 TiB 30.79 1.00 55 up osd.38 39 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.6 GiB 4.2 GiB 1.2 TiB 32.48 1.05 61 up osd.39 40 ssd 1.81929 1.00000 1.8 TiB 573 GiB 567 GiB 1.2 GiB 4.4 GiB 1.3 TiB 30.76 1.00 56 up osd.40 41 ssd 1.81929 1.00000 1.8 TiB 526 GiB 520 GiB 1.7 GiB 3.9 GiB 1.3 TiB 28.21 0.92 54 up osd.41 42 ssd 1.81929 1.00000 1.8 TiB 613 GiB 608 GiB 1010 MiB 4.4 GiB 1.2 TiB 32.91 1.07 58 up osd.42 43 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.7 GiB 4.3 GiB 1.2 TiB 32.51 1.06 61 up osd.43 44 ssd 1.81929 1.00000 1.8 TiB 583 GiB 577 GiB 1.6 GiB 4.2 GiB 1.3 TiB 31.29 1.02 60 up osd.44 45 ssd 1.81929 1.00000 1.8 TiB 618 GiB 613 GiB 1.4 GiB 4.3 GiB 1.2 TiB 33.18 1.08 62 up osd.45 46 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.2 GiB 1.3 TiB 29.50 0.96 54 up osd.46 47 ssd 1.81929 1.00000 1.8 TiB 526 GiB 522 GiB 692 MiB 3.7 GiB 1.3 TiB 28.25 0.92 50 up osd.47 -9 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 26 GiB 68 GiB 20 TiB 31.04 1.01 - host ud-04 48 ssd 1.81929 1.00000 1.8 TiB 540 GiB 534 GiB 2.2 GiB 3.6 GiB 1.3 TiB 28.96 0.94 58 up osd.48 49 ssd 1.81929 1.00000 1.8 TiB 617 GiB 611 GiB 1.4 GiB 4.5 GiB 1.2 TiB 33.11 1.07 61 up osd.49 50 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.2 GiB 4.8 GiB 1.2 TiB 33.17 1.08 61 up osd.50 51 ssd 1.81929 1.00000 1.8 TiB 618 GiB 612 GiB 1.5 GiB 4.5 GiB 1.2 TiB 33.19 1.08 61 up osd.51 52 ssd 1.81929 1.00000 1.8 TiB 526 GiB 521 GiB 1.4 GiB 4.1 GiB 1.3 TiB 28.25 0.92 53 up osd.52 53 ssd 1.81929 1.00000 1.8 TiB 618 GiB 611 GiB 2.4 GiB 4.3 GiB 1.2 TiB 33.17 1.08 66 up osd.53 54 ssd 1.81929 1.00000 1.8 TiB 550 GiB 544 GiB 1.5 GiB 4.3 GiB 1.3 TiB 29.54 0.96 55 up osd.54 55 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.3 GiB 4.0 GiB 1.3 TiB 28.29 0.92 52 up osd.55 56 ssd 1.81929 1.00000 1.8 TiB 525 GiB 519 GiB 1.2 GiB 4.1 GiB 1.3 TiB 28.16 0.91 52 up osd.56 57 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 2.3 GiB 4.2 GiB 1.2 TiB 33.03 1.07 65 up osd.57 58 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.6 GiB 3.7 GiB 1.3 TiB 28.31 0.92 55 up osd.58 59 ssd 1.81929 1.00000 1.8 TiB 615 GiB 609 GiB 1.2 GiB 4.6 GiB 1.2 TiB 33.01 1.07 60 up osd.59 60 ssd 1.81929 1.00000 1.8 TiB 594 GiB 588 GiB 1.2 GiB 4.4 GiB 1.2 TiB 31.88 1.03 59 up osd.60 61 ssd 1.81929 1.00000 1.8 TiB 616 GiB 610 GiB 1.9 GiB 4.1 GiB 1.2 TiB 33.04 1.07 64 up osd.61 62 ssd 1.81929 1.00000 1.8 TiB 620 GiB 614 GiB 1.9 GiB 4.4 GiB 1.2 TiB 33.27 1.08 63 up osd.62 63 ssd 1.81929 1.00000 1.8 TiB 527 GiB 522 GiB 1.5 GiB 4.0 GiB 1.3 TiB 28.30 0.92 53 up osd.63 -11 29.10864 - 29 TiB 9.0 TiB 8.9 TiB 23 GiB 65 GiB 20 TiB 30.91 1.00 - host ud-05 64 ssd 1.81929 1.00000 1.8 TiB 608 GiB 601 GiB 2.3 GiB 4.5 GiB 1.2 TiB 32.62 1.06 65 up osd.64 65 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 628 MiB 4.2 GiB 1.2 TiB 32.53 1.06 57 up osd.65 66 ssd 1.81929 1.00000 1.8 TiB 583 GiB 578 GiB 1.3 GiB 4.3 GiB 1.2 TiB 31.31 1.02 57 up osd.66 67 ssd 1.81929 1.00000 1.8 TiB 537 GiB 533 GiB 436 MiB 3.6 GiB 1.3 TiB 28.82 0.94 50 up osd.67 68 ssd 1.81929 1.00000 1.8 TiB 541 GiB 535 GiB 2.5 GiB 3.8 GiB 1.3 TiB 29.04 0.94 59 up osd.68 69 ssd 1.81929 1.00000 1.8 TiB 606 GiB 601 GiB 1.1 GiB 4.4 GiB 1.2 TiB 32.55 1.06 59 up osd.69 70 ssd 1.81929 1.00000 1.8 TiB 604 GiB 598 GiB 1.8 GiB 4.1 GiB 1.2 TiB 32.44 1.05 63 up osd.70 71 ssd 1.81929 1.00000 1.8 TiB 606 GiB 600 GiB 1.9 GiB 4.5 GiB 1.2 TiB 32.53 1.06 62 up osd.71 72 ssd 1.81929 1.00000 1.8 TiB 602 GiB 598 GiB 612 MiB 4.1 GiB 1.2 TiB 32.33 1.05 57 up osd.72 73 ssd 1.81929 1.00000 1.8 TiB 571 GiB 565 GiB 1.8 GiB 4.5 GiB 1.3 TiB 30.65 0.99 58 up osd.73 74 ssd 1.81929 1.00000 1.8 TiB 608 GiB 602 GiB 1.8 GiB 4.2 GiB 1.2 TiB 32.62 1.06 61 up osd.74 75 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.9 GiB 3.5 GiB 1.3 TiB 28.80 0.93 57 up osd.75 76 ssd 1.81929 1.00000 1.8 TiB 605 GiB 599 GiB 1.4 GiB 4.5 GiB 1.2 TiB 32.48 1.05 60 up osd.76 77 ssd 1.81929 1.00000 1.8 TiB 537 GiB 532 GiB 1.2 GiB 3.9 GiB 1.3 TiB 28.84 0.94 52 up osd.77 78 ssd 1.81929 1.00000 1.8 TiB 525 GiB 520 GiB 1.3 GiB 3.8 GiB 1.3 TiB 28.20 0.92 52 up osd.78 79 ssd 1.81929 1.00000 1.8 TiB 536 GiB 531 GiB 1.1 GiB 3.3 GiB 1.3 TiB 28.76 0.93 53 up osd.79 TOTAL 146 TiB 45 TiB 44 TiB 119 GiB 333 GiB 101 TiB 30.81 MIN/MAX VAR: 0.91/1.08 STDDEV: 1.90
Eugen Block <eblock@nde.ag>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:
There is no definitive answer wrt mds tuning. As it is everywhere mentioned, it's about finding the right setup for your specific workload. If you can synthesize your workload (maybe scale down a bit) try optimizing it in a test cluster without interrupting your developers too much. But what you haven't explained yet is what are you experiencing as a performance issue? Do you have numbers or a detailed description? From the fs status output you didn't seem to have too much activity going on (around 140 requests per second), but that's probably not the usual traffic? What does ceph report in its client IO output? Can you paste the 'ceph osd df' output as well? Do you have dedicated MDS servers or are they colocated with other services?
Zitat von Özkan Göksu <ozkangksu@gmail.com>:
Hello Eugen.
I read all of your MDS related topics and thank you so much for your effort on this. There is not much information and I couldn't find a MDS tuning guide at all. It seems that you are the correct person to discuss mds debugging and tuning.
Do you have any documents or may I learn what is the proper way to debug MDS and clients ? Which debug logs will guide me to understand the limitations and will help to tune according to the data flow?
While searching, I find this:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YO4SGL4DJQ6E...
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic caps recall eventually starts recalling those but the (el7 kernel) client won't release them. Stopping Codium seems to be the only way to release."
Because of this I think I also need to play around with the client side too.
My main goal is increasing the speed and reducing the latency and I wonder if these ideas are correct or not: - Maybe I need to increase client side cache size because via each client, multiple users request a lot of objects and clearly the client_cache_size=16 default is not enough. - Maybe I need to increase client side maximum cache limit for object "client_oc_max_objects=1000 to 10000" and data "client_oc_size=200mi to 400mi" - The client cache cleaning threshold is not aggressive enough to keep the free cache size in the desired range. I need to make it aggressive but this should not reduce speed and increase latency.
mds_cache_memory_limit=4gi to 16gi client_oc_max_objects=1000 to 10000 client_oc_size=200mi to 400mi client_permissions=false #to reduce latency. client_cache_size=16 to 128
What do you think?