Hi friends, I was re-reading documentation[1] when I noticed that 64GiB of RAM should suffice even for a 1000 clients. That really makes our issue that much more difficult to troubleshoot. There are no assumptions that I can make that can encompass all of the details I observe. With no assumptions, there is nothing to test, nothing to start from - feels like a dead end. If anyone has any ideas that we could explore and look into, I'd appreciate it. We made little to no configuration changes, and we believe we followed all the best practices. Cluster is by no means under extreme stress, I would even argue that it is a very dormant one. For the time being, automated cleanup of oudated backups is disabled, and is to be performed manually. [1] https://docs.ceph.com/docs/master/cephfs/add-remove-mds/#provisioning-hardwa... Kind regards, Momo. On Mon, 2020-08-24 at 16:39 +0200, Momčilo Medić wrote:

Hi Eugen,

On Mon, 2020-08-24 at 14:26 +0000, Eugen Block wrote:

...

Hi,

there have been several threads about this topic [1], most likely it's the metadata operation during the cleanup that saturates your disks.

The recommended settings seem to be:

[osd] osd op queue = wpq osd op queue cut off = high

Yeah, I've stumbled upon those settings recently. However, it seems to be the default nowadays...

root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue wpq root@0cephosd01:~# ceph config get mds.cephosd01 osd_op_queue_cut_off high root@cephosd01:~#

I do appreciate your input anyway.

...

This helped us a lot, the number of slow requests has decreased significantly.

Regards, Eugen

[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MK672ROJSW3X...

Zitat von Momčilo Medić <fedorauser@fedoraproject.org>:

...

Hi friends,

Since deployment of our Ceph cluster we've been plagued by slow metadata error. Namely, cluster goes into HEALTH_WARN with a message similar to this one:

2 MDSs report slow metadata IOs 1 MDSs report slow requests 1 slow ops, oldest one blocked for 32 sec, daemons [osd.22,osd.4] have slow ops.

Here is a brief overview of our setup: - 7 OSD nodes with 6 OSD drives each - three of those are also monitors, managers and MDS - there is a single Ceph client (at the moment) - there is only CephFS being used (at the moment) - metadata for CephFS is on HDD (was on HDD, but we moved it as suggested - no improvement)

Our expectation is that this is not a RAM issue as we have 64GiB of memory and is never fully utilized.

It might be a CPU problem, as issue happens mostly during high loads (load of ~12 on a 8-core Intel Xeon Bronze 3106). However, the load is present on all OSD nodes, not just MDS ones.

Cluster is used for (mostly nightly) backups and has no critical performance requirement. Interestingly, significant load across all nodes appears when running cleanup of outdated backups. This boils down to mostly truncating files and some removal, but it is usually small number of large files.

Bellow you can find an example of "dump_ops_in_flight" output during the problem (which you may find useful - I couldn't make sense out of it).

Should we invest into more powerfull CPU hardware (or should we move MDS roles to more powerful nodes)?

Please let me know if I can share any more information to help resolve this thing.

Thanks in advance!

Kind regards, Momo.

===

{ "ops": [ { "description": "client_request(client.22661659:706483006 create #0x10000002742/a-random-file 2020-08- 23T23:09:33.919740+0200 caller_uid=117, caller_gid=121{})", "initiated_at": "2020-08-23T23:09:33.926509+0200", "age": 30.193027896, "duration": 30.193083934000001, "type_data": { "flag_point": "failed to authpin, subtree is being exported", "reqid": "client.22661659:706483006", "op_type": "client_request", "client_info": { "client": "client.22661659", "tid": 706483006 }, "events": [ { "time": "2020-08- 23T23:09:33.926509+0200", "event": "initiated" }, { "time": "2020-08- 23T23:09:33.926510+0200", "event": "throttled" }, { "time": "2020-08- 23T23:09:33.926509+0200", "event": "header_read" }, { "time": "2020-08- 23T23:09:33.926516+0200", "event": "all_read" }, { "time": "2020-08- 23T23:09:33.926540+0200", "event": "dispatched" }, { "time": "2020-08- 23T23:09:33.926595+0200", "event": "failed to authpin, subtree is being exported" } ] } } ], "num_ops": 1 } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Dave, On Tue, 2020-08-25 at 15:25 +0100, david.neal wrote:

Hi Momo,

This can be caused by many things apart from the ceph sw.

For example I saw this once with the MTU in openvswitch not fully matching on a few nodes . We realised this using ping between nodes. For a 9000 MTU:

"linux- ping -M do -s 8972 -c 4 <other host ip>

I've performed a test as you suggested and had no hiccups for any node (long ping log is at the bottom of this email for the curious).

Perhaps starting from the ground up and testing might be the way to go?

This cluster is very new, deployed in December and has been kept up to date since. Like I said previously, we didn't do any config customization - everything is as vanilla as possible.

Kind regards,

Dave

<snip> Kind regards, Momo. Log: root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.33 PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data. 8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.020 ms 8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.047 ms 8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.030 ms 8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.027 ms --- 10.179.40.33 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3062ms rtt min/avg/max/mdev = 0.020/0.031/0.047/0.009 ms root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.34 PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data. 8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.141 ms 8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.094 ms 8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.106 ms 8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.139 ms --- 10.179.40.34 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3078ms rtt min/avg/max/mdev = 0.094/0.120/0.141/0.020 ms root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.35 PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data. 8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.113 ms 8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.169 ms 8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.138 ms 8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.081 ms --- 10.179.40.35 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3062ms rtt min/avg/max/mdev = 0.081/0.125/0.169/0.033 ms root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.36 PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data. 8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.147 ms 8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.163 ms 8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.132 ms 8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.077 ms --- 10.179.40.36 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3054ms rtt min/avg/max/mdev = 0.077/0.129/0.163/0.035 ms root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.37 PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data. 8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.095 ms 8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.153 ms 8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.145 ms 8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.122 ms --- 10.179.40.37 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3070ms rtt min/avg/max/mdev = 0.095/0.128/0.153/0.026 ms root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.38 PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data. 8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.132 ms 8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.156 ms 8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.101 ms 8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.143 ms --- 10.179.40.38 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3070ms rtt min/avg/max/mdev = 0.101/0.133/0.156/0.020 ms root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.39 PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data. 8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.140 ms 8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.094 ms 8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.155 ms 8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.155 ms --- 10.179.40.39 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3054ms rtt min/avg/max/mdev = 0.094/0.136/0.155/0.025 ms root@cephosd01:~# root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.33 PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data. 8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.143 ms 8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.149 ms 8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.137 ms 8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.081 ms --- 10.179.40.33 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3080ms rtt min/avg/max/mdev = 0.081/0.127/0.149/0.029 ms root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.34 PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data. 8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.019 ms 8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.047 ms 8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.048 ms 8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.047 ms --- 10.179.40.34 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3066ms rtt min/avg/max/mdev = 0.019/0.040/0.048/0.013 ms root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.35 PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data. 8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.074 ms 8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.104 ms 8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.113 ms 8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.085 ms --- 10.179.40.35 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3075ms rtt min/avg/max/mdev = 0.074/0.094/0.113/0.015 ms root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.36 PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data. 8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.093 ms 8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.144 ms 8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.109 ms 8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.110 ms --- 10.179.40.36 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3069ms rtt min/avg/max/mdev = 0.093/0.114/0.144/0.018 ms root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.37 PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data. 8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.150 ms 8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.141 ms 8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.120 ms 8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.139 ms --- 10.179.40.37 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3068ms rtt min/avg/max/mdev = 0.120/0.137/0.150/0.016 ms root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.38 PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data. 8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.118 ms 8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.152 ms 8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.100 ms 8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.091 ms --- 10.179.40.38 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3071ms rtt min/avg/max/mdev = 0.091/0.115/0.152/0.024 ms root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.39 PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data. 8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.116 ms 8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.095 ms 8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.143 ms 8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.134 ms --- 10.179.40.39 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3061ms rtt min/avg/max/mdev = 0.095/0.122/0.143/0.018 ms root@cephosd02:~# root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.33 PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data. 8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.174 ms 8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.150 ms 8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.161 ms 8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.082 ms --- 10.179.40.33 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3062ms rtt min/avg/max/mdev = 0.082/0.141/0.174/0.038 ms root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.34 PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data. 8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.144 ms 8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.111 ms 8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.101 ms 8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.076 ms --- 10.179.40.34 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3055ms rtt min/avg/max/mdev = 0.076/0.108/0.144/0.024 ms root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.35 PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data. 8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.020 ms 8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.048 ms 8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.028 ms 8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.029 ms --- 10.179.40.35 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3069ms rtt min/avg/max/mdev = 0.020/0.031/0.048/0.011 ms root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.36 PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data. 8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.140 ms 8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.149 ms 8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.073 ms 8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.138 ms --- 10.179.40.36 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3048ms rtt min/avg/max/mdev = 0.073/0.125/0.149/0.030 ms root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.37 PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data. 8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.109 ms 8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.178 ms 8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.140 ms 8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.118 ms --- 10.179.40.37 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3048ms rtt min/avg/max/mdev = 0.109/0.136/0.178/0.027 ms root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.38 PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data. 8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.128 ms 8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.134 ms 8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.116 ms 8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.070 ms --- 10.179.40.38 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3074ms rtt min/avg/max/mdev = 0.070/0.112/0.134/0.025 ms root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.39 PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data. 8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.066 ms 8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.196 ms 8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.156 ms 8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.127 ms --- 10.179.40.39 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3063ms rtt min/avg/max/mdev = 0.066/0.136/0.196/0.048 ms root@cephosd03:~# root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.33 PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data. 8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.110 ms 8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.146 ms 8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.156 ms 8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.111 ms --- 10.179.40.33 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3073ms rtt min/avg/max/mdev = 0.110/0.130/0.156/0.024 ms root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.34 PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data. 8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.143 ms 8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.118 ms 8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.165 ms 8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.110 ms --- 10.179.40.34 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3060ms rtt min/avg/max/mdev = 0.110/0.134/0.165/0.021 ms root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.35 PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data. 8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.108 ms 8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.091 ms 8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.081 ms 8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.170 ms --- 10.179.40.35 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3055ms rtt min/avg/max/mdev = 0.081/0.112/0.170/0.036 ms root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.36 PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data. 8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.022 ms 8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.031 ms 8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.027 ms 8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.047 ms --- 10.179.40.36 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3077ms rtt min/avg/max/mdev = 0.022/0.031/0.047/0.011 ms root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.37 PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data. 8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.114 ms 8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.122 ms 8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.087 ms 8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.118 ms --- 10.179.40.37 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3076ms rtt min/avg/max/mdev = 0.087/0.110/0.122/0.015 ms root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.38 PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data. 8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.120 ms 8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.175 ms 8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.077 ms 8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.194 ms --- 10.179.40.38 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3076ms rtt min/avg/max/mdev = 0.077/0.141/0.194/0.047 ms root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.39 PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data. 8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.081 ms 8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.132 ms 8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.136 ms 8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.152 ms --- 10.179.40.39 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3065ms rtt min/avg/max/mdev = 0.081/0.125/0.152/0.027 ms root@cephosd04:~# root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.33 PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data. 8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.101 ms 8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.096 ms 8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.109 ms 8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.155 ms --- 10.179.40.33 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3064ms rtt min/avg/max/mdev = 0.096/0.115/0.155/0.024 ms root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.34 PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data. 8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.089 ms 8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.157 ms 8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.166 ms 8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.111 ms --- 10.179.40.34 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3056ms rtt min/avg/max/mdev = 0.089/0.130/0.166/0.034 ms root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.35 PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data. 8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.144 ms 8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.088 ms 8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.123 ms 8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.084 ms --- 10.179.40.35 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3072ms rtt min/avg/max/mdev = 0.084/0.109/0.144/0.028 ms root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.36 PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data. 8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.148 ms 8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.086 ms 8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.152 ms 8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.112 ms --- 10.179.40.36 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3048ms rtt min/avg/max/mdev = 0.086/0.124/0.152/0.029 ms root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.37 PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data. 8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.017 ms 8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.041 ms 8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.042 ms 8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.041 ms --- 10.179.40.37 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3057ms rtt min/avg/max/mdev = 0.017/0.035/0.042/0.011 ms root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.38 PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data. 8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.137 ms 8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.126 ms 8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.148 ms 8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.075 ms --- 10.179.40.38 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3048ms rtt min/avg/max/mdev = 0.075/0.121/0.148/0.030 ms root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.39 PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data. 8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.081 ms 8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.132 ms 8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.130 ms 8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.100 ms --- 10.179.40.39 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3073ms rtt min/avg/max/mdev = 0.081/0.110/0.132/0.024 ms root@cephosd05:~# root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.33 PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data. 8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.065 ms 8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.140 ms 8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.115 ms 8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.152 ms --- 10.179.40.33 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3073ms rtt min/avg/max/mdev = 0.065/0.118/0.152/0.033 ms root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.34 PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data. 8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.152 ms 8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.152 ms 8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.120 ms 8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.110 ms --- 10.179.40.34 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3073ms rtt min/avg/max/mdev = 0.110/0.133/0.152/0.022 ms root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.35 PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data. 8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.173 ms 8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.110 ms 8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.094 ms 8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.147 ms --- 10.179.40.35 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3066ms rtt min/avg/max/mdev = 0.094/0.131/0.173/0.030 ms root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.36 PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data. 8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.142 ms 8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.137 ms 8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.147 ms 8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.127 ms --- 10.179.40.36 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3065ms rtt min/avg/max/mdev = 0.127/0.138/0.147/0.011 ms root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.37 PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data. 8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.109 ms 8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.123 ms 8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.104 ms 8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.125 ms --- 10.179.40.37 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3065ms rtt min/avg/max/mdev = 0.104/0.115/0.125/0.011 ms root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.38 PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data. 8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.013 ms 8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.037 ms 8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.020 ms 8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.037 ms --- 10.179.40.38 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3065ms rtt min/avg/max/mdev = 0.013/0.026/0.037/0.012 ms root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.39 PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data. 8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.124 ms 8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.143 ms 8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.142 ms 8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.092 ms --- 10.179.40.39 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3074ms rtt min/avg/max/mdev = 0.092/0.125/0.143/0.022 ms root@cephosd06:~# root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.33 PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data. 8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.148 ms 8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.138 ms 8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.158 ms 8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.152 ms --- 10.179.40.33 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3056ms rtt min/avg/max/mdev = 0.138/0.149/0.158/0.007 ms root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.34 PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data. 8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.133 ms 8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.128 ms 8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.083 ms 8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.126 ms --- 10.179.40.34 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3048ms rtt min/avg/max/mdev = 0.083/0.117/0.133/0.022 ms root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.35 PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data. 8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.166 ms 8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.089 ms 8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.134 ms 8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.103 ms --- 10.179.40.35 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3056ms rtt min/avg/max/mdev = 0.089/0.123/0.166/0.029 ms root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.36 PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data. 8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.135 ms 8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.125 ms 8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.136 ms 8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.139 ms --- 10.179.40.36 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3056ms rtt min/avg/max/mdev = 0.125/0.133/0.139/0.015 ms root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.37 PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data. 8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.148 ms 8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.115 ms 8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.141 ms 8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.118 ms --- 10.179.40.37 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3056ms rtt min/avg/max/mdev = 0.115/0.130/0.148/0.018 ms root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.38 PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data. 8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.130 ms 8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.144 ms 8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.106 ms 8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.143 ms --- 10.179.40.38 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3048ms rtt min/avg/max/mdev = 0.106/0.130/0.144/0.020 ms root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.39 PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data. 8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.017 ms 8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.046 ms 8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.037 ms 8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.038 ms --- 10.179.40.39 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3064ms rtt min/avg/max/mdev = 0.017/0.034/0.046/0.012 ms root@cephosd07:~#

Hey Eugen, On Wed, 2020-08-26 at 09:29 +0000, Eugen Block wrote:

Hi,

...

...

root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue wpq root@0cephosd01:~# ceph config get mds.cephosd01 osd_op_queue_cut_off high

just to make sure, I referred to OSD not MDS settings, maybe check again?

root@cephosd01:~# ceph config get osd.* osd_op_queue wpq root@cephosd01:~# ceph config get osd.* osd_op_queue_cut_off high root@cephosd01:~# ceph config get mon.* osd_op_queue wpq root@cephosd01:~# ceph config get mon.* osd_op_queue_cut_off high root@cephosd01:~# ceph config get mds.* osd_op_queue wpq root@cephosd01:~# ceph config get mds.* osd_op_queue_cut_off high root@cephosd01:~# It seems no matter which setting I query, it's always the same. Also, documentation for OSD clearly states[1] that it is the default.

I wouldn't focus too much on the MDS service, 64 GB RAM should be enough, but you could and should also check the actual RAM usage, of course. But in our case it's pretty clear that the hard disks are the bottleneck although we have rocksDB on SSD for all OSDs. We seem to have a similar use case (we have nightly compile jobs running in cephfs) just with fewer clients. Our HDDs are saturated especially if we also run deep-scrubs during the night, but the slow requests have been reduced since we changed the osd_op_queue settings for our OSDs.

Have you checked your disk utilization?

Disks are utilized roughly between 70 and 80 percent. Not sure why would operations slow down when disks are getting more utilization. If that would be the case, I'd expect Ceph to issue a warning. Have I understood correctly that the expectation is that if I used larger drives I wouldn't be seeing these warnings? I can understand that adding more disks would create better parallelisation, that's why I'm asking about larger drives. Thank you for discussing this with me, it's highly appreciated. <snip> [1] https://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/#operat... Kind regards, Momo.

On Mon, 2020-08-31 at 14:36 +0000, Eugen Block wrote:

...

Disks are utilized roughly between 70 and 80 percent. Not sure why would operations slow down when disks are getting more utilization. If that would be the case, I'd expect Ceph to issue a warning.

It is warning you, that's why you see slow requests. ;-) But just to be clear, by utilization I mean more than just the filling level of the OSD, have you watched iostat (or something similar) for your disks during usual and high load? Heavy metadata operation on rocksDB increases the load on the main device. I'm not sure if you mentioned it before, do you have stand-alone OSDs or with faster db devices? I believe you only mentioned cephfs_metadata on SSD.

Indeed DB is stored on HDDs and only metadata resides on SSDs. I accidentally stumbled upon someone mentioning disk caching should be disabled to increase performance. I'm now looking into how to configure that on these: - PERC H730P Adapter (has cache on controller) - Dell HBA330 Adp (doesn't have cache on controller) It is not as easy as executing "hdparm" command :( I'll also look into code that does the "truncating" as it may not be Ceph-friendly :/

...

Have I understood correctly that the expectation is that if I used larger drives I wouldn't be seeing these warnings? I can understand that adding more disks would create better parallelisation, that's why I'm asking about larger drives.

I don't think larger drives would improve that, probably even the opposite, depending on the drives, of course. More drives should scale, yes, but there's more to it.

Zitat von Momčilo Medić <fedorauser@fedoraproject.org>:

...

Hey Eugen,

On Wed, 2020-08-26 at 09:29 +0000, Eugen Block wrote:

...

Hi,

...

...

root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue wpq root@0cephosd01:~# ceph config get mds.cephosd01 osd_op_queue_cut_off high

just to make sure, I referred to OSD not MDS settings, maybe check again?

root@cephosd01:~# ceph config get osd.* osd_op_queue wpq root@cephosd01:~# ceph config get osd.* osd_op_queue_cut_off high root@cephosd01:~# ceph config get mon.* osd_op_queue wpq root@cephosd01:~# ceph config get mon.* osd_op_queue_cut_off high root@cephosd01:~# ceph config get mds.* osd_op_queue wpq root@cephosd01:~# ceph config get mds.* osd_op_queue_cut_off high root@cephosd01:~#

It seems no matter which setting I query, it's always the same. Also, documentation for OSD clearly states[1] that it is the default.

...

I wouldn't focus too much on the MDS service, 64 GB RAM should be enough, but you could and should also check the actual RAM usage, of course. But in our case it's pretty clear that the hard disks are the bottleneck although we have rocksDB on SSD for all OSDs. We seem to have a similar use case (we have nightly compile jobs running in cephfs) just with fewer clients. Our HDDs are saturated especially if we also run deep-scrubs during the night, but the slow requests have been reduced since we changed the osd_op_queue settings for our OSDs.

Have you checked your disk utilization?

Disks are utilized roughly between 70 and 80 percent. Not sure why would operations slow down when disks are getting more utilization. If that would be the case, I'd expect Ceph to issue a warning.

Have I understood correctly that the expectation is that if I used larger drives I wouldn't be seeing these warnings? I can understand that adding more disks would create better parallelisation, that's why I'm asking about larger drives.

Thank you for discussing this with me, it's highly appreciated.

<snip>

[1]

https://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/#operat...

...

Kind regards, Momo. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io