24 Aug
2020
24 Aug
'20
6:32 p.m.
Hi friends,
Since deployment of our Ceph cluster we've been plagued by slow
metadata error.
Namely, cluster goes into HEALTH_WARN with a message similar to this
one:
2 MDSs report slow metadata IOs
1 MDSs report slow requests
1 slow ops, oldest one blocked for 32 sec, daemons [osd.22,osd.4] have
slow ops.
Here is a brief overview of our setup:
- 7 OSD nodes with 6 OSD drives each
- three of those are also monitors, managers and MDS
- there is a single Ceph client (at the moment)
- there is only CephFS being used (at the moment)
- metadata for CephFS is on HDD (was on HDD, but we moved it as
suggested - no improvement)
Our expectation is that this is not a RAM issue as we have 64GiB of
memory and is never fully utilized.
It might be a CPU problem, as issue happens mostly during high loads
(load of ~12 on a 8-core Intel Xeon Bronze 3106).
However, the load is present on all OSD nodes, not just MDS ones.
Cluster is used for (mostly nightly) backups and has no critical
performance requirement.
Interestingly, significant load across all nodes appears when running
cleanup of outdated backups.
This boils down to mostly truncating files and some removal, but it is
usually small number of large files.
Bellow you can find an example of "dump_ops_in_flight" output during
the problem (which you may find useful - I couldn't make sense out of
it).
Should we invest into more powerfull CPU hardware (or should we move
MDS roles to more powerful nodes)?
Please let me know if I can share any more information to help resolve
this thing.
Thanks in advance!
Kind regards,
Momo.
===
{
"ops": [
{
"description": "client_request(client.22661659:706483006
create #0x10000002742/a-random-file 2020-08-23T23:09:33.919740+0200
caller_uid=117, caller_gid=121{})",
"initiated_at": "2020-08-23T23:09:33.926509+0200",
"age": 30.193027896,
"duration": 30.193083934000001,
"type_data": {
"flag_point": "failed to authpin, subtree is being
exported",
"reqid": "client.22661659:706483006",
"op_type": "client_request",
"client_info": {
"client": "client.22661659",
"tid": 706483006
},
"events": [
{
"time": "2020-08-23T23:09:33.926509+0200",
"event": "initiated"
},
{
"time": "2020-08-23T23:09:33.926510+0200",
"event": "throttled"
},
{
"time": "2020-08-23T23:09:33.926509+0200",
"event": "header_read"
},
{
"time": "2020-08-23T23:09:33.926516+0200",
"event": "all_read"
},
{
"time": "2020-08-23T23:09:33.926540+0200",
"event": "dispatched"
},
{
"time": "2020-08-23T23:09:33.926595+0200",
"event": "failed to authpin, subtree is being
exported"
}
]
}
}
],
"num_ops": 1
}
24 Aug
24 Aug
6:56 p.m.
Hi,
there have been several threads about this topic [1], most likely it's
the metadata operation during the cleanup that saturates your disks.
The recommended settings seem to be:
[osd]
osd op queue = wpq
osd op queue cut off = high
This helped us a lot, the number of slow requests has decreased significantly.
Regards,
Eugen
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MK672ROJSW3…
Zitat von Momčilo Medić <fedorauser(a)fedoraproject.org>rg>:
Hi friends,
Since deployment of our Ceph cluster we've been plagued by slow
metadata error.
Namely, cluster goes into HEALTH_WARN with a message similar to this
one:
2 MDSs report slow metadata IOs
1 MDSs report slow requests
1 slow ops, oldest one blocked for 32 sec, daemons [osd.22,osd.4] have
slow ops.
Here is a brief overview of our setup:
- 7 OSD nodes with 6 OSD drives each
- three of those are also monitors, managers and MDS
- there is a single Ceph client (at the moment)
- there is only CephFS being used (at the moment)
- metadata for CephFS is on HDD (was on HDD, but we moved it as
suggested - no improvement)
Our expectation is that this is not a RAM issue as we have 64GiB of
memory and is never fully utilized.
It might be a CPU problem, as issue happens mostly during high loads
(load of ~12 on a 8-core Intel Xeon Bronze 3106).
However, the load is present on all OSD nodes, not just MDS ones.
Cluster is used for (mostly nightly) backups and has no critical
performance requirement.
Interestingly, significant load across all nodes appears when running
cleanup of outdated backups.
This boils down to mostly truncating files and some removal, but it is
usually small number of large files.
Bellow you can find an example of "dump_ops_in_flight" output during
the problem (which you may find useful - I couldn't make sense out of
it).
Should we invest into more powerfull CPU hardware (or should we move
MDS roles to more powerful nodes)?
Please let me know if I can share any more information to help resolve
this thing.
Thanks in advance!
Kind regards,
Momo.
===
{
"ops": [
{
"description": "client_request(client.22661659:706483006
create #0x10000002742/a-random-file 2020-08-23T23:09:33.919740+0200
caller_uid=117, caller_gid=121{})",
"initiated_at": "2020-08-23T23:09:33.926509+0200",
"age": 30.193027896,
"duration": 30.193083934000001,
"type_data": {
"flag_point": "failed to authpin, subtree is being
exported",
"reqid": "client.22661659:706483006",
"op_type": "client_request",
"client_info": {
"client": "client.22661659",
"tid": 706483006
},
"events": [
{
"time": "2020-08-23T23:09:33.926509+0200",
"event": "initiated"
},
{
"time": "2020-08-23T23:09:33.926510+0200",
"event": "throttled"
},
{
"time": "2020-08-23T23:09:33.926509+0200",
"event": "header_read"
},
{
"time": "2020-08-23T23:09:33.926516+0200",
"event": "all_read"
},
{
"time": "2020-08-23T23:09:33.926540+0200",
"event": "dispatched"
},
{
"time": "2020-08-23T23:09:33.926595+0200",
"event": "failed to authpin, subtree is being
exported"
}
]
}
}
],
"num_ops": 1
}
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:09 p.m.
Hi Eugen,
On Mon, 2020-08-24 at 14:26 +0000, Eugen Block wrote:
Hi,
there have been several threads about this topic [1], most likely
it's
the metadata operation during the cleanup that saturates your disks.
The recommended settings seem to be:
[osd]
osd op queue = wpq
osd op queue cut off = high
Yeah, I've stumbled upon those settings recently.
However, it seems to be the default nowadays...
root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue
wpq
root@0cephosd01:~# ceph config get mds.cephosd01 osd_op_queue_cut_off
high
root@cephosd01:~#
I do appreciate your input anyway.
This helped us a lot, the number of slow requests has
decreased
significantly.
Regards,
Eugen
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MK672ROJSW3…
Zitat von Momčilo Medić <fedorauser(a)fedoraproject.org>rg>:
Hi friends,
Since deployment of our Ceph cluster we've been plagued by slow
metadata error.
Namely, cluster goes into HEALTH_WARN with a message similar to
this
one:
2 MDSs report slow metadata IOs
1 MDSs report slow requests
1 slow ops, oldest one blocked for 32 sec, daemons [osd.22,osd.4]
have
slow ops.
Here is a brief overview of our setup:
- 7 OSD nodes with 6 OSD drives each
- three of those are also monitors, managers and MDS
- there is a single Ceph client (at the moment)
- there is only CephFS being used (at the moment)
- metadata for CephFS is on HDD (was on HDD, but we moved it as
suggested - no improvement)
Our expectation is that this is not a RAM issue as we have 64GiB of
memory and is never fully utilized.
It might be a CPU problem, as issue happens mostly during high
loads
(load of ~12 on a 8-core Intel Xeon Bronze 3106).
However, the load is present on all OSD nodes, not just MDS ones.
Cluster is used for (mostly nightly) backups and has no critical
performance requirement.
Interestingly, significant load across all nodes appears when
running
cleanup of outdated backups.
This boils down to mostly truncating files and some removal, but it
is
usually small number of large files.
Bellow you can find an example of "dump_ops_in_flight" output
during
the problem (which you may find useful - I couldn't make sense out
of
it).
Should we invest into more powerfull CPU hardware (or should we
move
MDS roles to more powerful nodes)?
Please let me know if I can share any more information to help
resolve
this thing.
Thanks in advance!
Kind regards,
Momo.
===
{
"ops": [
{
"description":
"client_request(client.22661659:706483006
create #0x10000002742/a-random-file 2020-08-23T23:09:33.919740+0200
caller_uid=117, caller_gid=121{})",
"initiated_at": "2020-08-23T23:09:33.926509+0200",
"age": 30.193027896,
"duration": 30.193083934000001,
"type_data": {
"flag_point": "failed to authpin, subtree is being
exported",
"reqid": "client.22661659:706483006",
"op_type": "client_request",
"client_info": {
"client": "client.22661659",
"tid": 706483006
},
"events": [
{
"time": "2020-08-23T23:09:33.926509+0200",
"event": "initiated"
},
{
"time": "2020-08-23T23:09:33.926510+0200",
"event": "throttled"
},
{
"time": "2020-08-23T23:09:33.926509+0200",
"event": "header_read"
},
{
"time": "2020-08-23T23:09:33.926516+0200",
"event": "all_read"
},
{
"time": "2020-08-23T23:09:33.926540+0200",
"event": "dispatched"
},
{
"time": "2020-08-23T23:09:33.926595+0200",
"event": "failed to authpin, subtree is
being
exported"
}
]
}
}
],
"num_ops": 1
}
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 Aug
25 Aug
6:06 p.m.
Hi friends,
I was re-reading documentation[1] when I noticed that 64GiB of RAM
should suffice even for a 1000 clients.
That really makes our issue that much more difficult to troubleshoot.
There are no assumptions that I can make that can encompass all of the
details I observe.
With no assumptions, there is nothing to test, nothing to start from -
feels like a dead end.
If anyone has any ideas that we could explore and look into, I'd
appreciate it.
We made little to no configuration changes, and we believe we followed
all the best practices.
Cluster is by no means under extreme stress, I would even argue that it
is a very dormant one.
For the time being, automated cleanup of oudated backups is disabled,
and is to be performed manually.
[1]
https://docs.ceph.com/docs/master/cephfs/add-remove-mds/#provisioning-hardw…
Kind regards,
Momo.
On Mon, 2020-08-24 at 16:39 +0200, Momčilo Medić wrote:
Hi Eugen,
On Mon, 2020-08-24 at 14:26 +0000, Eugen Block wrote:
Hi,
there have been several threads about this topic [1], most likely
it's
the metadata operation during the cleanup that saturates your
disks.
The recommended settings seem to be:
[osd]
osd op queue = wpq
osd op queue cut off = high
Yeah, I've stumbled upon those settings recently.
However, it seems to be the default nowadays...
root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue
wpq
root@0cephosd01:~# ceph config get mds.cephosd01 osd_op_queue_cut_off
high
root@cephosd01:~#
I do appreciate your input anyway.
This helped us a lot, the number of slow requests
has decreased
significantly.
Regards,
Eugen
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MK672ROJSW3…
Zitat von Momčilo Medić <fedorauser(a)fedoraproject.org>rg>:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Hi friends,
Since deployment of our Ceph cluster we've been plagued by slow
metadata error.
Namely, cluster goes into HEALTH_WARN with a message similar to
this
one:
2 MDSs report slow metadata IOs
1 MDSs report slow requests
1 slow ops, oldest one blocked for 32 sec, daemons [osd.22,osd.4]
have
slow ops.
Here is a brief overview of our setup:
- 7 OSD nodes with 6 OSD drives each
- three of those are also monitors, managers and MDS
- there is a single Ceph client (at the moment)
- there is only CephFS being used (at the moment)
- metadata for CephFS is on HDD (was on HDD, but we moved it as
suggested - no improvement)
Our expectation is that this is not a RAM issue as we have 64GiB
of
memory and is never fully utilized.
It might be a CPU problem, as issue happens mostly during high
loads
(load of ~12 on a 8-core Intel Xeon Bronze 3106).
However, the load is present on all OSD nodes, not just MDS ones.
Cluster is used for (mostly nightly) backups and has no critical
performance requirement.
Interestingly, significant load across all nodes appears when
running
cleanup of outdated backups.
This boils down to mostly truncating files and some removal, but
it
is
usually small number of large files.
Bellow you can find an example of "dump_ops_in_flight" output
during
the problem (which you may find useful - I couldn't make sense
out
of
it).
Should we invest into more powerfull CPU hardware (or should we
move
MDS roles to more powerful nodes)?
Please let me know if I can share any more information to help
resolve
this thing.
Thanks in advance!
Kind regards,
Momo.
===
{
"ops": [
{
"description":
"client_request(client.22661659:706483006
create #0x10000002742/a-random-file 2020-08-
23T23:09:33.919740+0200
caller_uid=117, caller_gid=121{})",
"initiated_at": "2020-08-23T23:09:33.926509+0200",
"age": 30.193027896,
"duration": 30.193083934000001,
"type_data": {
"flag_point": "failed to authpin, subtree is
being
exported",
"reqid": "client.22661659:706483006",
"op_type": "client_request",
"client_info": {
"client": "client.22661659",
"tid": 706483006
},
"events": [
{
"time": "2020-08-
23T23:09:33.926509+0200",
"event": "initiated"
},
{
"time": "2020-08-
23T23:09:33.926510+0200",
"event": "throttled"
},
{
"time": "2020-08-
23T23:09:33.926509+0200",
"event": "header_read"
},
{
"time": "2020-08-
23T23:09:33.926516+0200",
"event": "all_read"
},
{
"time": "2020-08-
23T23:09:33.926540+0200",
"event": "dispatched"
},
{
"time": "2020-08-
23T23:09:33.926595+0200",
"event": "failed to authpin, subtree is
being
exported"
}
]
}
}
],
"num_ops": 1
}
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
6:55 p.m.
Hi Momo,
This can be caused by many things apart from the ceph sw.
For example I saw this once with the MTU in openvswitch not fully
matching on a few nodes . We realised this using ping between nodes. For
a 9000 MTU:
"linux- ping -M do -s 8972 -c 4 <other host ip>
Perhaps starting from the ground up and testing might be the way to go?
Kind regards,
Dave
------ Original Message ------
From: "Momčilo Medić" <fedorauser(a)fedoraproject.org>
To: ceph-users(a)ceph.io
Sent: Tuesday, 25 Aug, 2020 At 14:36
Subject: [ceph-users] Re: Persistent problem with slow metadata
Hi friends,
I was re-reading documentation[1] when I noticed that 64GiB of RAM
should suffice even for a 1000 clients.
That really makes our issue that much more difficult to troubleshoot.
There are no assumptions that I can make that can encompass all of the
details I observe.
With no assumptions, there is nothing to test, nothing to start from -
feels like a dead end.
If anyone has any ideas that we could explore and look into, I'd
appreciate it.
We made little to no configuration changes, and we believe we followed
all the best practices.
Cluster is by no means under extreme stress, I would even argue that it
is a very dormant one.
For the time being, automated cleanup of oudated backups is disabled,
and is to be performed manually.
[1]
https://docs.ceph.com/docs/master/cephfs/add-remove-mds/#provisioning-hardw…
<https://docs.ceph.com/docs/master/cephfs/add-remove-mds/#provisioning-hardware-for-an-mds>
Kind regards,
Momo.
On Mon, 2020-08-24 at 16:39 +0200, Momčilo Medić wrote:
Hi Eugen,
On Mon, 2020-08-24 at 14:26 +0000, Eugen Block wrote:
Hi,
there have been several threads about this topic [1], most likely
it's
the metadata operation during the cleanup that saturates your
disks.
The recommended settings seem to be:
[osd]
osd op queue = wpq
osd op queue cut off = high
Yeah, I've stumbled upon those settings recently.
However, it seems to be the default nowadays...
root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue
wpq
root@0cephosd01:~# ceph config get mds.cephosd01 osd_op_queue_cut_off
high
root@cephosd01:~#
I do appreciate your input anyway.
This helped us a lot, the number of slow requests has decreased
significantly.
Regards,
Eugen
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MK672ROJSW3…
<https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MK672ROJSW3X56PC2KWOK2GX7ENQP2LS/#FF3FMP5EEMOBCXAYB4ZVFIAAN6U4IRS3>
Zitat von Momčilo Medić <fedorauser(a)fedoraproject.org>rg>:
Hi friends,
Since deployment of our Ceph cluster we've been plagued by slow
metadata error.
Namely, cluster goes into HEALTH_WARN with a message similar to
this
one:
2 MDSs report slow metadata IOs
1 MDSs report slow requests
1 slow ops, oldest one blocked for 32 sec, daemons [osd.22,osd.4]
have
slow ops.
Here is a brief overview of our setup:
- 7 OSD nodes with 6 OSD drives each
- three of those are also monitors, managers and MDS
- there is a single Ceph client (at the moment)
- there is only CephFS being used (at the moment)
- metadata for CephFS is on HDD (was on HDD, but we moved it as
suggested - no improvement)
Our expectation is that this is not a RAM issue as we have 64GiB
of
memory and is never fully utilized.
It might be a CPU problem, as issue happens mostly during high
loads
(load of ~12 on a 8-core Intel Xeon Bronze 3106).
However, the load is present on all OSD nodes, not just MDS ones.
Cluster is used for (mostly nightly) backups and has no critical
performance requirement.
Interestingly, significant load across all nodes appears when
running
cleanup of outdated backups.
This boils down to mostly truncating files and some removal, but
it
is
usually small number of large files.
Bellow you can find an example of "dump_ops_in_flight" output
during
the problem (which you may find useful - I couldn't make sense
out
of
it).
Should we invest into more powerfull CPU hardware (or should we
move
MDS roles to more powerful nodes)?
Please let me know if I can share any more information to help
resolve
this thing.
Thanks in advance!
Kind regards,
Momo.
===
{
"ops": [
{
"description":
"client_request(client.22661659:706483006
create #0x10000002742/a-random-file 2020-08-
23T23:09:33.919740+0200
caller_uid=117, caller_gid=121{})",
"initiated_at": "2020-08-23T23:09:33.926509+0200",
"age": 30.193027896,
"duration": 30.193083934000001,
"type_data": {
"flag_point": "failed to authpin, subtree is
being
exported",
"reqid": "client.22661659:706483006",
"op_type": "client_request",
"client_info": {
"client": "client.22661659",
"tid": 706483006
},
"events": [
{
"time": "2020-08-
23T23:09:33.926509+0200",
"event": "initiated"
},
{
"time": "2020-08-
23T23:09:33.926510+0200",
"event": "throttled"
},
{
"time": "2020-08-
23T23:09:33.926509+0200",
"event": "header_read"
},
{
"time": "2020-08-
23T23:09:33.926516+0200",
"event": "all_read"
},
{
"time": "2020-08-
23T23:09:33.926540+0200",
"event": "dispatched"
},
{
"time": "2020-08-
23T23:09:33.926595+0200",
"event": "failed to authpin, subtree is
being
exported"
}
]
}
}
],
"num_ops": 1
}
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
31 Aug
31 Aug
2:16 p.m.
Hi Dave,
On Tue, 2020-08-25 at 15:25 +0100, david.neal wrote:
Hi Momo,
This can be caused by many things apart from the ceph sw.
For example I saw this once with the MTU in openvswitch not fully
matching on a few nodes . We realised this using ping between nodes.
For a 9000 MTU:
"linux- ping -M do -s 8972 -c 4 <other host ip>
I've performed a test as you suggested and had no hiccups for any node
(long ping log is at the bottom of this email for the curious).
Perhaps starting from the ground up and testing might
be the way to
go?
This cluster is very new, deployed in December and has been kept up to
date since. Like I said previously, we didn't do any config
customization - everything is as vanilla as possible.
Kind regards,
Dave
<snip>
Kind regards,
Momo.
Log:
root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.33
PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data.
8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.020 ms
8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.047 ms
8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.030 ms
8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.027 ms
--- 10.179.40.33 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3062ms
rtt min/avg/max/mdev = 0.020/0.031/0.047/0.009 ms
root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.34
PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data.
8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.141 ms
8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.094 ms
8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.106 ms
8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.139 ms
--- 10.179.40.34 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3078ms
rtt min/avg/max/mdev = 0.094/0.120/0.141/0.020 ms
root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.35
PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data.
8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.113 ms
8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.169 ms
8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.138 ms
8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.081 ms
--- 10.179.40.35 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3062ms
rtt min/avg/max/mdev = 0.081/0.125/0.169/0.033 ms
root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.36
PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data.
8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.147 ms
8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.163 ms
8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.132 ms
8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.077 ms
--- 10.179.40.36 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3054ms
rtt min/avg/max/mdev = 0.077/0.129/0.163/0.035 ms
root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.37
PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data.
8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.095 ms
8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.153 ms
8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.145 ms
8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.122 ms
--- 10.179.40.37 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3070ms
rtt min/avg/max/mdev = 0.095/0.128/0.153/0.026 ms
root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.38
PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data.
8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.132 ms
8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.156 ms
8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.101 ms
8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.143 ms
--- 10.179.40.38 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3070ms
rtt min/avg/max/mdev = 0.101/0.133/0.156/0.020 ms
root@cephosd01:~# ping -M do -s 8972 -c 4 10.179.40.39
PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data.
8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.140 ms
8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.094 ms
8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.155 ms
8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.155 ms
--- 10.179.40.39 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3054ms
rtt min/avg/max/mdev = 0.094/0.136/0.155/0.025 ms
root@cephosd01:~#
root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.33
PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data.
8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.143 ms
8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.149 ms
8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.137 ms
8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.081 ms
--- 10.179.40.33 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3080ms
rtt min/avg/max/mdev = 0.081/0.127/0.149/0.029 ms
root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.34
PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data.
8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.019 ms
8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.047 ms
8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.048 ms
8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.047 ms
--- 10.179.40.34 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3066ms
rtt min/avg/max/mdev = 0.019/0.040/0.048/0.013 ms
root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.35
PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data.
8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.074 ms
8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.104 ms
8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.113 ms
8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.085 ms
--- 10.179.40.35 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3075ms
rtt min/avg/max/mdev = 0.074/0.094/0.113/0.015 ms
root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.36
PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data.
8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.093 ms
8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.144 ms
8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.109 ms
8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.110 ms
--- 10.179.40.36 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3069ms
rtt min/avg/max/mdev = 0.093/0.114/0.144/0.018 ms
root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.37
PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data.
8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.150 ms
8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.141 ms
8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.120 ms
8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.139 ms
--- 10.179.40.37 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3068ms
rtt min/avg/max/mdev = 0.120/0.137/0.150/0.016 ms
root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.38
PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data.
8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.118 ms
8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.152 ms
8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.100 ms
8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.091 ms
--- 10.179.40.38 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3071ms
rtt min/avg/max/mdev = 0.091/0.115/0.152/0.024 ms
root@cephosd02:~# ping -M do -s 8972 -c 4 10.179.40.39
PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data.
8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.116 ms
8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.095 ms
8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.143 ms
8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.134 ms
--- 10.179.40.39 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3061ms
rtt min/avg/max/mdev = 0.095/0.122/0.143/0.018 ms
root@cephosd02:~#
root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.33
PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data.
8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.174 ms
8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.150 ms
8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.161 ms
8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.082 ms
--- 10.179.40.33 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3062ms
rtt min/avg/max/mdev = 0.082/0.141/0.174/0.038 ms
root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.34
PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data.
8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.144 ms
8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.111 ms
8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.101 ms
8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.076 ms
--- 10.179.40.34 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3055ms
rtt min/avg/max/mdev = 0.076/0.108/0.144/0.024 ms
root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.35
PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data.
8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.020 ms
8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.048 ms
8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.028 ms
8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.029 ms
--- 10.179.40.35 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3069ms
rtt min/avg/max/mdev = 0.020/0.031/0.048/0.011 ms
root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.36
PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data.
8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.140 ms
8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.149 ms
8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.073 ms
8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.138 ms
--- 10.179.40.36 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3048ms
rtt min/avg/max/mdev = 0.073/0.125/0.149/0.030 ms
root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.37
PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data.
8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.109 ms
8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.178 ms
8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.140 ms
8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.118 ms
--- 10.179.40.37 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3048ms
rtt min/avg/max/mdev = 0.109/0.136/0.178/0.027 ms
root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.38
PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data.
8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.128 ms
8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.134 ms
8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.116 ms
8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.070 ms
--- 10.179.40.38 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3074ms
rtt min/avg/max/mdev = 0.070/0.112/0.134/0.025 ms
root@cephosd03:~# ping -M do -s 8972 -c 4 10.179.40.39
PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data.
8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.066 ms
8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.196 ms
8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.156 ms
8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.127 ms
--- 10.179.40.39 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3063ms
rtt min/avg/max/mdev = 0.066/0.136/0.196/0.048 ms
root@cephosd03:~#
root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.33
PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data.
8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.110 ms
8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.146 ms
8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.156 ms
8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.111 ms
--- 10.179.40.33 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3073ms
rtt min/avg/max/mdev = 0.110/0.130/0.156/0.024 ms
root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.34
PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data.
8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.143 ms
8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.118 ms
8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.165 ms
8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.110 ms
--- 10.179.40.34 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3060ms
rtt min/avg/max/mdev = 0.110/0.134/0.165/0.021 ms
root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.35
PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data.
8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.108 ms
8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.091 ms
8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.081 ms
8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.170 ms
--- 10.179.40.35 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3055ms
rtt min/avg/max/mdev = 0.081/0.112/0.170/0.036 ms
root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.36
PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data.
8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.022 ms
8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.031 ms
8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.027 ms
8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.047 ms
--- 10.179.40.36 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3077ms
rtt min/avg/max/mdev = 0.022/0.031/0.047/0.011 ms
root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.37
PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data.
8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.114 ms
8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.122 ms
8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.087 ms
8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.118 ms
--- 10.179.40.37 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3076ms
rtt min/avg/max/mdev = 0.087/0.110/0.122/0.015 ms
root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.38
PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data.
8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.120 ms
8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.175 ms
8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.077 ms
8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.194 ms
--- 10.179.40.38 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3076ms
rtt min/avg/max/mdev = 0.077/0.141/0.194/0.047 ms
root@cephosd04:~# ping -M do -s 8972 -c 4 10.179.40.39
PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data.
8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.081 ms
8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.132 ms
8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.136 ms
8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.152 ms
--- 10.179.40.39 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3065ms
rtt min/avg/max/mdev = 0.081/0.125/0.152/0.027 ms
root@cephosd04:~#
root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.33
PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data.
8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.101 ms
8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.096 ms
8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.109 ms
8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.155 ms
--- 10.179.40.33 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3064ms
rtt min/avg/max/mdev = 0.096/0.115/0.155/0.024 ms
root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.34
PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data.
8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.089 ms
8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.157 ms
8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.166 ms
8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.111 ms
--- 10.179.40.34 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3056ms
rtt min/avg/max/mdev = 0.089/0.130/0.166/0.034 ms
root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.35
PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data.
8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.144 ms
8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.088 ms
8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.123 ms
8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.084 ms
--- 10.179.40.35 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3072ms
rtt min/avg/max/mdev = 0.084/0.109/0.144/0.028 ms
root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.36
PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data.
8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.148 ms
8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.086 ms
8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.152 ms
8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.112 ms
--- 10.179.40.36 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3048ms
rtt min/avg/max/mdev = 0.086/0.124/0.152/0.029 ms
root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.37
PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data.
8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.017 ms
8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.041 ms
8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.042 ms
8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.041 ms
--- 10.179.40.37 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3057ms
rtt min/avg/max/mdev = 0.017/0.035/0.042/0.011 ms
root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.38
PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data.
8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.137 ms
8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.126 ms
8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.148 ms
8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.075 ms
--- 10.179.40.38 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3048ms
rtt min/avg/max/mdev = 0.075/0.121/0.148/0.030 ms
root@cephosd05:~# ping -M do -s 8972 -c 4 10.179.40.39
PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data.
8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.081 ms
8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.132 ms
8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.130 ms
8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.100 ms
--- 10.179.40.39 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3073ms
rtt min/avg/max/mdev = 0.081/0.110/0.132/0.024 ms
root@cephosd05:~#
root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.33
PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data.
8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.065 ms
8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.140 ms
8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.115 ms
8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.152 ms
--- 10.179.40.33 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3073ms
rtt min/avg/max/mdev = 0.065/0.118/0.152/0.033 ms
root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.34
PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data.
8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.152 ms
8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.152 ms
8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.120 ms
8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.110 ms
--- 10.179.40.34 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3073ms
rtt min/avg/max/mdev = 0.110/0.133/0.152/0.022 ms
root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.35
PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data.
8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.173 ms
8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.110 ms
8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.094 ms
8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.147 ms
--- 10.179.40.35 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3066ms
rtt min/avg/max/mdev = 0.094/0.131/0.173/0.030 ms
root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.36
PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data.
8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.142 ms
8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.137 ms
8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.147 ms
8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.127 ms
--- 10.179.40.36 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3065ms
rtt min/avg/max/mdev = 0.127/0.138/0.147/0.011 ms
root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.37
PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data.
8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.109 ms
8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.123 ms
8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.104 ms
8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.125 ms
--- 10.179.40.37 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3065ms
rtt min/avg/max/mdev = 0.104/0.115/0.125/0.011 ms
root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.38
PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data.
8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.013 ms
8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.037 ms
8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.020 ms
8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.037 ms
--- 10.179.40.38 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3065ms
rtt min/avg/max/mdev = 0.013/0.026/0.037/0.012 ms
root@cephosd06:~# ping -M do -s 8972 -c 4 10.179.40.39
PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data.
8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.124 ms
8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.143 ms
8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.142 ms
8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.092 ms
--- 10.179.40.39 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3074ms
rtt min/avg/max/mdev = 0.092/0.125/0.143/0.022 ms
root@cephosd06:~#
root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.33
PING 10.179.40.33 (10.179.40.33) 8972(9000) bytes of data.
8980 bytes from 10.179.40.33: icmp_seq=1 ttl=64 time=0.148 ms
8980 bytes from 10.179.40.33: icmp_seq=2 ttl=64 time=0.138 ms
8980 bytes from 10.179.40.33: icmp_seq=3 ttl=64 time=0.158 ms
8980 bytes from 10.179.40.33: icmp_seq=4 ttl=64 time=0.152 ms
--- 10.179.40.33 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3056ms
rtt min/avg/max/mdev = 0.138/0.149/0.158/0.007 ms
root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.34
PING 10.179.40.34 (10.179.40.34) 8972(9000) bytes of data.
8980 bytes from 10.179.40.34: icmp_seq=1 ttl=64 time=0.133 ms
8980 bytes from 10.179.40.34: icmp_seq=2 ttl=64 time=0.128 ms
8980 bytes from 10.179.40.34: icmp_seq=3 ttl=64 time=0.083 ms
8980 bytes from 10.179.40.34: icmp_seq=4 ttl=64 time=0.126 ms
--- 10.179.40.34 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3048ms
rtt min/avg/max/mdev = 0.083/0.117/0.133/0.022 ms
root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.35
PING 10.179.40.35 (10.179.40.35) 8972(9000) bytes of data.
8980 bytes from 10.179.40.35: icmp_seq=1 ttl=64 time=0.166 ms
8980 bytes from 10.179.40.35: icmp_seq=2 ttl=64 time=0.089 ms
8980 bytes from 10.179.40.35: icmp_seq=3 ttl=64 time=0.134 ms
8980 bytes from 10.179.40.35: icmp_seq=4 ttl=64 time=0.103 ms
--- 10.179.40.35 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3056ms
rtt min/avg/max/mdev = 0.089/0.123/0.166/0.029 ms
root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.36
PING 10.179.40.36 (10.179.40.36) 8972(9000) bytes of data.
8980 bytes from 10.179.40.36: icmp_seq=1 ttl=64 time=0.135 ms
8980 bytes from 10.179.40.36: icmp_seq=2 ttl=64 time=0.125 ms
8980 bytes from 10.179.40.36: icmp_seq=3 ttl=64 time=0.136 ms
8980 bytes from 10.179.40.36: icmp_seq=4 ttl=64 time=0.139 ms
--- 10.179.40.36 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3056ms
rtt min/avg/max/mdev = 0.125/0.133/0.139/0.015 ms
root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.37
PING 10.179.40.37 (10.179.40.37) 8972(9000) bytes of data.
8980 bytes from 10.179.40.37: icmp_seq=1 ttl=64 time=0.148 ms
8980 bytes from 10.179.40.37: icmp_seq=2 ttl=64 time=0.115 ms
8980 bytes from 10.179.40.37: icmp_seq=3 ttl=64 time=0.141 ms
8980 bytes from 10.179.40.37: icmp_seq=4 ttl=64 time=0.118 ms
--- 10.179.40.37 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3056ms
rtt min/avg/max/mdev = 0.115/0.130/0.148/0.018 ms
root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.38
PING 10.179.40.38 (10.179.40.38) 8972(9000) bytes of data.
8980 bytes from 10.179.40.38: icmp_seq=1 ttl=64 time=0.130 ms
8980 bytes from 10.179.40.38: icmp_seq=2 ttl=64 time=0.144 ms
8980 bytes from 10.179.40.38: icmp_seq=3 ttl=64 time=0.106 ms
8980 bytes from 10.179.40.38: icmp_seq=4 ttl=64 time=0.143 ms
--- 10.179.40.38 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3048ms
rtt min/avg/max/mdev = 0.106/0.130/0.144/0.020 ms
root@cephosd07:~# ping -M do -s 8972 -c 4 10.179.40.39
PING 10.179.40.39 (10.179.40.39) 8972(9000) bytes of data.
8980 bytes from 10.179.40.39: icmp_seq=1 ttl=64 time=0.017 ms
8980 bytes from 10.179.40.39: icmp_seq=2 ttl=64 time=0.046 ms
8980 bytes from 10.179.40.39: icmp_seq=3 ttl=64 time=0.037 ms
8980 bytes from 10.179.40.39: icmp_seq=4 ttl=64 time=0.038 ms
--- 10.179.40.39 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3064ms
rtt min/avg/max/mdev = 0.017/0.034/0.046/0.012 ms
root@cephosd07:~#
26 Aug
26 Aug
1:59 p.m.
Hi,
> root@cephosd01:~# ceph config get mds.cephosd01
osd_op_queue
> wpq
> root@0cephosd01:~# ceph config get mds.cephosd01 osd_op_queue_cut_off
> high
just to make sure, I referred to OSD not MDS settings, maybe check again?
I wouldn't focus too much on the MDS service, 64 GB RAM should be
enough, but you could and should also check the actual RAM usage, of
course. But in our case it's pretty clear that the hard disks are the
bottleneck although we have rocksDB on SSD for all OSDs. We seem to
have a similar use case (we have nightly compile jobs running in
cephfs) just with fewer clients. Our HDDs are saturated especially if
we also run deep-scrubs during the night, but the slow requests have
been reduced since we changed the osd_op_queue settings for our OSDs.
Have you checked your disk utilization?
Regards,
Eugen
Zitat von Momčilo Medić <fedorauser(a)fedoraproject.org>rg>:
> Hi friends,
>
> I was re-reading documentation[1] when I noticed that 64GiB of RAM
> should suffice even for a 1000 clients.
> That really makes our issue that much more difficult to troubleshoot.
>
> There are no assumptions that I can make that can encompass all of the
> details I observe.
> With no assumptions, there is nothing to test, nothing to start from -
> feels like a dead end.
>
> If anyone has any ideas that we could explore and look into, I'd
> appreciate it.
>
> We made little to no configuration changes, and we believe we followed
> all the best practices.
> Cluster is by no means under extreme stress, I would even argue that it
> is a very dormant one.
>
> For the time being, automated cleanup of oudated backups is disabled,
> and is to be performed manually.
>
> [1]
>
https://docs.ceph.com/docs/master/cephfs/add-remove-mds/#provisioning-hardw…
>
> Kind regards,
> Momo.
>
> On Mon, 2020-08-24 at 16:39 +0200, Momčilo Medić wrote:
>> Hi Eugen,
>>
>> On Mon, 2020-08-24 at 14:26 +0000, Eugen Block wrote:
>> > Hi,
>> >
>> > there have been several threads about this topic [1], most likely
>> > it's
>> > the metadata operation during the cleanup that saturates your
>> > disks.
>> >
>> > The recommended settings seem to be:
>> >
>> > [osd]
>> > osd op queue = wpq
>> > osd op queue cut off = high
>>
>> Yeah, I've stumbled upon those settings recently.
>> However, it seems to be the default nowadays...
>>
> root@cephosd01:~# ceph config get mds.cephosd01
osd_op_queue
> wpq
> root@0cephosd01:~# ceph config get mds.cephosd01 osd_op_queue_cut_off
> high
>> root@cephosd01:~#
>>
>> I do appreciate your input anyway.
>>
>> > This helped us a lot, the number of slow requests has decreased
>> > significantly.
>> >
>> > Regards,
>> > Eugen
>> >
>> >
>> > [1]
>> >
>>
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MK672ROJSW3…
>> >
>> >
>> > Zitat von Momčilo Medić <fedorauser(a)fedoraproject.org>rg>:
>> >
>> > > Hi friends,
>> > >
>> > > Since deployment of our Ceph cluster we've been plagued by slow
>> > > metadata error.
>> > > Namely, cluster goes into HEALTH_WARN with a message similar to
>> > > this
>> > > one:
>> > >
>> > > 2 MDSs report slow metadata IOs
>> > > 1 MDSs report slow requests
>> > > 1 slow ops, oldest one blocked for 32 sec, daemons [osd.22,osd.4]
>> > > have
>> > > slow ops.
>> > >
>> > > Here is a brief overview of our setup:
>> > > - 7 OSD nodes with 6 OSD drives each
>> > > - three of those are also monitors, managers and MDS
>> > > - there is a single Ceph client (at the moment)
>> > > - there is only CephFS being used (at the moment)
>> > > - metadata for CephFS is on HDD (was on HDD, but we moved it as
>> > > suggested - no improvement)
>> > >
>> > > Our expectation is that this is not a RAM issue as we have 64GiB
>> > > of
>> > > memory and is never fully utilized.
>> > >
>> > > It might be a CPU problem, as issue happens mostly during high
>> > > loads
>> > > (load of ~12 on a 8-core Intel Xeon Bronze 3106).
>> > > However, the load is present on all OSD nodes, not just MDS ones.
>> > >
>> > > Cluster is used for (mostly nightly) backups and has no critical
>> > > performance requirement.
>> > > Interestingly, significant load across all nodes appears when
>> > > running
>> > > cleanup of outdated backups.
>> > > This boils down to mostly truncating files and some removal, but
>> > > it
>> > > is
>> > > usually small number of large files.
>> > >
>> > > Bellow you can find an example of "dump_ops_in_flight"
output
>> > > during
>> > > the problem (which you may find useful - I couldn't make sense
>> > > out
>> > > of
>> > > it).
>> > >
>> > > Should we invest into more powerfull CPU hardware (or should we
>> > > move
>> > > MDS roles to more powerful nodes)?
>> > >
>> > > Please let me know if I can share any more information to help
>> > > resolve
>> > > this thing.
>> > >
>> > > Thanks in advance!
>> > >
>> > > Kind regards,
>> > > Momo.
>> > >
>> > > ===
>> > >
>> > > {
>> > > "ops": [
>> > > {
>> > > "description":
>> > > "client_request(client.22661659:706483006
>> > > create #0x10000002742/a-random-file 2020-08-
>> > > 23T23:09:33.919740+0200
>> > > caller_uid=117, caller_gid=121{})",
>> > > "initiated_at":
"2020-08-23T23:09:33.926509+0200",
>> > > "age": 30.193027896,
>> > > "duration": 30.193083934000001,
>> > > "type_data": {
>> > > "flag_point": "failed to authpin,
subtree is
>> > > being
>> > > exported",
>> > > "reqid":
"client.22661659:706483006",
>> > > "op_type": "client_request",
>> > > "client_info": {
>> > > "client": "client.22661659",
>> > > "tid": 706483006
>> > > },
>> > > "events": [
>> > > {
>> > > "time": "2020-08-
>> > > 23T23:09:33.926509+0200",
>> > > "event": "initiated"
>> > > },
>> > > {
>> > > "time": "2020-08-
>> > > 23T23:09:33.926510+0200",
>> > > "event": "throttled"
>> > > },
>> > > {
>> > > "time": "2020-08-
>> > > 23T23:09:33.926509+0200",
>> > > "event": "header_read"
>> > > },
>> > > {
>> > > "time": "2020-08-
>> > > 23T23:09:33.926516+0200",
>> > > "event": "all_read"
>> > > },
>> > > {
>> > > "time": "2020-08-
>> > > 23T23:09:33.926540+0200",
>> > > "event": "dispatched"
>> > > },
>> > > {
>> > > "time": "2020-08-
>> > > 23T23:09:33.926595+0200",
>> > > "event": "failed to authpin,
subtree is
>> > > being
>> > > exported"
>> > > }
>> > > ]
>> > > }
>> > > }
>> > > ],
>> > > "num_ops": 1
>> > > }
>> > > _______________________________________________
>> > > ceph-users mailing list -- ceph-users(a)ceph.io
>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> >
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users(a)ceph.io
>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
31 Aug
31 Aug
2:19 p.m.
Hey Eugen,
On Wed, 2020-08-26 at 09:29 +0000, Eugen Block wrote:
Hi,
root@cephosd01:~# ceph config get osd.* osd_op_queue
wpq
root@cephosd01:~# ceph config get osd.* osd_op_queue_cut_off
high
root@cephosd01:~# ceph config get mon.* osd_op_queue
wpq
root@cephosd01:~# ceph config get mon.* osd_op_queue_cut_off
high
root@cephosd01:~# ceph config get mds.* osd_op_queue
wpq
root@cephosd01:~# ceph config get mds.* osd_op_queue_cut_off
high
root@cephosd01:~#
It seems no matter which setting I query, it's always the same.
Also, documentation for OSD clearly states[1] that it is the default.
> root@cephosd01:~# ceph config get
mds.cephosd01 osd_op_queue
> wpq
> root@0cephosd01:~# ceph config get mds.cephosd01
> osd_op_queue_cut_off
> high
just to make sure, I referred to OSD not MDS settings, maybe check
again? I wouldn't focus too much on the MDS service, 64
GB RAM should be
enough, but you could and should also check the actual RAM usage,
of
course. But in our case it's pretty clear that the hard disks are
the
bottleneck although we have rocksDB on SSD for all OSDs. We seem
to
have a similar use case (we have nightly compile jobs running in
cephfs) just with fewer clients. Our HDDs are saturated especially
if
we also run deep-scrubs during the night, but the slow requests
have
been reduced since we changed the osd_op_queue settings for our OSDs.
Have you checked your disk utilization?
Disks are utilized roughly between 70 and 80 percent. Not sure why
would operations slow down when disks are getting more utilization.
If that would be the case, I'd expect Ceph to issue a warning.
Have I understood correctly that the expectation is that if I used
larger drives I wouldn't be seeing these warnings?
I can understand that adding more disks would create better
parallelisation, that's why I'm asking about larger drives.
Thank you for discussing this with me, it's highly appreciated.
<snip>
[1]
https://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/#opera…
Kind regards,
Momo.
7:06 p.m.
Disks are utilized roughly between 70 and 80 percent.
Not sure why
would operations slow down when disks are getting more utilization.
If that would be the case, I'd expect Ceph to issue a warning.
It is warning you, that's why you see slow requests. ;-) But just to
be clear, by utilization I mean more than just the filling level of
the OSD, have you watched iostat (or something similar) for your disks
during usual and high load? Heavy metadata operation on rocksDB
increases the load on the main device. I'm not sure if you mentioned
it before, do you have stand-alone OSDs or with faster db devices? I
believe you only mentioned cephfs_metadata on SSD.
Have I understood correctly that the expectation is
that if I used
larger drives I wouldn't be seeing these warnings?
I can understand that adding more disks would create better
parallelisation, that's why I'm asking about larger drives.
I don't think larger drives would improve that, probably even the
opposite, depending on the drives, of course. More drives should
scale, yes, but there's more to it.
Zitat von Momčilo Medić <fedorauser(a)fedoraproject.org>rg>:
> Hey Eugen,
>
> On Wed, 2020-08-26 at 09:29 +0000, Eugen Block wrote:
>> Hi,
>>
>> > > root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue
>> > > wpq
>> > > root@0cephosd01:~# ceph config get mds.cephosd01
>> > > osd_op_queue_cut_off
>> > > high
>>
>> just to make sure, I referred to OSD not MDS settings, maybe check
>> again?
>
> root@cephosd01:~# ceph config get osd.* osd_op_queue
> wpq
> root@cephosd01:~# ceph config get osd.* osd_op_queue_cut_off
> high
> root@cephosd01:~# ceph config get mon.* osd_op_queue
> wpq
> root@cephosd01:~# ceph config get mon.* osd_op_queue_cut_off
> high
> root@cephosd01:~# ceph config get mds.* osd_op_queue
> wpq
> root@cephosd01:~# ceph config get mds.* osd_op_queue_cut_off
> high
> root@cephosd01:~#
>
> It seems no matter which setting I query, it's always the same.
> Also, documentation for OSD clearly states[1] that it is the default.
>
>> I wouldn't focus too much on the MDS service, 64 GB RAM should be
>> enough, but you could and should also check the actual RAM usage,
>> of
>> course. But in our case it's pretty clear that the hard disks are
>> the
>> bottleneck although we have rocksDB on SSD for all OSDs. We seem
>> to
>> have a similar use case (we have nightly compile jobs running in
>> cephfs) just with fewer clients. Our HDDs are saturated especially
>> if
>> we also run deep-scrubs during the night, but the slow requests
>> have
>> been reduced since we changed the osd_op_queue settings for our OSDs.
>>
>> Have you checked your disk utilization?
>
Disks are utilized roughly between 70 and 80 percent.
Not sure why
would operations slow down when disks are getting more utilization.
If that would be the case, I'd expect Ceph to issue a warning.
>
Have I understood correctly that the expectation is
that if I used
larger drives I wouldn't be seeing these warnings?
I can understand that adding more disks would create better
parallelisation, that's why I'm asking about larger drives.
>
> Thank you for discussing this with me, it's highly appreciated.
>
> <snip>
>
> [1]
> https://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/#opera…
>
> Kind regards,
> Momo.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:19 p.m.
On Mon, 2020-08-31 at 14:36 +0000, Eugen Block wrote:
Indeed DB is stored on HDDs and only metadata resides on SSDs.
I accidentally stumbled upon someone mentioning disk caching should be
disabled to increase performance.
I'm now looking into how to configure that on these:
- PERC H730P Adapter (has cache on controller)
- Dell HBA330 Adp (doesn't have cache on controller)
It is not as easy as executing "hdparm" command :(
I'll also look into code that does the "truncating" as it may not be
Ceph-friendly :/
https://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/#opera…
Disks are
utilized roughly between 70 and 80 percent. Not sure why
would operations slow down when disks are getting more utilization.
If that would be the case, I'd expect Ceph to issue a warning.
It is warning you, that's why you see slow requests. ;-) But just
to
be clear, by utilization I mean more than just the filling level of
the OSD, have you watched iostat (or something similar) for your
disks
during usual and high load? Heavy metadata operation on rocksDB
increases the load on the main device. I'm not sure if you
mentioned
it before, do you have stand-alone OSDs or with faster db devices?
I
believe you only mentioned cephfs_metadata on SSD. Have I
understood correctly that the expectation is that if I used
larger drives I wouldn't be seeing these warnings?
I can understand that adding more disks would create better
parallelisation, that's why I'm asking about larger drives.
I don't think larger drives would improve that, probably even the
opposite, depending on the drives, of course. More drives should
scale, yes, but there's more to it.
Zitat von Momčilo Medić <fedorauser(a)fedoraproject.org>rg>:
> Hey Eugen,
>
> On Wed, 2020-08-26 at 09:29 +0000, Eugen Block wrote:
> > Hi,
> >
> > > > root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue
> > > > wpq
> > > > root@0cephosd01:~# ceph config get mds.cephosd01
> > > > osd_op_queue_cut_off
> > > > high
> >
> > just to make sure, I referred to OSD not MDS settings, maybe
> > check
> > again?
>
> root@cephosd01:~# ceph config get osd.* osd_op_queue
> wpq
> root@cephosd01:~# ceph config get osd.* osd_op_queue_cut_off
> high
> root@cephosd01:~# ceph config get mon.* osd_op_queue
> wpq
> root@cephosd01:~# ceph config get mon.* osd_op_queue_cut_off
> high
> root@cephosd01:~# ceph config get mds.* osd_op_queue
> wpq
> root@cephosd01:~# ceph config get mds.* osd_op_queue_cut_off
> high
> root@cephosd01:~#
>
> It seems no matter which setting I query, it's always the same.
> Also, documentation for OSD clearly states[1] that it is the
> default.
>
> > I wouldn't focus too much on the MDS service, 64 GB RAM should be
> > enough, but you could and should also check the actual RAM usage,
> > of
> > course. But in our case it's pretty clear that the hard disks are
> > the
> > bottleneck although we have rocksDB on SSD for all OSDs. We seem
> > to
> > have a similar use case (we have nightly compile jobs running in
> > cephfs) just with fewer clients. Our HDDs are saturated
> > especially
> > if
> > we also run deep-scrubs during the night, but the slow requests
> > have
> > been reduced since we changed the osd_op_queue settings for our
> > OSDs.
> >
> > Have you checked your disk utilization?
>
> Disks are utilized roughly between 70 and 80 percent. Not sure why
> would operations slow down when disks are getting more utilization.
> If that would be the case, I'd expect Ceph to issue a warning.
>
Have I understood correctly that the expectation
is that if I used
larger drives I wouldn't be seeing these warnings?
I can understand that adding more disks would create better
parallelisation, that's why I'm asking about larger drives.
>
> Thank you for discussing this with me, it's highly appreciated.
>
> <snip>
>
> [1]
>
Kind regards,
Momo.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1337
days inactive
1344
days old
9 comments
3 participants
participants (3)
-
david.neal -
Eugen Block -
Momčilo Medić