24 Feb
2021
24 Feb
'21
11:40 a.m.
Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus)
and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a
temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are
some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM,
but are not using more than 2GB, this looks either very inefficient, or
wrong ;-)
"ceph config dump |grep mds":
mds basic mds_cache_memory_limit
107374182400
mds advanced mds_max_scrub_ops_in_progress 10
Perhaps we require more or different settings to properly use the MDS
memory?
- On all our OSD nodes, the memory line is red in "atop", though no swap
is in use, it seems the memory on the OSD nodes is taking quite a
beating, is this normal, or can we tweak settings to make it less stressed?
This is the first time we are having performance issues like this, I
think, I'd like to learn some commands to help me analyse this...
I hope this will ring a bell with someone...
Cheers
/Simon
24 Feb
24 Feb
3:42 p.m.
On 24/02/2021 12:40, Simon Oosthoek wrote:
Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus)
and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a
temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are
some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM,
but are not using more than 2GB, this looks either very inefficient, or
wrong ;-)
After looking at our monitoring history, it seems the mds cache is
actually used more fully, but most of our servers are getting a weekly
reboot by default. This clears the mds cache obviously. I wonder if
that's a smart idea for an MDS node...? ;-)
"ceph config dump |grep mds":
mds basic mds_cache_memory_limit
107374182400
mds advanced mds_max_scrub_ops_in_progress 10
Perhaps we require more or different settings to properly use the MDS
memory?
- On all our OSD nodes, the memory line is red in "atop", though no swap
is in use, it seems the memory on the OSD nodes is taking quite a
beating, is this normal, or can we tweak settings to make it less stressed?
This is the first time we are having performance issues like this, I
think, I'd like to learn some commands to help me analyse this...
I hope this will ring a bell with someone...
Cheers
/Simon
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:28 p.m.
Hello Simon,
On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek <s.oosthoek(a)science.ru.nl> wrote:
On 24/02/2021 12:40, Simon Oosthoek wrote:
No, it's not. Can you also check that you do not have mds_cache_size
configured, perhaps on the MDS local ceph.conf?
--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus)
and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a
temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are
some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM,
but are not using more than 2GB, this looks either very inefficient, or
wrong ;-)
After looking at our monitoring history, it seems the mds cache is
actually used more fully, but most of our servers are getting a weekly
reboot by default. This clears the mds cache obviously. I wonder if
that's a smart idea for an MDS node...? ;-)
25 Feb
25 Feb
10:06 a.m.
On 24/02/2021 22:28, Patrick Donnelly wrote:
Hello Simon,
On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek <s.oosthoek(a)science.ru.nl> wrote:
Hi Patrick,
I've already changed the reboot period to 1 month.
The mds_cache_size is not configured locally in the /etc/ceph/ceph.conf
file, so I guess it's just the weekly reboot that cleared the memory of
cache data...
I'm starting to think that a full ceph cluster could probably be the
only cause of performance problems. Though I don't know why that would be.
Cheers
/Simon
On 24/02/2021 12:40, Simon Oosthoek wrote:
No, it's not. Can you also check that you do not have mds_cache_size
configured, perhaps on the MDS local ceph.conf?
Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus)
and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a
temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are
some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM,
but are not using more than 2GB, this looks either very inefficient, or
wrong ;-)
After looking at our monitoring history, it seems the mds cache is
actually used more fully, but most of our servers are getting a weekly
reboot by default. This clears the mds cache obviously. I wonder if
that's a smart idea for an MDS node...? ;-) 
10:19 a.m.
Simon Oosthoek wrote:
On 24/02/2021 22:28, Patrick Donnelly wrote:
Did the performance issue only arise when OSDs in the cluster reached 80% usage? What is
your osd nearfull_ratio?
$ ceph osd dump | grep ratio
Is the cluster in HEALTH_WARN with nearfull OSDs?
We noticed recently when one of our clusters had nearfull OSDs that cephfs client
performance was heavily impacted.
Our cluster is nautilus 14.2.15. Clients are kernel 4.19.154.
We determined that it was most likely due to the ceph client forcing sync file writes when
nearfull flag is present.
https://github.com/ceph/ceph-client/commit/7614209736fbc4927584d4387faade4f…
Increasing and decreasing the nearfull ratio confirmed that performance was only impacted
while the nearfull flag was present.
Not sure if that's relevant for your case.
Hello Simon,
On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek
<s.oosthoek(a)science.ru.nl> wrote:
On 24/02/2021 12:40, Simon Oosthoek wrote:
Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus)
and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a
temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are
some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM,
but are not using more than 2GB, this looks either very inefficient, or
wrong ;-)
After looking at our monitoring history, it seems the mds cache is
actually used more fully, but most of our servers are getting a weekly
reboot by default. This clears the mds cache obviously. I wonder if
that's a smart idea for an MDS node...? ;-)
No, it's not. Can you also check that you do not have mds_cache_size
configured, perhaps on the MDS local ceph.conf?
Hi Patrick,
I've already changed the reboot period to 1 month.
The mds_cache_size is not configured locally in the /etc/ceph/ceph.conf
file, so I guess it's just the weekly reboot that cleared the memory of
cache data...
I'm starting to think that a full ceph cluster could probably be the
only cause of performance problems. Though I don't know why that would be.
Cheers
/Simon
10:47 a.m.
On 25/02/2021 11:19, Dylan McCulloch wrote:
Simon Oosthoek wrote:
Did the performance issue only arise when OSDs in the cluster reached
80% usage? What is your osd nearfull_ratio?
$ ceph osd dump | grep ratio
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
On 24/02/2021 22:28, Patrick Donnelly wrote:
> Hello Simon,
>
> On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek
<s.oosthoek(a)science.ru.nl> wrote:
>
> On 24/02/2021 12:40, Simon Oosthoek wrote:
> Hi
>
> we've been running our Ceph cluster for nearly 2 years now (Nautilus)
> and recently, due to a temporary situation the cluster is at 80% full.
>
> We are only using CephFS on the cluster.
>
> Normally, I realize we should be adding OSD nodes, but this is a
> temporary situation, and I expect the cluster to go to <60% full
quite
soon.
Anyway, we are noticing some really problematic slowdowns. There are
some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM,
but are not using more than 2GB, this looks either very inefficient, or
wrong ;-)
After looking at our monitoring history, it seems the mds cache is
actually used more fully, but most of our servers are getting a weekly
reboot by default. This clears the mds cache obviously. I wonder if
that's a smart idea for an MDS node...? ;-)
No, it's not. Can you also check that you do not have mds_cache_size
configured, perhaps on the MDS local ceph.conf?
Hi Patrick,
I've already changed the reboot period to 1 month.
The mds_cache_size is not configured locally in the /etc/ceph/ceph.conf
file, so I guess it's just the weekly reboot that cleared the memory of
cache data...
I'm starting to think that a full ceph cluster could probably be the
only cause of performance problems. Though I don't know why that would be. Is the cluster in HEALTH_WARN with nearfull OSDs?
]# ceph -s
cluster:
id: b489547c-ba50-4745-a914-23eb78e0e5dc
health: HEALTH_WARN
2 pgs not deep-scrubbed in time
957 pgs not scrubbed in time
services:
mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 7d)
mgr: cephmon3(active, since 2M), standbys: cephmon1, cephmon2
mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby
osd: 168 osds: 168 up (since 11w), 168 in (since 9M); 43 remapped pgs
task status:
scrub status:
mds.cephmds2: idle
data:
pools: 10 pools, 5280 pgs
objects: 587.71M objects, 804 TiB
usage: 1.4 PiB used, 396 TiB / 1.8 PiB avail
pgs: 9634168/5101965463 objects misplaced (0.189%)
5232 active+clean
29 active+remapped+backfill_wait
14 active+remapped+backfilling
5 active+clean+scrubbing+deep+repair
io:
client: 136 MiB/s rd, 600 MiB/s wr, 386 op/s rd, 359 op/s wr
recovery: 328 MiB/s, 169 objects/s
We noticed recently when one of our clusters had
nearfull OSDs that
cephfs client performance was heavily impacted.
Our cluster is nautilus 14.2.15. Clients are kernel 4.19.154.
We determined that it was most likely due to the ceph client forcing
sync file writes when nearfull flag is present.
https://github.com/ceph/ceph-client/commit/7614209736fbc4927584d4387faade4f…
Increasing and decreasing the nearfull ratio confirmed that performance
was only impacted while the nearfull flag was present.
Not sure if that's relevant for your case.
I think this could be very similar in our cluster, thanks for sharing
your insights!
Cheers
/Simon
1155
days inactive
1156
days old
5 comments
3 participants
participants (3)
-
Dylan McCulloch -
Patrick Donnelly -
Simon Oosthoek