ceph slow at 80% full, mds nodes lots of unused memory
Hi we've been running our Ceph cluster for nearly 2 years now (Nautilus) and recently, due to a temporary situation the cluster is at 80% full. We are only using CephFS on the cluster. Normally, I realize we should be adding OSD nodes, but this is a temporary situation, and I expect the cluster to go to <60% full quite soon. Anyway, we are noticing some really problematic slowdowns. There are some things that could be related but we are unsure... - Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM, but are not using more than 2GB, this looks either very inefficient, or wrong ;-) "ceph config dump |grep mds": mds basic mds_cache_memory_limit 107374182400 mds advanced mds_max_scrub_ops_in_progress 10 Perhaps we require more or different settings to properly use the MDS memory? - On all our OSD nodes, the memory line is red in "atop", though no swap is in use, it seems the memory on the OSD nodes is taking quite a beating, is this normal, or can we tweak settings to make it less stressed? This is the first time we are having performance issues like this, I think, I'd like to learn some commands to help me analyse this... I hope this will ring a bell with someone... Cheers /Simon
On 24/02/2021 12:40, Simon Oosthoek wrote:
Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus) and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM, but are not using more than 2GB, this looks either very inefficient, or wrong ;-)
After looking at our monitoring history, it seems the mds cache is actually used more fully, but most of our servers are getting a weekly reboot by default. This clears the mds cache obviously. I wonder if that's a smart idea for an MDS node...? ;-)
"ceph config dump |grep mds": mds basic mds_cache_memory_limit 107374182400 mds advanced mds_max_scrub_ops_in_progress 10
Perhaps we require more or different settings to properly use the MDS memory?
- On all our OSD nodes, the memory line is red in "atop", though no swap is in use, it seems the memory on the OSD nodes is taking quite a beating, is this normal, or can we tweak settings to make it less stressed?
This is the first time we are having performance issues like this, I think, I'd like to learn some commands to help me analyse this...
I hope this will ring a bell with someone...
Cheers
/Simon _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Simon, On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek <s.oosthoek@science.ru.nl> wrote:
On 24/02/2021 12:40, Simon Oosthoek wrote:
Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus) and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM, but are not using more than 2GB, this looks either very inefficient, or wrong ;-)
After looking at our monitoring history, it seems the mds cache is actually used more fully, but most of our servers are getting a weekly reboot by default. This clears the mds cache obviously. I wonder if that's a smart idea for an MDS node...? ;-)
No, it's not. Can you also check that you do not have mds_cache_size configured, perhaps on the MDS local ceph.conf? -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 24/02/2021 22:28, Patrick Donnelly wrote:
Hello Simon,
On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek <s.oosthoek@science.ru.nl> wrote:
On 24/02/2021 12:40, Simon Oosthoek wrote:
Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus) and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM, but are not using more than 2GB, this looks either very inefficient, or wrong ;-)
After looking at our monitoring history, it seems the mds cache is actually used more fully, but most of our servers are getting a weekly reboot by default. This clears the mds cache obviously. I wonder if that's a smart idea for an MDS node...? ;-)
No, it's not. Can you also check that you do not have mds_cache_size configured, perhaps on the MDS local ceph.conf?
Hi Patrick, I've already changed the reboot period to 1 month. The mds_cache_size is not configured locally in the /etc/ceph/ceph.conf file, so I guess it's just the weekly reboot that cleared the memory of cache data... I'm starting to think that a full ceph cluster could probably be the only cause of performance problems. Though I don't know why that would be. Cheers /Simon
Simon Oosthoek wrote:
On 24/02/2021 22:28, Patrick Donnelly wrote:
Hello Simon,
On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek <s.oosthoek(a)science.ru.nl> wrote:
On 24/02/2021 12:40, Simon Oosthoek wrote: Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus) and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM, but are not using more than 2GB, this looks either very inefficient, or wrong ;-) After looking at our monitoring history, it seems the mds cache is actually used more fully, but most of our servers are getting a weekly reboot by default. This clears the mds cache obviously. I wonder if that's a smart idea for an MDS node...? ;-) No, it's not. Can you also check that you do not have mds_cache_size configured, perhaps on the MDS local ceph.conf?
Hi Patrick,
I've already changed the reboot period to 1 month.
The mds_cache_size is not configured locally in the /etc/ceph/ceph.conf file, so I guess it's just the weekly reboot that cleared the memory of cache data...
I'm starting to think that a full ceph cluster could probably be the only cause of performance problems. Though I don't know why that would be.
Did the performance issue only arise when OSDs in the cluster reached 80% usage? What is your osd nearfull_ratio? $ ceph osd dump | grep ratio Is the cluster in HEALTH_WARN with nearfull OSDs? We noticed recently when one of our clusters had nearfull OSDs that cephfs client performance was heavily impacted. Our cluster is nautilus 14.2.15. Clients are kernel 4.19.154. We determined that it was most likely due to the ceph client forcing sync file writes when nearfull flag is present. https://github.com/ceph/ceph-client/commit/7614209736fbc4927584d4387faade4f3... Increasing and decreasing the nearfull ratio confirmed that performance was only impacted while the nearfull flag was present. Not sure if that's relevant for your case.
Cheers
/Simon
On 25/02/2021 11:19, Dylan McCulloch wrote:
Simon Oosthoek wrote:
On 24/02/2021 22:28, Patrick Donnelly wrote:
Hello Simon, On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek <s.oosthoek(a)science.ru.nl> wrote: On 24/02/2021 12:40, Simon Oosthoek wrote: Hi
we've been running our Ceph cluster for nearly 2 years now (Nautilus) and recently, due to a temporary situation the cluster is at 80% full.
We are only using CephFS on the cluster.
Normally, I realize we should be adding OSD nodes, but this is a temporary situation, and I expect the cluster to go to <60% full quite soon.
Anyway, we are noticing some really problematic slowdowns. There are some things that could be related but we are unsure...
- Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM, but are not using more than 2GB, this looks either very inefficient, or wrong ;-) After looking at our monitoring history, it seems the mds cache is actually used more fully, but most of our servers are getting a weekly reboot by default. This clears the mds cache obviously. I wonder if that's a smart idea for an MDS node...? ;-) No, it's not. Can you also check that you do not have mds_cache_size configured, perhaps on the MDS local ceph.conf? Hi Patrick,
I've already changed the reboot period to 1 month.
The mds_cache_size is not configured locally in the /etc/ceph/ceph.conf file, so I guess it's just the weekly reboot that cleared the memory of cache data...
I'm starting to think that a full ceph cluster could probably be the only cause of performance problems. Though I don't know why that would be.
Did the performance issue only arise when OSDs in the cluster reached 80% usage? What is your osd nearfull_ratio? $ ceph osd dump | grep ratio full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85
Is the cluster in HEALTH_WARN with nearfull OSDs?
]# ceph -s cluster: id: b489547c-ba50-4745-a914-23eb78e0e5dc health: HEALTH_WARN 2 pgs not deep-scrubbed in time 957 pgs not scrubbed in time services: mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 7d) mgr: cephmon3(active, since 2M), standbys: cephmon1, cephmon2 mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby osd: 168 osds: 168 up (since 11w), 168 in (since 9M); 43 remapped pgs task status: scrub status: mds.cephmds2: idle data: pools: 10 pools, 5280 pgs objects: 587.71M objects, 804 TiB usage: 1.4 PiB used, 396 TiB / 1.8 PiB avail pgs: 9634168/5101965463 objects misplaced (0.189%) 5232 active+clean 29 active+remapped+backfill_wait 14 active+remapped+backfilling 5 active+clean+scrubbing+deep+repair io: client: 136 MiB/s rd, 600 MiB/s wr, 386 op/s rd, 359 op/s wr recovery: 328 MiB/s, 169 objects/s
We noticed recently when one of our clusters had nearfull OSDs that cephfs client performance was heavily impacted. Our cluster is nautilus 14.2.15. Clients are kernel 4.19.154. We determined that it was most likely due to the ceph client forcing sync file writes when nearfull flag is present. https://github.com/ceph/ceph-client/commit/7614209736fbc4927584d4387faade4f3... Increasing and decreasing the nearfull ratio confirmed that performance was only impacted while the nearfull flag was present. Not sure if that's relevant for your case.
I think this could be very similar in our cluster, thanks for sharing your insights! Cheers /Simon
participants (3)
-
Dylan McCulloch
-
Patrick Donnelly
-
Simon Oosthoek