Hi Kyle,
On Thu, May 6, 2021 at 7:56 AM Kyle Dean <k.s-dean(a)outlook.com> wrote:
Hi, hoping someone could help me get to the bottom of this particular issue I'm
having.
I have ceph octopus installed using ceph-ansible.
Currently, I have 3 MDS servers running, and one client connected to the active MDS.
I'm currently storing a very large encrypted container on the CephFS file system, 8TB
worth, and I'm writing data into it from the client host.
recently I have noticed a severe impact on performance, and the time take to do
processing on file within the container has increased from 1 minute to 11 minutes.
in the ceph dashboard, when I take a look at the performance tab on the file system page,
the Write Ops are increasing exponentially over time.
At the end of April around the 22nd I had 49 write Ops on the performance page for the
MDS deamons. This is now at 266467 Write Ops and increasing.
Also the client requests have gone from 14 to 67 to 117 and is now at 283
would someone be able to help me make sense of why the performance has decreased and what
is going on with the client requests and write operations.
I suggest you look at the "perf dump" statistics from the MDS (via
ceph tell or admin socket) over a period of time to get an idea what
operations it's performing. It's probable your workload changed
somehow and that is the cause.
--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D