Too slow CephFS MDS Restart (Recovery) Performance with Many Sessions and Large Cache Size
31 Mar
2021
31 Mar
'21
8:13 a.m.
Hi Ceph users and developers,
[Apologies if duplicated. I posted the same content to ceph-users two days ago. But it
cannot be seen so far.]
The object of this is to find a solution to MDS slow recovery/rejoin process. Our group at
LINE has provided a shared file system service based on CephFS with K8S and Openstack
users since last year. The performance and functionalities of CephFS are good for us,
whereas MDS availability is not feasible. In reality, restarting active MDS(s) is needed
for some reasons such as version upgrade and daemon crash, and parameter change. In our
experience, it takes from a few minutes to tens of minutes with hundreds of sessions and
two active MDSs in a cluster, where the mds_cache_memory_limit is set to more than 16GB.
So, we have hesitation about restarting MDS and our customers are also not satisfied with
this situation.
To analyze and reproduce the slow MDS recovery process, some experiments have been
conducted using our test environment as described below.
- CentOS 7.9 kernel 3.10.0-1160.11.1.el7.x86_64
- Nautilus 14.2.16
- mds_cache_memory_limit 16GB
- MDS (two active MDSs, two standby-replay MDSs, one single MDS)
- OSD (20 OSDs, each OSD maintains a 100GB virtual disk)
- 500 sessions using kernel driver (among them, only 50 sessions generating workloads are
considered as active clients, while other sessions are just mounted and rarely issue disk
stat requests.)
- VDbench tool is employed to generate metadata-intensive workloads
In this experiment, each session has its own subvolume allocated for testing. VDbench for
each session is configured with set depth=1, width=10, files=24576, filesize=4K and
elapsed = 10800s. Each VDbench instance on a session first creates directories and files.
When dirs/files are created as many as predefined values, it randomly issues getattr,
create, and unlink operations for some hours.
During VDbench running, our restart test procedure written in Python restarts an active
MDS when the average numbers of inodes and caps are greater than 4,915,200 and 1,000,000.
Recovery times from stopping an active MDS to becoming an active MDS were measured as
listed below. the results show that numbers are varied from a few minutes to tens of
minutes when experiments are conducted.
recovery_count, recovery_time(s)
1, 1557
2, 1386
3, 846
4, 1012
5, 1119
6, 1272
A few things I have analyzed
- Rejoining process consumes a considerable amount of time. That's a known issue.
(Sometimes respawning MDS happened. Increasing mds_heartbeat_grace doesn't help.)
- Reducing caps and mds_cache_memory_limit has a somewhat impact on the recovery
performance. As the inodes/dentries caches are reduced, reply latencies are sharply
increased.
- If all clients are using ceph-fuse, the recovery time may be a few times increased
compared to using kernel driver.
- Even though only one active MDS is restarted, other MDS is sometimes also abruptly
restarted.
- Dropping cache (e.g., ceph dameon mds.$(hostname) cache drop) helps reduce the recovery
time, but it takes a few hours under active client workloads and latency spikes are
inevitable.
A few of my questions.
- Why does the MDS recovery take so long despite a graceful restart? (What mainly depends
on the time?)
- Are there some solutions to this problem with many active sessions and large MDS cache
size? (1~2 minutes recovery time will be satisfied, that's our target value.)
- How can we make it deterministic in less than a few minutes?
Thanks
Yongseok
31 Mar
31 Mar
2:23 p.m.
MDS has its own journals, so I guess it is the cause of slow recovery.
And maybe the re-election of authority MDS of particular inode/subtree in
paxos service also contributes to its slowdown.
Have you try setting up a stand-by-reply mds for each active one?
I mean not standy, but stand-by-reply mds which actively replying journal
from active mds.t
That one may help mitigate the no-service time?
On Wed, Mar 31, 2021 at 4:13 PM Yongseok Oh <yongseok.oh(a)linecorp.com>
wrote:
Hi Ceph users and developers,
[Apologies if duplicated. I posted the same content to ceph-users two days
ago. But it cannot be seen so far.]
The object of this is to find a solution to MDS slow recovery/rejoin
process. Our group at LINE has provided a shared file system service based
on CephFS with K8S and Openstack users since last year. The performance and
functionalities of CephFS are good for us, whereas MDS availability is not
feasible. In reality, restarting active MDS(s) is needed for some reasons
such as version upgrade and daemon crash, and parameter change. In our
experience, it takes from a few minutes to tens of minutes with hundreds of
sessions and two active MDSs in a cluster, where the mds_cache_memory_limit
is set to more than 16GB. So, we have hesitation about restarting MDS and
our customers are also not satisfied with this situation.
To analyze and reproduce the slow MDS recovery process, some experiments
have been conducted using our test environment as described below.
- CentOS 7.9 kernel 3.10.0-1160.11.1.el7.x86_64
- Nautilus 14.2.16
- mds_cache_memory_limit 16GB
- MDS (two active MDSs, two standby-replay MDSs, one single MDS)
- OSD (20 OSDs, each OSD maintains a 100GB virtual disk)
- 500 sessions using kernel driver (among them, only 50 sessions
generating workloads are considered as active clients, while other sessions
are just mounted and rarely issue disk stat requests.)
- VDbench tool is employed to generate metadata-intensive workloads
In this experiment, each session has its own subvolume allocated for
testing. VDbench for each session is configured with set depth=1, width=10,
files=24576, filesize=4K and elapsed = 10800s. Each VDbench instance on a
session first creates directories and files. When dirs/files are created as
many as predefined values, it randomly issues getattr, create, and unlink
operations for some hours.
During VDbench running, our restart test procedure written in Python
restarts an active MDS when the average numbers of inodes and caps are
greater than 4,915,200 and 1,000,000. Recovery times from stopping an
active MDS to becoming an active MDS were measured as listed below. the
results show that numbers are varied from a few minutes to tens of minutes
when experiments are conducted.
recovery_count, recovery_time(s)
1, 1557
2, 1386
3, 846
4, 1012
5, 1119
6, 1272
A few things I have analyzed
- Rejoining process consumes a considerable amount of time. That's a known
issue. (Sometimes respawning MDS happened. Increasing mds_heartbeat_grace
doesn't help.)
- Reducing caps and mds_cache_memory_limit has a somewhat impact on the
recovery performance. As the inodes/dentries caches are reduced, reply
latencies are sharply increased.
- If all clients are using ceph-fuse, the recovery time may be a few times
increased compared to using kernel driver.
- Even though only one active MDS is restarted, other MDS is sometimes
also abruptly restarted.
- Dropping cache (e.g., ceph dameon mds.$(hostname) cache drop) helps
reduce the recovery time, but it takes a few hours under active client
workloads and latency spikes are inevitable.
A few of my questions.
- Why does the MDS recovery take so long despite a graceful restart? (What
mainly depends on the time?)
- Are there some solutions to this problem with many active sessions and
large MDS cache size? (1~2 minutes recovery time will be satisfied, that's
our target value.)
- How can we make it deterministic in less than a few minutes?
Thanks
Yongseok
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
1 Apr
1 Apr
1:12 p.m.
Two standby-replay daemons along with active daemons have be deployed in my test cluster.
It can help reduce replay time.
-----Original Message-----
From: "Zizon Qiu"<zzdtsv(a)gmail.com>
To: "Yongseok Oh"<yongseok.oh(a)linecorp.com>orp.com>;
Cc: "dev"<dev(a)ceph.io>;dev@ceph.io>;
Sent: 2021. 3. 31. (수) 23:23 (GMT+09:00)
Subject: Re: Too slow CephFS MDS Restart (Recovery) Performance with Many Sessions and
Large Cache Size
MDS has its own journals, so I guess it is the cause of slow recovery.
And maybe the re-election of authority MDS of particular inode/subtree in paxos service
also contributes to its slowdown.
Have you try setting up a stand-by-reply mds for each active one?
I mean not standy, but stand-by-reply mds which actively replying journal from active
mds.t
That one may help mitigate the no-service time?
On Wed, Mar 31, 2021 at 4:13 PM Yongseok Oh <yongseok.oh(a)linecorp.com> wrote:
Hi Ceph users and developers,
[Apologies if duplicated. I posted the same content to ceph-users two days ago. But it
cannot be seen so far.]
The object of this is to find a solution to MDS slow recovery/rejoin process. Our group at
LINE has provided a shared file system service based on CephFS with K8S and Openstack
users since last year. The performance and functionalities of CephFS are good for us,
whereas MDS availability is not feasible. In reality, restarting active MDS(s) is needed
for some reasons such as version upgrade and daemon crash, and parameter change. In our
experience, it takes from a few minutes to tens of minutes with hundreds of sessions and
two active MDSs in a cluster, where the mds_cache_memory_limit is set to more than 16GB.
So, we have hesitation about restarting MDS and our customers are also not satisfied with
this situation.
To analyze and reproduce the slow MDS recovery process, some experiments have been
conducted using our test environment as described below.
- CentOS 7.9 kernel 3.10.0-1160.11.1.el7.x86_64
- Nautilus 14.2.16
- mds_cache_memory_limit 16GB
- MDS (two active MDSs, two standby-replay MDSs, one single MDS)
- OSD (20 OSDs, each OSD maintains a 100GB virtual disk)
- 500 sessions using kernel driver (among them, only 50 sessions generating workloads are
considered as active clients, while other sessions are just mounted and rarely issue disk
stat requests.)
- VDbench tool is employed to generate metadata-intensive workloads
In this experiment, each session has its own subvolume allocated for testing. VDbench for
each session is configured with set depth=1, width=10, files=24576, filesize=4K and
elapsed = 10800s. Each VDbench instance on a session first creates directories and files.
When dirs/files are created as many as predefined values, it randomly issues getattr,
create, and unlink operations for some hours.
During VDbench running, our restart test procedure written in Python restarts an active
MDS when the average numbers of inodes and caps are greater than 4,915,200 and 1,000,000.
Recovery times from stopping an active MDS to becoming an active MDS were measured as
listed below. the results show that numbers are varied from a few minutes to tens of
minutes when experiments are conducted.
recovery_count, recovery_time(s)
1, 1557
2, 1386
3, 846
4, 1012
5, 1119
6, 1272
A few things I have analyzed
- Rejoining process consumes a considerable amount of time. That's a known issue.
(Sometimes respawning MDS happened. Increasing mds_heartbeat_grace doesn't help.)
- Reducing caps and mds_cache_memory_limit has a somewhat impact on the recovery
performance. As the inodes/dentries caches are reduced, reply latencies are sharply
increased.
- If all clients are using ceph-fuse, the recovery time may be a few times increased
compared to using kernel driver.
- Even though only one active MDS is restarted, other MDS is sometimes also abruptly
restarted.
- Dropping cache (e.g., ceph dameon mds.$(hostname) cache drop) helps reduce the recovery
time, but it takes a few hours under active client workloads and latency spikes are
inevitable.
A few of my questions.
- Why does the MDS recovery take so long despite a graceful restart? (What mainly depends
on the time?)
- Are there some solutions to this problem with many active sessions and large MDS cache
size? (1~2 minutes recovery time will be satisfied, that's our target value.)
- How can we make it deterministic in less than a few minutes?
Thanks
Yongseok
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
31 Mar
31 Mar
4:46 p.m.
Hello Yongseok,
On Wed, Mar 31, 2021 at 1:13 AM Yongseok Oh <yongseok.oh(a)linecorp.com> wrote:
Hi Ceph users and developers,
[Apologies if duplicated. I posted the same content to ceph-users two days ago. But it
cannot be seen so far.]
The object of this is to find a solution to MDS slow recovery/rejoin process. Our group
at LINE has provided a shared file system service based on CephFS with K8S and Openstack
users since last year. The performance and functionalities of CephFS are good for us,
whereas MDS availability is not feasible. In reality, restarting active MDS(s) is needed
for some reasons such as version upgrade and daemon crash, and parameter change. In our
experience, it takes from a few minutes to tens of minutes with hundreds of sessions and
two active MDSs in a cluster, where the mds_cache_memory_limit is set to more than 16GB.
So, we have hesitation about restarting MDS and our customers are also not satisfied with
this situation.
To analyze and reproduce the slow MDS recovery process, some experiments have been
conducted using our test environment as described below.
- CentOS 7.9 kernel 3.10.0-1160.11.1.el7.x86_64
- Nautilus 14.2.16
Please try upgrading to v14.2.17+. The recall/cache config defaults
were updated which may help some.
- mds_cache_memory_limit 16GB
- MDS (two active MDSs, two standby-replay MDSs, one single MDS)
- OSD (20 OSDs, each OSD maintains a 100GB virtual disk)
- 500 sessions using kernel driver (among them, only 50 sessions generating workloads are
considered as active clients, while other sessions are just mounted and rarely issue disk
stat requests.)
- VDbench tool is employed to generate metadata-intensive workloads
In this experiment, each session has its own subvolume allocated for testing. VDbench for
each session is configured with set depth=1, width=10, files=24576, filesize=4K and
elapsed = 10800s. Each VDbench instance on a session first creates directories and files.
When dirs/files are created as many as predefined values, it randomly issues getattr,
create, and unlink operations for some hours.
During VDbench running, our restart test procedure written in Python restarts an active
MDS when the average numbers of inodes and caps are greater than 4,915,200 and 1,000,000.
Recovery times from stopping an active MDS to becoming an active MDS were measured as
listed below. the results show that numbers are varied from a few minutes to tens of
minutes when experiments are conducted.
recovery_count, recovery_time(s)
1, 1557
2, 1386
3, 846
4, 1012
5, 1119
6, 1272
What is the average cap count per session?
A few things I have analyzed
- Rejoining process consumes a considerable amount of time. That's a known issue.
(Sometimes respawning MDS happened. Increasing mds_heartbeat_grace doesn't help.)
Please turn up logging to:
debug_mds = 5
to get an idea what the MDS is doing when respawn occurs.
- Reducing caps and mds_cache_memory_limit has a
somewhat impact on the recovery performance. As the inodes/dentries caches are reduced,
reply latencies are sharply increased.
- If all clients are using ceph-fuse, the recovery time may be a few times increased
compared to using kernel driver.
- Even though only one active MDS is restarted, other MDS is sometimes also abruptly
restarted.
This is also curious, the added debug may help diagnose.
- Dropping cache (e.g., ceph dameon mds.$(hostname)
cache drop) helps reduce the recovery time, but it takes a few hours under active client
workloads and latency spikes are inevitable.
Dropping the MDS cache is limited by the ability to recall state from
clients. It can take a significant amount of time (and may not be able
to reach a satisfactory "zero" size).
A few of my questions.
- Why does the MDS recovery take so long despite a graceful restart? (What mainly depends
on the time?)
It may be related to the open file table; there were several
improvements there recently. Some of those fixes have not (and will
not) been backported to Nautilus/Octopus.
- Are there some solutions to this problem with many
active sessions and large MDS cache size? (1~2 minutes recovery time will be satisfied,
that's our target value.)
This has not been well examined recently with MDS with large caches
(although 16GB is not at all atypical nowadays). So solutions will be
speculative. It would help to know what the MDS is doing that's taking
so long. The debugging will help. Upgrading may help.
--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
1 Apr
1 Apr
12:38 p.m.
On Wed, Mar 31, 2021 at 6:46 PM Patrick Donnelly <pdonnell(a)redhat.com> wrote:
If it helps, here is a log with 2/5 from a recent failover which took
3.5 minutes: https://termbin.com/b022
This is 14.2.11 with the optimized recall/cache tuning.
Indeed rejoin is always the longest step -- even with cephfs_metadata
on SSDs. These MDSs had the cache limit set to 8GB, and you can see
that the rejoining MDS needed 56GB while booting.
I haven't had a chance to test the rejoin/openfiletables optimizations
yet. (https://github.com/ceph/ceph/pull/37383)
But I had understood that this is intended to decrease that rejoin
memory usage -- will it also speed things up?
-- dan
Hello Yongseok,
On Wed, Mar 31, 2021 at 1:13 AM Yongseok Oh <yongseok.oh(a)linecorp.com> wrote:
>
...
A few things I
have analyzed
- Rejoining process consumes a considerable amount of time. That's a known issue.
(Sometimes respawning MDS happened. Increasing mds_heartbeat_grace doesn't help.)
Please turn up logging to:
debug_mds = 5
to get an idea what the MDS is doing when respawn occurs.
2:31 p.m.
On Thu, Apr 1, 2021 at 8:39 PM Dan van der Ster <dan(a)vanderster.com> wrote:
On Wed, Mar 31, 2021 at 6:46 PM Patrick Donnelly
<pdonnell(a)redhat.com>
wrote:
the "Updating MDS map to version" keep showing from init-rejoin at
around
2021-03-18 17:12:52.863
until
2021-03-18 17:22:36.356
while the rejoin itself finished at
2021-03-18 17:15:58.325
So,if it do associated to paxos(with too much changes between then), maybe
pinning some
subtree/directory to particular mds/rank would help too.
Indeed rejoin is always the longest step -- even with cephfs_metadata
Hello Yongseok,
On Wed, Mar 31, 2021 at 1:13 AM Yongseok Oh <yongseok.oh(a)linecorp.com>
wrote:
>
...
> A few things I have analyzed
> - Rejoining process consumes a considerable amount of time. That's a
known issue. (Sometimes respawning MDS happened. Increasing
mds_heartbeat_grace doesn't help.)
Please turn up logging to:
debug_mds = 5
to get an idea what the MDS is doing when respawn occurs.
If it helps, here is a log with 2/5 from a recent failover which took
3.5 minutes: https://termbin.com/b022
This is 14.2.11 with the optimized recall/cache tuning.
on SSDs. These MDSs had the cache limit set to 8GB,
and you can see
that the rejoining MDS needed 56GB while booting.
I haven't had a chance to test the rejoin/openfiletables optimizations
yet. (https://github.com/ceph/ceph/pull/37383)
But I had understood that this is intended to decrease that rejoin
memory usage -- will it also speed things up?
-- dan
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
1:20 p.m.
Hi Patrick,
The average cap counter per session is varied from 50K to 250K, while the total caps in
MDS is ranged from up to 4mill roughtly. I will collect MDS log with debug_mds = 2. But it
will take several days because I need to check our security issue. Once logs are
available, I will let you know.
I will try to upgrade the Ceph cluster version as you suggested. Thanks
Yongseok
-----Original Message-----
From: "Patrick Donnelly"<pdonnell(a)redhat.com>
To: "Yongseok Oh"<yongseok.oh(a)linecorp.com>orp.com>;
Cc: "dev"<dev(a)ceph.io>;dev@ceph.io>;
Sent: 2021. 4. 1. (목) 01:46 (GMT+09:00)
Subject: Re: Too slow CephFS MDS Restart (Recovery) Performance with Many Sessions and
Large Cache Size
Hello Yongseok,
On Wed, Mar 31, 2021 at 1:13 AM Yongseok Oh <yongseok.oh(a)linecorp.com> wrote:
Hi Ceph users and developers,
[Apologies if duplicated. I posted the same content to ceph-users two days ago. But it
cannot be seen so far.]
The object of this is to find a solution to MDS slow recovery/rejoin process. Our group
at LINE has provided a shared file system service based on CephFS with K8S and Openstack
users since last year. The performance and functionalities of CephFS are good for us,
whereas MDS availability is not feasible. In reality, restarting active MDS(s) is needed
for some reasons such as version upgrade and daemon crash, and parameter change. In our
experience, it takes from a few minutes to tens of minutes with hundreds of sessions and
two active MDSs in a cluster, where the mds_cache_memory_limit is set to more than 16GB.
So, we have hesitation about restarting MDS and our customers are also not satisfied with
this situation.
To analyze and reproduce the slow MDS recovery process, some experiments have been
conducted using our test environment as described below.
- CentOS 7.9 kernel 3.10.0-1160.11.1.el7.x86_64
- Nautilus 14.2.16
Please try upgrading to v14.2.17+. The recall/cache config defaults
were updated which may help some.
- mds_cache_memory_limit 16GB
- MDS (two active MDSs, two standby-replay MDSs, one single MDS)
- OSD (20 OSDs, each OSD maintains a 100GB virtual disk)
- 500 sessions using kernel driver (among them, only 50 sessions generating workloads are
considered as active clients, while other sessions are just mounted and rarely issue disk
stat requests.)
- VDbench tool is employed to generate metadata-intensive workloads
In this experiment, each session has its own subvolume allocated for testing. VDbench for
each session is configured with set depth=1, width=10, files=24576, filesize=4K and
elapsed = 10800s. Each VDbench instance on a session first creates directories and files.
When dirs/files are created as many as predefined values, it randomly issues getattr,
create, and unlink operations for some hours.
During VDbench running, our restart test procedure written in Python restarts an active
MDS when the average numbers of inodes and caps are greater than 4,915,200 and 1,000,000.
Recovery times from stopping an active MDS to becoming an active MDS were measured as
listed below. the results show that numbers are varied from a few minutes to tens of
minutes when experiments are conducted.
recovery_count, recovery_time(s)
1, 1557
2, 1386
3, 846
4, 1012
5, 1119
6, 1272
What is the average cap count per session?
A few things I have analyzed
- Rejoining process consumes a considerable amount of time. That's a known issue.
(Sometimes respawning MDS happened. Increasing mds_heartbeat_grace doesn't help.)
Please turn up logging to:
debug_mds = 5
to get an idea what the MDS is doing when respawn occurs.
- Reducing caps and mds_cache_memory_limit has a
somewhat impact on the recovery performance. As the inodes/dentries caches are reduced,
reply latencies are sharply increased.
- If all clients are using ceph-fuse, the recovery time may be a few times increased
compared to using kernel driver.
- Even though only one active MDS is restarted, other MDS is sometimes also abruptly
restarted.
This is also curious, the added debug may help diagnose.
- Dropping cache (e.g., ceph dameon mds.$(hostname)
cache drop) helps reduce the recovery time, but it takes a few hours under active client
workloads and latency spikes are inevitable.
Dropping the MDS cache is limited by the ability to recall state from
clients. It can take a significant amount of time (and may not be able
to reach a satisfactory "zero" size).
A few of my questions.
- Why does the MDS recovery take so long despite a graceful restart? (What mainly depends
on the time?)
It may be related to the open file table; there were several
improvements there recently. Some of those fixes have not (and will
not) been backported to Nautilus/Octopus.
- Are there some solutions to this problem with many
active sessions and large MDS cache size? (1~2 minutes recovery time will be satisfied,
that's our target value.)
This has not been well examined recently with MDS with large caches
(although 16GB is not at all atypical nowadays). So solutions will be
speculative. It would help to know what the MDS is doing that's taking
so long. The debugging will help. Upgrading may help.
--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
6 Apr
6 Apr
1:17 p.m.
Hi Patrick,
I've rerun the MDS restart test. The results and MDS logs are available at
https://yongseokoh.github.io/cephfs/2021/04/06/mds-restart-issue.html. From the results,
the MDS state cannot sometimes be transitioned from "up:rejoin" to
"up:clientreplay" or "up:active" as MDS internal heartbeat is not
health! Let's go over results.
Thanks
Yongseok
-----Original Message-----
From: "오용석[Yongseok Oh]"<yongseok.oh(a)linecorp.com>
To: "Patrick Donnelly"<pdonnell(a)redhat.com>hat.com>;
Cc: "dev"<dev(a)ceph.io>;dev@ceph.io>;
Sent: 2021. 4. 1. (목) 22:20 (GMT+09:00)
Subject: Re: Too slow CephFS MDS Restart (Recovery) Performance with Many Sessions and
Large Cache Size
Hi Patrick,
The average cap counter per session is varied from 50K to 250K, while the total caps in
MDS is ranged from up to 4mill roughtly. I will collect MDS log with debug_mds = 2. But it
will take several days because I need to check our security issue. Once logs are
available, I will let you know.
I will try to upgrade the Ceph cluster version as you suggested. Thanks
Yongseok
-----Original Message-----
From: "Patrick Donnelly"<pdonnell(a)redhat.com>
To: "Yongseok Oh"<yongseok.oh(a)linecorp.com>orp.com>;
Cc: "dev"<dev(a)ceph.io>;dev@ceph.io>;
Sent: 2021. 4. 1. (목) 01:46 (GMT+09:00)
Subject: Re: Too slow CephFS MDS Restart (Recovery) Performance with Many Sessions and
Large Cache Size
Hello Yongseok,
On Wed, Mar 31, 2021 at 1:13 AM Yongseok Oh <yongseok.oh(a)linecorp.com> wrote:
Hi Ceph users and developers,
[Apologies if duplicated. I posted the same content to ceph-users two days ago. But it
cannot be seen so far.]
The object of this is to find a solution to MDS slow recovery/rejoin process. Our group
at LINE has provided a shared file system service based on CephFS with K8S and Openstack
users since last year. The performance and functionalities of CephFS are good for us,
whereas MDS availability is not feasible. In reality, restarting active MDS(s) is needed
for some reasons such as version upgrade and daemon crash, and parameter change. In our
experience, it takes from a few minutes to tens of minutes with hundreds of sessions and
two active MDSs in a cluster, where the mds_cache_memory_limit is set to more than 16GB.
So, we have hesitation about restarting MDS and our customers are also not satisfied with
this situation.
To analyze and reproduce the slow MDS recovery process, some experiments have been
conducted using our test environment as described below.
- CentOS 7.9 kernel 3.10.0-1160.11.1.el7.x86_64
- Nautilus 14.2.16
Please try upgrading to v14.2.17+. The recall/cache config defaults
were updated which may help some.
- mds_cache_memory_limit 16GB
- MDS (two active MDSs, two standby-replay MDSs, one single MDS)
- OSD (20 OSDs, each OSD maintains a 100GB virtual disk)
- 500 sessions using kernel driver (among them, only 50 sessions generating workloads are
considered as active clients, while other sessions are just mounted and rarely issue disk
stat requests.)
- VDbench tool is employed to generate metadata-intensive workloads
In this experiment, each session has its own subvolume allocated for testing. VDbench for
each session is configured with set depth=1, width=10, files=24576, filesize=4K and
elapsed = 10800s. Each VDbench instance on a session first creates directories and files.
When dirs/files are created as many as predefined values, it randomly issues getattr,
create, and unlink operations for some hours.
During VDbench running, our restart test procedure written in Python restarts an active
MDS when the average numbers of inodes and caps are greater than 4,915,200 and 1,000,000.
Recovery times from stopping an active MDS to becoming an active MDS were measured as
listed below. the results show that numbers are varied from a few minutes to tens of
minutes when experiments are conducted.
recovery_count, recovery_time(s)
1, 1557
2, 1386
3, 846
4, 1012
5, 1119
6, 1272
What is the average cap count per session?
A few things I have analyzed
- Rejoining process consumes a considerable amount of time. That's a known issue.
(Sometimes respawning MDS happened. Increasing mds_heartbeat_grace doesn't help.)
Please turn up logging to:
debug_mds = 5
to get an idea what the MDS is doing when respawn occurs.
- Reducing caps and mds_cache_memory_limit has a
somewhat impact on the recovery performance. As the inodes/dentries caches are reduced,
reply latencies are sharply increased.
- If all clients are using ceph-fuse, the recovery time may be a few times increased
compared to using kernel driver.
- Even though only one active MDS is restarted, other MDS is sometimes also abruptly
restarted.
This is also curious, the added debug may help diagnose.
- Dropping cache (e.g., ceph dameon mds.$(hostname)
cache drop) helps reduce the recovery time, but it takes a few hours under active client
workloads and latency spikes are inevitable.
Dropping the MDS cache is limited by the ability to recall state from
clients. It can take a significant amount of time (and may not be able
to reach a satisfactory "zero" size).
A few of my questions.
- Why does the MDS recovery take so long despite a graceful restart? (What mainly depends
on the time?)
It may be related to the open file table; there were several
improvements there recently. Some of those fixes have not (and will
not) been backported to Nautilus/Octopus.
- Are there some solutions to this problem with many
active sessions and large MDS cache size? (1~2 minutes recovery time will be satisfied,
that's our target value.)
This has not been well examined recently with MDS with large caches
(although 16GB is not at all atypical nowadays). So solutions will be
speculative. It would help to know what the MDS is doing that's taking
so long. The debugging will help. Upgrading may help.
--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
1115
days inactive
1121
days old
7 comments
5 participants
participants (5)
-
Dan van der Ster -
Patrick Donnelly -
Yongseok Oh -
Zizon Qiu -
오용석[Yongseok Oh]