There are some unhandled race conditions in the MDS cluster in rare circumstances. We had this issue with mimic and octopus and it went away after manually pinning sub-dirs to MDS ranks; see https://docs.ceph.com/en/nautilus/cephfs/multimds/?highlight=dir%20pin#manua.... This has the added advantage that one can bypass the internal load-balancer, which was horrible for our work loads. I have a related post about ephemeral pinning on this list one-two years ago. You should be able to find it. Short story: after manually pinning all user directories to ranks, all our problems disappeared and performance improved a lot. MDS load dropped from 130% average to 10-20%. So did memory consumption and cache recycling. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Wednesday, November 22, 2023 12:30 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: mds slow request with “failed to authpin, subtree is being exported" Hi, we've seen this a year ago in a Nautilus cluster with multi-active MDS as well. It turned up only once within several years and we decided not to look too closely at that time. How often do you see it? Is it reproducable? In that case I'd recommend to create a tracker issue. Regards, Eugen Zitat von zxcs <zhuxiongcs@163.com>:

HI, Experts,

we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes mount with ceph-fuse due to the old os system.

and one nodes run a python script with `glob.glob(path)`, and another client doing `cp` operation on the same path.

then we see some log about `mds slow request`, and logs complain “failed to authpin, subtree is being exported"

then need to restart mds,

our question is, does there any dead lock? how can we avoid this and how to fix it without restart mds(it will influence other users) ?

Thanks a ton!

xz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Thanks a ton, Xiubo! it not disappear. even we umount the ceph directory on these two old os node. after dump ops flight , we can see some request, and the earliest complain “failed to authpin, subtree is being exported" And how to avoid this, would you please help to shed some light here? Thanks, xz

2023年11月22日 19:44，Xiubo Li <xiubli@redhat.com> 写道：

On 11/22/23 16:02, zxcs wrote:

...

HI, Experts,

we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes mount with ceph-fuse due to the old os system.

and one nodes run a python script with `glob.glob(path)`, and another client doing `cp` operation on the same path.

then we see some log about `mds slow request`, and logs complain “failed to authpin, subtree is being exported"

then need to restart mds,

our question is, does there any dead lock? how can we avoid this and how to fix it without restart mds(it will influence other users) ?

BTW, won't the slow requests disappear themself later ?

It looks like the exporting is slow or there too many exports are going on.

Thanks

- Xiubo

...

Thanks a ton!

xz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

with the same mds configuration, we see exactly the same(problem, log and solution) with 17.2.5, constantly happening again and again in couples days intervals. MDS servers are stuck somewhere, ceph status reports no issue however. We need to restart some of the mds (if not all of them) to restore them back. Hopefully this could be fixed soon or get docs updated with warning for the balancer's usage in production environment. thanks and regards Xiubo Li <xiubli@redhat.com> 于2023年11月23日周四 15:47写道：

On 11/23/23 11:25, zxcs wrote:

...

Thanks a ton, Xiubo!

it not disappear.

even we umount the ceph directory on these two old os node.

after dump ops flight , we can see some request, and the earliest complain “failed to authpin, subtree is being exported"

And how to avoid this, would you please help to shed some light here?

Okay, as Frank mentioned you can try to disable the balancer by pining the directories. As I remembered the balancer is buggy.

And also you can raise one ceph tracker and provide the debug logs if you have.

Thanks

- Xiubo

...

Thanks, xz

...

2023年11月22日 19:44，Xiubo Li <xiubli@redhat.com> 写道：

On 11/22/23 16:02, zxcs wrote:

...

HI, Experts,

we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes mount with ceph-fuse due to the old os system.

and one nodes run a python script with `glob.glob(path)`, and another client doing `cp` operation on the same path.

then we see some log about `mds slow request`, and logs complain “failed to authpin, subtree is being exported"

then need to restart mds,

our question is, does there any dead lock? how can we avoid this and how to fix it without restart mds(it will influence other users) ? BTW, won't the slow requests disappear themself later ?

It looks like the exporting is slow or there too many exports are going on.

Thanks

- Xiubo

...

Thanks a ton!

xz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On 11/27/23 13:12, zxcs wrote:

current, we using `ceph config set mds mds_bal_interval 3600` to set a fixed time(1 hour).

we also have a question about how to set no balance for multi active mds.

means, we will enable multi active mds(to improve throughput) and no balance for these mds.

and if we set mds_bal_interval as big number seems can void this issue?

You can just set 'mds_bal_interval' to 0.

Thanks, xz

...

2023年11月27日 10:56，Ben <ruidong.gao@gmail.com> 写道：

with the same mds configuration, we see exactly the same(problem, log and solution) with 17.2.5, constantly happening again and again in couples days intervals. MDS servers are stuck somewhere, ceph status reports no issue however. We need to restart some of the mds (if not all of them) to restore them back. Hopefully this could be fixed soon or get docs updated with warning for the balancer's usage in production environment.

thanks and regards

Xiubo Li <xiubli@redhat.com> 于2023年11月23日周四 15:47写道：

...

On 11/23/23 11:25, zxcs wrote:

...

Thanks a ton, Xiubo!

it not disappear.

even we umount the ceph directory on these two old os node.

after dump ops flight , we can see some request, and the earliest complain “failed to authpin, subtree is being exported"

And how to avoid this, would you please help to shed some light here?

Okay, as Frank mentioned you can try to disable the balancer by pining the directories. As I remembered the balancer is buggy.

And also you can raise one ceph tracker and provide the debug logs if you have.

Thanks

- Xiubo

...

Thanks, xz

...

2023年11月22日 19:44，Xiubo Li <xiubli@redhat.com> 写道：

On 11/22/23 16:02, zxcs wrote:

...

HI, Experts,

we are using cephfs with  16.2.* with multi active mds, and recently, we have two nodes mount with ceph-fuse due to the old os system.

and  one nodes run a python script with `glob.glob(path)`, and another client doing `cp` operation on the same path.

then we see some log about `mds slow request`, and logs complain “failed to authpin, subtree is being exported"

then need to restart mds,

our question is, does there any dead lock?  how can we avoid this and how to fix it without restart mds(it will influence other users) ? BTW, won't the slow requests disappear themself later ?

It looks like the exporting is slow or there too many exports are going on.

Thanks

- Xiubo

...

Thanks a ton!

xz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On 12/4/23 16:25, zxcs wrote:

Thanks a lot, Xiubo!

we already set ‘mds_bal_interval’ to 0. and the slow mds seems decrease.

But somehow we still see mds complain slow request. and from mds log , can see

“slow request *** seconds old, received at 2023-12-04T…: internal op exportdir:mds.* currently acquired locks”

so our question is, why it still see "internal op exportdir”, any other config also need to set 0? and could please shed light here which config we need set .

IMO, this should be enough. Venky, Did I miss something here ? Thanks - Xiubo

Thanks, xz

...

2023年11月27日 13:19，Xiubo Li <xiubli@redhat.com> 写道：

On 11/27/23 13:12, zxcs wrote:

...

current, we using `ceph config set mds mds_bal_interval 3600` to set a fixed time(1 hour).

we also have a question about how to set no balance for multi active mds.

means, we will enable multi active mds(to improve throughput) and no balance for these mds.

and if we set mds_bal_interval as big number seems can void this issue?

You can just set 'mds_bal_interval' to 0.

...

Thanks, xz

...

2023年11月27日 10:56，Ben <ruidong.gao@gmail.com> 写道：

with the same mds configuration, we see exactly the same(problem, log and solution) with 17.2.5, constantly happening again and again in couples days intervals. MDS servers are stuck somewhere, ceph status reports no issue however. We need to restart some of the mds (if not all of them) to restore them back. Hopefully this could be fixed soon or get docs updated with warning for the balancer's usage in production environment.

thanks and regards

Xiubo Li <xiubli@redhat.com> 于2023年11月23日周四 15:47写道：

...

...

Thanks a ton, Xiubo!

it not disappear.

even we umount the ceph directory on these two old os node.

after dump ops flight , we can see some request, and the earliest complain “failed to authpin, subtree is being exported" And how to avoid this, would you please help to shed some light here? Okay, as Frank mentioned you can try to disable the balancer by pining

On 11/23/23 11:25, zxcs wrote: the directories. As I remembered the balancer is buggy.

And also you can raise one ceph tracker and provide the debug logs if you have.

Thanks

- Xiubo

...

Thanks, xz

> 2023年11月22日 19:44，Xiubo Li <xiubli@redhat.com> 写道： > > > On 11/22/23 16:02, zxcs wrote: >> HI, Experts, >> >> we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes mount with ceph-fuse due to the old os system. >> and one nodes run a python script with `glob.glob(path)`, and another client doing `cp` operation on the same path. >> then we see some log about `mds slow request`, and logs complain “failed to authpin, subtree is being exported" >> then need to restart mds, >> >> >> our question is, does there any dead lock? how can we avoid this and how to fix it without restart mds(it will influence other users) ? > BTW, won't the slow requests disappear themself later ? > > It looks like the exporting is slow or there too many exports are going on. > Thanks > > - Xiubo > >> Thanks a ton! >> >> >> xz >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io