22 Nov
2023
22 Nov
'23
11:02 a.m.
HI, Experts,
we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes
mount with ceph-fuse due to the old os system.
and one nodes run a python script with `glob.glob(path)`, and another client doing `cp`
operation on the same path.
then we see some log about `mds slow request`, and logs complain “failed to authpin,
subtree is being exported"
then need to restart mds,
our question is, does there any dead lock? how can we avoid this and how to fix it
without restart mds(it will influence other users) ?
Thanks a ton!
xz
22 Nov
22 Nov
2:30 p.m.
Hi,
we've seen this a year ago in a Nautilus cluster with multi-active MDS
as well. It turned up only once within several years and we decided
not to look too closely at that time. How often do you see it? Is it
reproducable? In that case I'd recommend to create a tracker issue.
Regards,
Eugen
Zitat von zxcs <zhuxiongcs(a)163.com>om>:
HI, Experts,
we are using cephfs with 16.2.* with multi active mds, and
recently, we have two nodes mount with ceph-fuse due to the old os
system.
and one nodes run a python script with `glob.glob(path)`, and
another client doing `cp` operation on the same path.
then we see some log about `mds slow request`, and logs complain
“failed to authpin, subtree is being exported"
then need to restart mds,
our question is, does there any dead lock? how can we avoid this
and how to fix it without restart mds(it will influence other users) ?
Thanks a ton!
xz
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:47 p.m.
There are some unhandled race conditions in the MDS cluster in rare circumstances.
We had this issue with mimic and octopus and it went away after manually pinning sub-dirs
to MDS ranks; see
https://docs.ceph.com/en/nautilus/cephfs/multimds/?highlight=dir%20pin#manu….
This has the added advantage that one can bypass the internal load-balancer, which was
horrible for our work loads. I have a related post about ephemeral pinning on this list
one-two years ago. You should be able to find it. Short story: after manually pinning all
user directories to ranks, all our problems disappeared and performance improved a lot.
MDS load dropped from 130% average to 10-20%. So did memory consumption and cache
recycling.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Eugen Block <eblock(a)nde.ag>
Sent: Wednesday, November 22, 2023 12:30 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: mds slow request with “failed to authpin, subtree is being
exported"
Hi,
we've seen this a year ago in a Nautilus cluster with multi-active MDS
as well. It turned up only once within several years and we decided
not to look too closely at that time. How often do you see it? Is it
reproducable? In that case I'd recommend to create a tracker issue.
Regards,
Eugen
Zitat von zxcs <zhuxiongcs(a)163.com>om>:
HI, Experts,
we are using cephfs with 16.2.* with multi active mds, and
recently, we have two nodes mount with ceph-fuse due to the old os
system.
and one nodes run a python script with `glob.glob(path)`, and
another client doing `cp` operation on the same path.
then we see some log about `mds slow request`, and logs complain
“failed to authpin, subtree is being exported"
then need to restart mds,
our question is, does there any dead lock? how can we avoid this
and how to fix it without restart mds(it will influence other users) ?
Thanks a ton!
xz
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:44 p.m.
On 11/22/23 16:02, zxcs wrote:
HI, Experts,
we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes
mount with ceph-fuse due to the old os system.
and one nodes run a python script with `glob.glob(path)`, and another client doing `cp`
operation on the same path.
then we see some log about `mds slow request`, and logs complain “failed to authpin,
subtree is being exported"
then need to restart mds,
our question is, does there any dead lock? how can we avoid this and how to fix it
without restart mds(it will influence other users) ?
BTW, won't the slow requests disappear themself later ?
It looks like the exporting is slow or there too many exports are going on.
Thanks
- Xiubo
Thanks a ton!
xz
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 Nov
23 Nov
6:25 a.m.
Thanks a ton, Xiubo!
it not disappear.
even we umount the ceph directory on these two old os node.
after dump ops flight , we can see some request, and the earliest complain “failed to
authpin, subtree is being exported"
And how to avoid this, would you please help to shed some light here?
Thanks,
xz
2023年11月22日 19:44，Xiubo Li <xiubli(a)redhat.com>
写道：
On 11/22/23 16:02, zxcs wrote:
HI, Experts,
we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes
mount with ceph-fuse due to the old os system.
and one nodes run a python script with `glob.glob(path)`, and another client doing `cp`
operation on the same path.
then we see some log about `mds slow request`, and logs complain “failed to authpin,
subtree is being exported"
then need to restart mds,
our question is, does there any dead lock? how can we avoid this and how to fix it
without restart mds(it will influence other users) ?
BTW, won't the slow requests disappear themself later ?
It looks like the exporting is slow or there too many exports are going on.
Thanks
- Xiubo
Thanks a ton!
xz
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:45 a.m.
On 11/23/23 11:25, zxcs wrote:
Thanks a ton, Xiubo!
it not disappear.
even we umount the ceph directory on these two old os node.
after dump ops flight , we can see some request, and the earliest complain “failed to
authpin, subtree is being exported"
And how to avoid this, would you please help to shed some light here?
Okay, as Frank mentioned you can try to disable the balancer by pining
the directories. As I remembered the balancer is buggy.
And also you can raise one ceph tracker and provide the debug logs if
you have.
Thanks
- Xiubo
Thanks,
xz
> 2023年11月22日 19:44，Xiubo Li <xiubli(a)redhat.com> 写道：
>
>
> On 11/22/23 16:02, zxcs wrote:
>> HI, Experts,
>>
>> we are using cephfs with 16.2.* with multi active mds, and recently, we have two
nodes mount with ceph-fuse due to the old os system.
>>
>> and one nodes run a python script with `glob.glob(path)`, and another client
doing `cp` operation on the same path.
>>
>> then we see some log about `mds slow request`, and logs complain “failed to
authpin, subtree is being exported"
>>
>> then need to restart mds,
>>
>>
>> our question is, does there any dead lock? how can we avoid this and how to fix
it without restart mds(it will influence other users) ?
> BTW, won't the slow requests disappear themself later ?
>
> It looks like the exporting is slow or there too many exports are going on.
>
> Thanks
>
> - Xiubo
>
>> Thanks a ton!
>>
>>
>> xz
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
27 Nov
27 Nov
5:56 a.m.
with the same mds configuration, we see exactly the same(problem, log and
solution) with 17.2.5, constantly happening again and again in couples days
intervals. MDS servers are stuck somewhere, ceph status reports no issue
however. We need to restart some of the mds (if not all of them) to restore
them back. Hopefully this could be fixed soon or get docs updated with
warning for the balancer's usage in production environment.
thanks and regards
Xiubo Li <xiubli(a)redhat.com> 于2023年11月23日周四 15:47写道：
On 11/23/23 11:25, zxcs wrote:
Thanks a ton, Xiubo!
it not disappear.
even we umount the ceph directory on these two old os node.
after dump ops flight , we can see some request, and the earliest
complain “failed
to authpin, subtree is being exported"
And how to avoid this, would you please help to shed some light here?
Okay, as Frank mentioned you can try to disable the balancer by pining
the directories. As I remembered the balancer is buggy.
And also you can raise one ceph tracker and provide the debug logs if
you have.
Thanks
- Xiubo
Thanks,
xz
> 2023年11月22日 19:44，Xiubo Li <xiubli(a)redhat.com> 写道：
>
>
> On 11/22/23 16:02, zxcs wrote:
>> HI, Experts,
>>
>> we are using cephfs with 16.2.* with multi active mds, and recently,
we
have two nodes mount with ceph-fuse due to the old os system.
>>
>> and one nodes run a python script with `glob.glob(path)`, and another
client doing `cp` operation on the same path.
>>
>> then we see some log about `mds slow request`, and logs complain
“failed
to authpin, subtree is being exported"
>>
>> then need to restart mds,
>>
>>
>> our question is, does there any dead lock? how can we avoid this and
how
to fix it without restart mds(it will influence other users) ?
> BTW, won't the slow requests disappear
themself later ?
>
> It looks like the exporting is slow or there too many exports are going
on.
>
> Thanks
>
> - Xiubo
>
>> Thanks a ton!
>>
>>
>> xz
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:12 a.m.
current, we using `ceph config set mds mds_bal_interval 3600` to set a fixed time(1
hour).
we also have a question about how to set no balance for multi active mds.
means, we will enable multi active mds(to improve throughput) and no balance for these
mds.
and if we set mds_bal_interval as big number seems can void this issue?
Thanks,
xz
2023年11月27日 10:56，Ben <ruidong.gao(a)gmail.com>
写道：
with the same mds configuration, we see exactly the same(problem, log and
solution) with 17.2.5, constantly happening again and again in couples days
intervals. MDS servers are stuck somewhere, ceph status reports no issue
however. We need to restart some of the mds (if not all of them) to restore
them back. Hopefully this could be fixed soon or get docs updated with
warning for the balancer's usage in production environment.
thanks and regards
Xiubo Li <xiubli(a)redhat.com> 于2023年11月23日周四 15:47写道：
On 11/23/23 11:25, zxcs wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Thanks a ton, Xiubo!
it not disappear.
even we umount the ceph directory on these two old os node.
after dump ops flight , we can see some request, and the earliest
complain “failed
to authpin, subtree is being exported"
And how to avoid this, would you please help to shed some light here?
Okay, as Frank mentioned you can try to disable the balancer by pining
the directories. As I remembered the balancer is buggy.
And also you can raise one ceph tracker and provide the debug logs if
you have.
Thanks
- Xiubo
Thanks,
xz
> 2023年11月22日 19:44，Xiubo Li <xiubli(a)redhat.com> 写道：
>
>
> On 11/22/23 16:02, zxcs wrote:
>> HI, Experts,
>>
>> we are using cephfs with 16.2.* with multi active mds, and recently,
we
have two nodes mount with ceph-fuse due to the old os system.
>>
>> and one nodes run a python script with `glob.glob(path)`, and another
client doing `cp` operation on the same path.
>>
>> then we see some log about `mds slow request`, and logs complain
“failed
to authpin, subtree is being exported"
>>
>> then need to restart mds,
>>
>>
>> our question is, does there any dead lock? how can we avoid this and
how
to fix it without restart mds(it will influence other users) ?
> BTW, won't the slow requests disappear
themself later ?
>
> It looks like the exporting is slow or there too many exports are going
on.
>
> Thanks
>
> - Xiubo
>
>> Thanks a ton!
>>
>>
>> xz
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:19 a.m.
On 11/27/23 13:12, zxcs wrote:
current, we using `ceph config set mds
mds_bal_interval 3600` to set a
fixed time(1 hour).
we also have a question about how to set no balance for multi active mds.
means, we will enable multi active mds(to improve throughput) and no
balance for these mds.
and if we set mds_bal_interval as big number seems can void this issue?
You can just set 'mds_bal_interval' to 0.
>
>
> Thanks,
> xz
>
>> 2023年11月27日 10:56，Ben <ruidong.gao(a)gmail.com> 写道：
>>
>> with the same mds configuration, we see exactly the same(problem, log and
>> solution) with 17.2.5, constantly happening again and again in
>> couples days
>> intervals. MDS servers are stuck somewhere, ceph status reports no issue
>> however. We need to restart some of the mds (if not all of them) to
>> restore
>> them back. Hopefully this could be fixed soon or get docs updated with
>> warning for the balancer's usage in production environment.
>>
>> thanks and regards
>>
>> Xiubo Li <xiubli(a)redhat.com> 于2023年11月23日周四 15:47写道：
>>
>>>
>>> On 11/23/23 11:25, zxcs wrote:
>>>> Thanks a ton, Xiubo!
>>>>
>>>> it not disappear.
>>>>
>>>> even we umount the ceph directory on these two old os node.
>>>>
>>>> after dump ops flight , we can see some request, and the earliest
>>> complain “failed to authpin, subtree is being exported"
>>>>
>>>> And how to avoid this, would you please help to shed some light here?
>>>
>>> Okay, as Frank mentioned you can try to disable the balancer by pining
>>> the directories. As I remembered the balancer is buggy.
>>>
>>> And also you can raise one ceph tracker and provide the debug logs if
>>> you have.
>>>
>>> Thanks
>>>
>>> - Xiubo
>>>
>>>
>>>> Thanks,
>>>> xz
>>>>
>>>>
>>>>> 2023年11月22日 19:44，Xiubo Li <xiubli(a)redhat.com> 写道：
>>>>>
>>>>>
>>>>> On 11/22/23 16:02, zxcs wrote:
>>>>>> HI, Experts,
>>>>>>
>>>>>> we are using cephfs with 16.2.* with multi active mds, and
recently,
>>> we have two nodes mount with ceph-fuse due to the old os system.
>>>>>>
>>>>>> and one nodes run a python script with `glob.glob(path)`, and
>>>>>> another
>>> client doing `cp` operation on the same path.
>>>>>>
>>>>>> then we see some log about `mds slow request`, and logs complain
>>> “failed to authpin, subtree is being exported"
>>>>>>
>>>>>> then need to restart mds,
>>>>>>
>>>>>>
>>>>>> our question is, does there any dead lock? how can we avoid this
and
>>> how to fix it without restart mds(it will influence other users) ?
>>>>> BTW, won't the slow requests disappear themself later ?
>>>>>
>>>>> It looks like the exporting is slow or there too many exports are
>>>>> going
>>> on.
>>>>>
>>>>> Thanks
>>>>>
>>>>> - Xiubo
>>>>>
>>>>>> Thanks a ton!
>>>>>>
>>>>>>
>>>>>> xz
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
4 Dec
4 Dec
11:25 a.m.
Thanks a lot, Xiubo!
we already set ‘mds_bal_interval’ to 0. and the slow mds seems decrease.
But somehow we still see mds complain slow request. and from mds log , can see
“slow request *** seconds old, received at 2023-12-04T…: internal op exportdir:mds.*
currently acquired locks”
so our question is, why it still see "internal op exportdir”, any other config also
need to set 0? and could please shed light here which config we need set .
Thanks,
xz
2023年11月27日 13:19，Xiubo Li <xiubli(a)redhat.com>
写道：
On 11/27/23 13:12, zxcs wrote:
current, we using `ceph config set mds
mds_bal_interval 3600` to set a fixed time(1 hour).
we also have a question about how to set no balance for multi active mds.
means, we will enable multi active mds(to improve throughput) and no balance for these
mds.
and if we set mds_bal_interval as big number seems can void this issue?
You can just set 'mds_bal_interval' to 0.
Thanks,
xz
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 2023年11月27日 10:56，Ben
<ruidong.gao(a)gmail.com> 写道：
with the same mds configuration, we see exactly the same(problem, log and
solution) with 17.2.5, constantly happening again and again in couples days
intervals. MDS servers are stuck somewhere, ceph status reports no issue
however. We need to restart some of the mds (if not all of them) to restore
them back. Hopefully this could be fixed soon or get docs updated with
warning for the balancer's usage in production environment.
thanks and regards
Xiubo Li <xiubli(a)redhat.com> 于2023年11月23日周四 15:47写道：
On 11/23/23 11:25, zxcs wrote:
> Thanks a ton, Xiubo!
>
> it not disappear.
>
> even we umount the ceph directory on these two old os node.
>
> after dump ops flight , we can see some request, and the earliest
complain “failed to authpin, subtree is being exported"
>
> And how to avoid this, would you please help to shed some light here?
Okay, as Frank mentioned you can try to disable the balancer by pining
the directories. As I remembered the balancer is buggy.
And also you can raise one ceph tracker and provide the debug logs if
you have.
Thanks
- Xiubo
> Thanks,
> xz
>
>
>> 2023年11月22日 19:44，Xiubo Li <xiubli(a)redhat.com> 写道：
>>
>>
>> On 11/22/23 16:02, zxcs wrote:
>>> HI, Experts,
>>>
>>> we are using cephfs with 16.2.* with multi active mds, and recently,
we have two nodes mount with ceph-fuse due to the old os system.
>>>
>>> and one nodes run a python script with `glob.glob(path)`, and another
client doing `cp` operation on the same path.
>>>
>>> then we see some log about `mds slow request`, and logs complain
“failed to authpin, subtree is being exported"
>>>
>>> then need to restart mds,
>>>
>>>
>>> our question is, does there any dead lock? how can we avoid this and
how to fix it without restart mds(it will influence other users) ?
>> BTW, won't the slow requests disappear themself later ?
>>
>> It looks like the exporting is slow or there too many exports are going
on.
>>
>> Thanks
>>
>> - Xiubo
>>
>>> Thanks a ton!
>>>
>>>
>>> xz
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5 Dec
5 Dec
4:03 a.m.
On 12/4/23 16:25, zxcs wrote:
Thanks a lot, Xiubo!
we already set ‘mds_bal_interval’ to 0. and the slow mds seems decrease.
But somehow we still see mds complain slow request. and from mds log , can see
“slow request *** seconds old, received at 2023-12-04T…: internal op exportdir:mds.*
currently acquired locks”
so our question is, why it still see "internal op exportdir”, any other config also
need to set 0? and could please shed light here which config we need set .
IMO, this should be enough.
Venky,
Did I miss something here ?
Thanks
- Xiubo
Thanks,
xz
> 2023年11月27日 13:19，Xiubo Li <xiubli(a)redhat.com> 写道：
>
>
> On 11/27/23 13:12, zxcs wrote:
>> current, we using `ceph config set mds mds_bal_interval 3600` to set a fixed
time(1 hour).
>>
>> we also have a question about how to set no balance for multi active mds.
>>
>> means, we will enable multi active mds(to improve throughput) and no balance for
these mds.
>>
>> and if we set mds_bal_interval as big number seems can void this issue?
>>
> You can just set 'mds_bal_interval' to 0.
>
>
>>
>> Thanks,
>> xz
>>
>>> 2023年11月27日 10:56，Ben <ruidong.gao(a)gmail.com> 写道：
>>>
>>> with the same mds configuration, we see exactly the same(problem, log and
>>> solution) with 17.2.5, constantly happening again and again in couples days
>>> intervals. MDS servers are stuck somewhere, ceph status reports no issue
>>> however. We need to restart some of the mds (if not all of them) to restore
>>> them back. Hopefully this could be fixed soon or get docs updated with
>>> warning for the balancer's usage in production environment.
>>>
>>> thanks and regards
>>>
>>> Xiubo Li <xiubli(a)redhat.com> 于2023年11月23日周四 15:47写道：
>>>
>>>> On 11/23/23 11:25, zxcs wrote:
>>>>> Thanks a ton, Xiubo!
>>>>>
>>>>> it not disappear.
>>>>>
>>>>> even we umount the ceph directory on these two old os node.
>>>>>
>>>>> after dump ops flight , we can see some request, and the earliest
>>>> complain “failed to authpin, subtree is being exported"
>>>>> And how to avoid this, would you please help to shed some light
here?
>>>> Okay, as Frank mentioned you can try to disable the balancer by pining
>>>> the directories. As I remembered the balancer is buggy.
>>>>
>>>> And also you can raise one ceph tracker and provide the debug logs if
>>>> you have.
>>>>
>>>> Thanks
>>>>
>>>> - Xiubo
>>>>
>>>>
>>>>> Thanks,
>>>>> xz
>>>>>
>>>>>
>>>>>> 2023年11月22日 19:44，Xiubo Li <xiubli(a)redhat.com> 写道：
>>>>>>
>>>>>>
>>>>>> On 11/22/23 16:02, zxcs wrote:
>>>>>>> HI, Experts,
>>>>>>>
>>>>>>> we are using cephfs with 16.2.* with multi active mds, and
recently,
>>>> we have two nodes mount with ceph-fuse due to the old os system.
>>>>>>> and one nodes run a python script with `glob.glob(path)`,
and another
>>>> client doing `cp` operation on the same path.
>>>>>>> then we see some log about `mds slow request`, and logs
complain
>>>> “failed to authpin, subtree is being exported"
>>>>>>> then need to restart mds,
>>>>>>>
>>>>>>>
>>>>>>> our question is, does there any dead lock? how can we avoid
this and
>>>> how to fix it without restart mds(it will influence other users) ?
>>>>>> BTW, won't the slow requests disappear themself later ?
>>>>>>
>>>>>> It looks like the exporting is slow or there too many exports are
going
>>>> on.
>>>>>> Thanks
>>>>>>
>>>>>> - Xiubo
>>>>>>
>>>>>>> Thanks a ton!
>>>>>>>
>>>>>>>
>>>>>>> xz
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:11 a.m.
On Tue, Dec 5, 2023 at 6:34 AM Xiubo Li <xiubli(a)redhat.com> wrote:
On 12/4/23 16:25, zxcs wrote:
You missed nothing. Setting `mds_bal_interval = 0` disables the
balancer. I guess there are in-progress exports that would take some
time to backoff and the slow ops should eventually get cleaned up.
I'd say wait a bit and see if the slow request resolves by itself.
FWIW, there was a feature request a while back to cancel an ongoing
export. We should prioritize having that.
Thanks a lot, Xiubo!
we already set ‘mds_bal_interval’ to 0. and the slow mds seems decrease.
But somehow we still see mds complain slow request. and from mds log , can see
“slow request *** seconds old, received at 2023-12-04T…: internal op exportdir:mds.*
currently acquired locks”
so our question is, why it still see "internal op exportdir”, any other config also
need to set 0? and could please shed light here which config we need set .
IMO, this should be enough.
Venky,
Did I miss something here ?
Thanks
- Xiubo
--
Cheers,
Venky
Thanks,
xz
> 2023年11月27日 13:19，Xiubo Li <xiubli(a)redhat.com> 写道：
>
>
> On 11/27/23 13:12, zxcs wrote:
>> current, we using `ceph config set mds mds_bal_interval 3600` to set a fixed
time(1 hour).
>>
>> we also have a question about how to set no balance for multi active mds.
>>
>> means, we will enable multi active mds(to improve throughput) and no balance for
these mds.
>>
>> and if we set mds_bal_interval as big number seems can void this issue?
>>
> You can just set 'mds_bal_interval' to 0.
>
>
>>
>> Thanks,
>> xz
>>
>>> 2023年11月27日 10:56，Ben <ruidong.gao(a)gmail.com> 写道：
>>>
>>> with the same mds configuration, we see exactly the same(problem, log and
>>> solution) with 17.2.5, constantly happening again and again in couples days
>>> intervals. MDS servers are stuck somewhere, ceph status reports no issue
>>> however. We need to restart some of the mds (if not all of them) to restore
>>> them back. Hopefully this could be fixed soon or get docs updated with
>>> warning for the balancer's usage in production environment.
>>>
>>> thanks and regards
>>>
>>> Xiubo Li <xiubli(a)redhat.com> 于2023年11月23日周四 15:47写道：
>>>
>>>> On 11/23/23 11:25, zxcs wrote:
>>>>> Thanks a ton, Xiubo!
>>>>>
>>>>> it not disappear.
>>>>>
>>>>> even we umount the ceph directory on these two old os node.
>>>>>
>>>>> after dump ops flight , we can see some request, and the earliest
>>>> complain “failed to authpin, subtree is being exported"
>>>>> And how to avoid this, would you please help to shed some light
here?
>>>> Okay, as Frank mentioned you can try to disable the balancer by pining
>>>> the directories. As I remembered the balancer is buggy.
>>>>
>>>> And also you can raise one ceph tracker and provide the debug logs if
>>>> you have.
>>>>
>>>> Thanks
>>>>
>>>> - Xiubo
>>>>
>>>>
>>>>> Thanks,
>>>>> xz
>>>>>
>>>>>
>>>>>> 2023年11月22日 19:44，Xiubo Li <xiubli(a)redhat.com> 写道：
>>>>>>
>>>>>>
>>>>>> On 11/22/23 16:02, zxcs wrote:
>>>>>>> HI, Experts,
>>>>>>>
>>>>>>> we are using cephfs with 16.2.* with multi active mds, and
recently,
>>>> we have two nodes mount with ceph-fuse due to the old os system.
>>>>>>> and one nodes run a python script with `glob.glob(path)`,
and another
>>>> client doing `cp` operation on the same path.
>>>>>>> then we see some log about `mds slow request`, and logs
complain
>>>> “failed to authpin, subtree is being exported"
>>>>>>> then need to restart mds,
>>>>>>>
>>>>>>>
>>>>>>> our question is, does there any dead lock? how can we avoid
this and
>>>> how to fix it without restart mds(it will influence other users) ?
>>>>>> BTW, won't the slow requests disappear themself later ?
>>>>>>
>>>>>> It looks like the exporting is slow or there too many exports are
going
>>>> on.
>>>>>> Thanks
>>>>>>
>>>>>> - Xiubo
>>>>>>
>>>>>>> Thanks a ton!
>>>>>>>
>>>>>>>
>>>>>>> xz
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
167
days inactive
180
days old
11 comments
6 participants
participants (6)
-
Ben -
Eugen Block -
Frank Schilder -
Venky Shankar -
Xiubo Li -
zxcs