31 Mar
2020
31 Mar
'20
1:32 a.m.
Hi guys,
This is documented as an experimental feature, but it doesn’t explain how to ensure that
affinity for a given MDS sticks to the second filesystem you create. Has anyone had
success implementing a second CephFS? In my case it will be based on a completely
different pool from my first one.
Thanks.
J
31 Mar
31 Mar
9:57 a.m.
Hi,
to create a second filesystem you have to use different pools anyway.
If you already have one CephFS up and running then you also should
have at least one standby daemon, right? If you create a new FS and
that standby daemon is not configured to any specific rank then it
will be used for the second filesystem. Now you'll have two active MDS
both with rank 0 (active):
---snip---
ceph:~ # ceph fs status
cephfs - 1 clients
======
+------+--------+-------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+-------+---------------+-------+-------+
| 0 | active | host6 | Reqs: 0 /s | 10 | 13 |
+------+--------+-------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 1536k | 92.0G |
| cephfs_data | data | 5053M | 92.0G |
+-----------------+----------+-------+-------+
cephfs2 - 0 clients
=======
+------+--------+-------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+-------+---------------+-------+-------+
| 0 | active | host5 | Reqs: 0 /s | 10 | 13 |
+------+--------+-------+---------------+-------+-------+
+------------------+----------+-------+-------+
| Pool | type | used | avail |
+------------------+----------+-------+-------+
| cephfs2_metadata | metadata | 1536k | 92.0G |
| cephfs2_data | data | 0 | 92.0G |
+------------------+----------+-------+-------+
+-------------+
| Standby MDS |
+-------------+
+-------------+
---snip---
For the standby daemon you have to be aware of this:
By default, if none of these settings are used, all
MDS daemons
which do not hold a rank will
be used as 'standbys' for any rank.
[...]
When a daemon has entered the standby replay state, it will only be
used as a standby for
the rank that it is following. If another rank fails, this standby
replay daemon will not be
used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank
mds_standby_for_name
mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them
handle the failover.
Regards,
Eugen
Zitat von Jarett DeAngelis <jarett(a)reticulum.us>us>:
Hi guys,
This is documented as an experimental feature, but it doesn’t
explain how to ensure that affinity for a given MDS sticks to the
second filesystem you create. Has anyone had success implementing a
second CephFS? In my case it will be based on a completely different
pool from my first one.
Thanks.
J
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:12 a.m.
On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock(a)nde.ag> wrote:
For the standby daemon you have to be aware of this:
This has changed in Octopus. The above config variables are removed.
Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
By default, if none of these settings are used,
all MDS daemons
which do not hold a rank will
be used as 'standbys' for any rank.
[...]
When a daemon has entered the standby replay state, it will only be
used as a standby for
the rank that it is following. If another rank fails, this standby
replay daemon will not be
used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank
mds_standby_for_name
mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them
handle the failover.
10:15 a.m.
Thanks for this. Still on Nautilus here because this is a Proxmox cluster
but good for folks tracking master to know.
J
On Tue, Mar 31, 2020, 3:14 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote:
On Mon, Mar 30, 2020 at 11:57 PM Eugen Block
<eblock(a)nde.ag> wrote:
For the standby daemon you have to be aware of
this:
This has changed in Octopus. The above config variables are removed.
Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
By default, if none of these settings are used,
all MDS daemons
which do not hold a rank will
be used as 'standbys' for any rank.
[...]
When a daemon has entered the standby replay state, it will only be
used as a standby for
the rank that it is following. If another rank fails, this standby
replay daemon will not be
used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank
mds_standby_for_name
mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them
handle the failover.
10:19 a.m.
This has changed in Octopus. The above config
variables are removed.
Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
Thanks for the clarification, IIRC I had troubles applying the
mds_standby settings in Nautilus already, but I haven't verified yet
so I didn't mention that in my response. I'll take another look at it.
Zitat von Patrick Donnelly <pdonnell(a)redhat.com>om>:
> On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock(a)nde.ag> wrote:
>> For the standby daemon you have to be aware of this:
>>
>> > By default, if none of these settings are used, all MDS daemons
>> > which do not hold a rank will
>> > be used as 'standbys' for any rank.
>> > [...]
>> > When a daemon has entered the standby replay state, it will only be
>> > used as a standby for
>> > the rank that it is following. If another rank fails, this standby
>> > replay daemon will not be
>> > used as a replacement, even if no other standbys are available.
>>
>> Some of the mentioned settings are for example:
>>
>> mds_standby_for_rank
>> mds_standby_for_name
>> mds_standby_for_fscid
>>
>> The easiest way is to have one standby daemon per CephFS and let them
>> handle the failover.
>
This has changed in Octopus. The above config
variables are removed.
Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
>
> --
> Patrick Donnelly, Ph.D.
> He / Him / His
> Senior Software Engineer
> Red Hat Sunnyvale, CA
> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
7:51 p.m.
So, for the record, this doesn’t appears to work in Nautilus.
Does this mean that I should just count on my standby MDS to “step in” when a new FS is
created?
On Mar 31, 2020, at 3:19 AM, Eugen Block
<eblock(a)nde.ag> wrote:
This has changed in Octopus. The above config
variables are removed.
Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
Thanks for the clarification, IIRC I had troubles applying the mds_standby settings in
Nautilus already, but I haven't verified yet so I didn't mention that in my
response. I'll take another look at it.
Zitat von Patrick Donnelly <pdonnell(a)redhat.com>om>:
> On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock(a)nde.ag> wrote:
>> For the standby daemon you have to be aware of this:
>>
>> > By default, if none of these settings are used, all MDS daemons
>> > which do not hold a rank will
>> > be used as 'standbys' for any rank.
>> > [...]
>> > When a daemon has entered the standby replay state, it will only be
>> > used as a standby for
>> > the rank that it is following. If another rank fails, this standby
>> > replay daemon will not be
>> > used as a replacement, even if no other standbys are available.
>>
>> Some of the mentioned settings are for example:
>>
>> mds_standby_for_rank
>> mds_standby_for_name
>> mds_standby_for_fscid
>>
>> The easiest way is to have one standby daemon per CephFS and let them
>> handle the failover.
>
This has changed in Octopus. The above config
variables are removed.
Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
>
> --
> Patrick Donnelly, Ph.D.
> He / Him / His
> Senior Software Engineer
> Red Hat Sunnyvale, CA
> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
9:15 p.m.
Yes, standby (as opposed to standby-replay) MDS' form a shared pool
from which the mons will promote an MDS to the required role.
On Tue, Mar 31, 2020 at 12:52 PM Jarett DeAngelis <jarett(a)reticulum.us> wrote:
>
> So, for the record, this doesn’t appears to work in Nautilus.
>
>
>
> Does this mean that I should just count on my standby MDS to “step in” when a new FS
is created?
>
> > On Mar 31, 2020, at 3:19 AM, Eugen Block <eblock(a)nde.ag> wrote:
> >
> >> This has changed in Octopus. The above config variables are removed.
> >> Instead, follow this procedure.:
> >>
> >>
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
> >
> > Thanks for the clarification, IIRC I had troubles applying the mds_standby
settings in Nautilus already, but I haven't verified yet so I didn't mention that
in my response. I'll take another look at it.
> >
> >
> > Zitat von Patrick Donnelly <pdonnell(a)redhat.com>om>:
> >
> >> On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock(a)nde.ag> wrote:
> >>> For the standby daemon you have to be aware of this:
> >>>
> >>> > By default, if none of these settings are used, all MDS daemons
> >>> > which do not hold a rank will
> >>> > be used as 'standbys' for any rank.
> >>> > [...]
> >>> > When a daemon has entered the standby replay state, it will only
be
> >>> > used as a standby for
> >>> > the rank that it is following. If another rank fails, this standby
> >>> > replay daemon will not be
> >>> > used as a replacement, even if no other standbys are available.
> >>>
> >>> Some of the mentioned settings are for example:
> >>>
> >>> mds_standby_for_rank
> >>> mds_standby_for_name
> >>> mds_standby_for_fscid
> >>>
> >>> The easiest way is to have one standby daemon per CephFS and let them
> >>> handle the failover.
> >>
> >> This has changed in Octopus. The above config variables are removed.
> >> Instead, follow this procedure.:
> >>
> >>
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
> >>
> >> --
> >> Patrick Donnelly, Ph.D.
> >> He / Him / His
> >> Senior Software Engineer
> >> Red Hat Sunnyvale, CA
> >> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1 Apr
1 Apr
7:24 a.m.
Thanks. I’m now trying to figure out how to get Proxmox to pass the “-o
mds_namespace=otherfs” option to its mounting of the filesystem, but that’s a bit out of
scope for this list (though if anyone has done this please let me know!).
On Mar 31, 2020, at 2:15 PM, Nathan Fish
<lordcirth(a)gmail.com> wrote:
Yes, standby (as opposed to standby-replay) MDS' form a shared pool
from which the mons will promote an MDS to the required role.
On Tue, Mar 31, 2020 at 12:52 PM Jarett DeAngelis <jarett(a)reticulum.us> wrote:
>
> So, for the record, this doesn’t appears to work in Nautilus.
>
>
>
> Does this mean that I should just count on my standby MDS to “step in” when a new FS
is created?
>
>> On Mar 31, 2020, at 3:19 AM, Eugen Block <eblock(a)nde.ag> wrote:
>>
>>> This has changed in Octopus. The above config variables are removed.
>>> Instead, follow this procedure.:
>>>
>>>
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
>>
>> Thanks for the clarification, IIRC I had troubles applying the mds_standby
settings in Nautilus already, but I haven't verified yet so I didn't mention that
in my response. I'll take another look at it.
>>
>>
>> Zitat von Patrick Donnelly <pdonnell(a)redhat.com>om>:
>>
>>> On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock(a)nde.ag> wrote:
>>>> For the standby daemon you have to be aware of this:
>>>>
>>>>> By default, if none of these settings are used, all MDS daemons
>>>>> which do not hold a rank will
>>>>> be used as 'standbys' for any rank.
>>>>> [...]
>>>>> When a daemon has entered the standby replay state, it will only be
>>>>> used as a standby for
>>>>> the rank that it is following. If another rank fails, this standby
>>>>> replay daemon will not be
>>>>> used as a replacement, even if no other standbys are available.
>>>>
>>>> Some of the mentioned settings are for example:
>>>>
>>>> mds_standby_for_rank
>>>> mds_standby_for_name
>>>> mds_standby_for_fscid
>>>>
>>>> The easiest way is to have one standby daemon per CephFS and let them
>>>> handle the failover.
>>>
>>> This has changed in Octopus. The above config variables are removed.
>>> Instead, follow this procedure.:
>>>
>>>
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
>>>
>>> --
>>> Patrick Donnelly, Ph.D.
>>> He / Him / His
>>> Senior Software Engineer
>>> Red Hat Sunnyvale, CA
>>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:56 a.m.
You already have the correct option, there's not much to it:
mount -t ceph mon1,mon2,mon3:/<path>/ -o
name=<client>,secretfile=<keyring_file>,mds_namespace=<otherfs>
/<mountpoint>/
If your caps and path restrictions are correct this should work.
Zitat von Jarett DeAngelis <jarett(a)reticulum.us>us>:
Thanks. I’m now trying to figure out how to get
Proxmox to pass the
“-o mds_namespace=otherfs” option to its mounting of the filesystem,
but that’s a bit out of scope for this list (though if anyone has
done this please let me know!).
> On Mar 31, 2020, at 2:15 PM, Nathan Fish <lordcirth(a)gmail.com> wrote:
>
> Yes, standby (as opposed to standby-replay) MDS' form a shared pool
> from which the mons will promote an MDS to the required role.
>
> On Tue, Mar 31, 2020 at 12:52 PM Jarett DeAngelis
> <jarett(a)reticulum.us> wrote:
>>
>> So, for the record, this doesn’t appears to work in Nautilus.
>>
>>
>>
>> Does this mean that I should just count on my standby MDS to “step
>> in” when a new FS is created?
>>
>>> On Mar 31, 2020, at 3:19 AM, Eugen Block <eblock(a)nde.ag> wrote:
>>>
>>>> This has changed in Octopus. The above config variables are removed.
>>>> Instead, follow this procedure.:
>>>>
>>>>
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
>>>
>>> Thanks for the clarification, IIRC I had troubles applying the
>>> mds_standby settings in Nautilus already, but I haven't verified
>>> yet so I didn't mention that in my response. I'll take another
>>> look at it.
>>>
>>>
>>> Zitat von Patrick Donnelly <pdonnell(a)redhat.com>om>:
>>>
>>>> On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock(a)nde.ag>
wrote:
>>>>> For the standby daemon you have to be aware of this:
>>>>>
>>>>>> By default, if none of these settings are used, all MDS daemons
>>>>>> which do not hold a rank will
>>>>>> be used as 'standbys' for any rank.
>>>>>> [...]
>>>>>> When a daemon has entered the standby replay state, it will only
be
>>>>>> used as a standby for
>>>>>> the rank that it is following. If another rank fails, this
standby
>>>>>> replay daemon will not be
>>>>>> used as a replacement, even if no other standbys are available.
>>>>>
>>>>> Some of the mentioned settings are for example:
>>>>>
>>>>> mds_standby_for_rank
>>>>> mds_standby_for_name
>>>>> mds_standby_for_fscid
>>>>>
>>>>> The easiest way is to have one standby daemon per CephFS and let
them
>>>>> handle the failover.
>>>>
>>>> This has changed in Octopus. The above config variables are removed.
>>>> Instead, follow this procedure.:
>>>>
>>>>
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-sys…
>>>>
>>>> --
>>>> Patrick Donnelly, Ph.D.
>>>> He / Him / His
>>>> Senior Software Engineer
>>>> Red Hat Sunnyvale, CA
>>>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1494
days inactive
1496
days old
8 comments
4 participants
participants (4)
-
Eugen Block -
Jarett DeAngelis -
Nathan Fish -
Patrick Donnelly