Multiple CephFS creation
Hi guys, This is documented as an experimental feature, but it doesn’t explain how to ensure that affinity for a given MDS sticks to the second filesystem you create. Has anyone had success implementing a second CephFS? In my case it will be based on a completely different pool from my first one. Thanks. J
Hi, to create a second filesystem you have to use different pools anyway. If you already have one CephFS up and running then you also should have at least one standby daemon, right? If you create a new FS and that standby daemon is not configured to any specific rank then it will be used for the second filesystem. Now you'll have two active MDS both with rank 0 (active): ---snip--- ceph:~ # ceph fs status cephfs - 1 clients ====== +------+--------+-------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+-------+---------------+-------+-------+ | 0 | active | host6 | Reqs: 0 /s | 10 | 13 | +------+--------+-------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 1536k | 92.0G | | cephfs_data | data | 5053M | 92.0G | +-----------------+----------+-------+-------+ cephfs2 - 0 clients ======= +------+--------+-------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+-------+---------------+-------+-------+ | 0 | active | host5 | Reqs: 0 /s | 10 | 13 | +------+--------+-------+---------------+-------+-------+ +------------------+----------+-------+-------+ | Pool | type | used | avail | +------------------+----------+-------+-------+ | cephfs2_metadata | metadata | 1536k | 92.0G | | cephfs2_data | data | 0 | 92.0G | +------------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ ---snip--- For the standby daemon you have to be aware of this:
By default, if none of these settings are used, all MDS daemons which do not hold a rank will be used as 'standbys' for any rank. [...] When a daemon has entered the standby replay state, it will only be used as a standby for the rank that it is following. If another rank fails, this standby replay daemon will not be used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example: mds_standby_for_rank mds_standby_for_name mds_standby_for_fscid The easiest way is to have one standby daemon per CephFS and let them handle the failover. Regards, Eugen Zitat von Jarett DeAngelis <jarett@reticulum.us>:
Hi guys,
This is documented as an experimental feature, but it doesn’t explain how to ensure that affinity for a given MDS sticks to the second filesystem you create. Has anyone had success implementing a second CephFS? In my case it will be based on a completely different pool from my first one.
Thanks. J _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock@nde.ag> wrote:
For the standby daemon you have to be aware of this:
By default, if none of these settings are used, all MDS daemons which do not hold a rank will be used as 'standbys' for any rank. [...] When a daemon has entered the standby replay state, it will only be used as a standby for the rank that it is following. If another rank fails, this standby replay daemon will not be used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank mds_standby_for_name mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them handle the failover.
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.: https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst... -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Thanks for this. Still on Nautilus here because this is a Proxmox cluster but good for folks tracking master to know. J On Tue, Mar 31, 2020, 3:14 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock@nde.ag> wrote:
For the standby daemon you have to be aware of this:
By default, if none of these settings are used, all MDS daemons which do not hold a rank will be used as 'standbys' for any rank. [...] When a daemon has entered the standby replay state, it will only be used as a standby for the rank that it is following. If another rank fails, this standby replay daemon will not be used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank mds_standby_for_name mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them handle the failover.
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
Thanks for the clarification, IIRC I had troubles applying the mds_standby settings in Nautilus already, but I haven't verified yet so I didn't mention that in my response. I'll take another look at it. Zitat von Patrick Donnelly <pdonnell@redhat.com>:
On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock@nde.ag> wrote:
For the standby daemon you have to be aware of this:
By default, if none of these settings are used, all MDS daemons which do not hold a rank will be used as 'standbys' for any rank. [...] When a daemon has entered the standby replay state, it will only be used as a standby for the rank that it is following. If another rank fails, this standby replay daemon will not be used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank mds_standby_for_name mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them handle the failover.
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
So, for the record, this doesn’t appears to work in Nautilus. Does this mean that I should just count on my standby MDS to “step in” when a new FS is created?
On Mar 31, 2020, at 3:19 AM, Eugen Block <eblock@nde.ag> wrote:
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
Thanks for the clarification, IIRC I had troubles applying the mds_standby settings in Nautilus already, but I haven't verified yet so I didn't mention that in my response. I'll take another look at it.
Zitat von Patrick Donnelly <pdonnell@redhat.com>:
On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock@nde.ag> wrote:
For the standby daemon you have to be aware of this:
By default, if none of these settings are used, all MDS daemons which do not hold a rank will be used as 'standbys' for any rank. [...] When a daemon has entered the standby replay state, it will only be used as a standby for the rank that it is following. If another rank fails, this standby replay daemon will not be used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank mds_standby_for_name mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them handle the failover.
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, standby (as opposed to standby-replay) MDS' form a shared pool from which the mons will promote an MDS to the required role. On Tue, Mar 31, 2020 at 12:52 PM Jarett DeAngelis <jarett@reticulum.us> wrote:
So, for the record, this doesn’t appears to work in Nautilus.
Does this mean that I should just count on my standby MDS to “step in” when a new FS is created?
On Mar 31, 2020, at 3:19 AM, Eugen Block <eblock@nde.ag> wrote:
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
Thanks for the clarification, IIRC I had troubles applying the mds_standby settings in Nautilus already, but I haven't verified yet so I didn't mention that in my response. I'll take another look at it.
Zitat von Patrick Donnelly <pdonnell@redhat.com>:
On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock@nde.ag> wrote:
For the standby daemon you have to be aware of this:
By default, if none of these settings are used, all MDS daemons which do not hold a rank will be used as 'standbys' for any rank. [...] When a daemon has entered the standby replay state, it will only be used as a standby for the rank that it is following. If another rank fails, this standby replay daemon will not be used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank mds_standby_for_name mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them handle the failover.
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks. I’m now trying to figure out how to get Proxmox to pass the “-o mds_namespace=otherfs” option to its mounting of the filesystem, but that’s a bit out of scope for this list (though if anyone has done this please let me know!).
On Mar 31, 2020, at 2:15 PM, Nathan Fish <lordcirth@gmail.com> wrote:
Yes, standby (as opposed to standby-replay) MDS' form a shared pool from which the mons will promote an MDS to the required role.
On Tue, Mar 31, 2020 at 12:52 PM Jarett DeAngelis <jarett@reticulum.us> wrote:
So, for the record, this doesn’t appears to work in Nautilus.
Does this mean that I should just count on my standby MDS to “step in” when a new FS is created?
On Mar 31, 2020, at 3:19 AM, Eugen Block <eblock@nde.ag> wrote:
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
Thanks for the clarification, IIRC I had troubles applying the mds_standby settings in Nautilus already, but I haven't verified yet so I didn't mention that in my response. I'll take another look at it.
Zitat von Patrick Donnelly <pdonnell@redhat.com>:
On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock@nde.ag> wrote:
For the standby daemon you have to be aware of this:
By default, if none of these settings are used, all MDS daemons which do not hold a rank will be used as 'standbys' for any rank. [...] When a daemon has entered the standby replay state, it will only be used as a standby for the rank that it is following. If another rank fails, this standby replay daemon will not be used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank mds_standby_for_name mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them handle the failover.
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You already have the correct option, there's not much to it: mount -t ceph mon1,mon2,mon3:/<path>/ -o name=<client>,secretfile=<keyring_file>,mds_namespace=<otherfs> /<mountpoint>/ If your caps and path restrictions are correct this should work. Zitat von Jarett DeAngelis <jarett@reticulum.us>:
Thanks. I’m now trying to figure out how to get Proxmox to pass the “-o mds_namespace=otherfs” option to its mounting of the filesystem, but that’s a bit out of scope for this list (though if anyone has done this please let me know!).
On Mar 31, 2020, at 2:15 PM, Nathan Fish <lordcirth@gmail.com> wrote:
Yes, standby (as opposed to standby-replay) MDS' form a shared pool from which the mons will promote an MDS to the required role.
On Tue, Mar 31, 2020 at 12:52 PM Jarett DeAngelis <jarett@reticulum.us> wrote:
So, for the record, this doesn’t appears to work in Nautilus.
Does this mean that I should just count on my standby MDS to “step in” when a new FS is created?
On Mar 31, 2020, at 3:19 AM, Eugen Block <eblock@nde.ag> wrote:
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
Thanks for the clarification, IIRC I had troubles applying the mds_standby settings in Nautilus already, but I haven't verified yet so I didn't mention that in my response. I'll take another look at it.
Zitat von Patrick Donnelly <pdonnell@redhat.com>:
On Mon, Mar 30, 2020 at 11:57 PM Eugen Block <eblock@nde.ag> wrote:
For the standby daemon you have to be aware of this:
> By default, if none of these settings are used, all MDS daemons > which do not hold a rank will > be used as 'standbys' for any rank. > [...] > When a daemon has entered the standby replay state, it will only be > used as a standby for > the rank that it is following. If another rank fails, this standby > replay daemon will not be > used as a replacement, even if no other standbys are available.
Some of the mentioned settings are for example:
mds_standby_for_rank mds_standby_for_name mds_standby_for_fscid
The easiest way is to have one standby daemon per CephFS and let them handle the failover.
This has changed in Octopus. The above config variables are removed. Instead, follow this procedure.:
https://docs.ceph.com/docs/octopus/cephfs/standby/#configuring-mds-file-syst...
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
Jarett DeAngelis
-
Nathan Fish
-
Patrick Donnelly