Hi List, In order te reproduce an issue we see on a production cluster (cephFS client: ceph-fuse outperform kernel client by a factor of 5) we would like to have a test cluster to have the same cephfs "flags" as production. However, it's not completely clear how certain features influence the cephfs flags. What I could find in the source code, cephfs_features.h, is that it *seems* to correspond to the Ceph release. For example CEPHFS_FEATURE_NAUTILUS gets a "12" as feature bit. An upgraded (Luminous -> Mimic -> Nautilus) cephfs gives us the following cephfs flags: "1c". A (newly installed) Nautilus cluster gives "10" when new snapshots are not allowed (ceph fs set cephfs allow_new_snaps false) and "12" when new snapshots are allowed (ceph fs set cephfs allow_new_snaps true). We would like to have the test cluster get the "1c" flags and see if we can reproduce the issue. How can we achieve that? Any info on how those cephfs flags are constructed is welcome. Thanks, Gr. Stefan
On Thu, Dec 17, 2020 at 3:23 AM Stefan Kooman <stefan@bit.nl> wrote:
Hi List,
In order te reproduce an issue we see on a production cluster (cephFS client: ceph-fuse outperform kernel client by a factor of 5) we would like to have a test cluster to have the same cephfs "flags" as production. However, it's not completely clear how certain features influence the cephfs flags. What I could find in the source code, cephfs_features.h, is that it *seems* to correspond to the Ceph release. For example CEPHFS_FEATURE_NAUTILUS gets a "12" as feature bit. An upgraded (Luminous -> Mimic -> Nautilus) cephfs gives us the following cephfs flags: "1c".
A (newly installed) Nautilus cluster gives "10" when new snapshots are not allowed (ceph fs set cephfs allow_new_snaps false) and "12" when new snapshots are allowed (ceph fs set cephfs allow_new_snaps true).
file system flags are not the same as the "feature" flags. See this doc for the feature flags: https://docs.ceph.com/en/latest/cephfs/administration/#minimum-client-versio... Note that the new "fs feature" and "fs required_client_features" commands will be new in Pacific. They provide better control on the exact features you want to require. The old style of specifying the minimum release was inflexible and made it difficult to require specific features the kernel client supports. (For example, the kernel client is only now just about to support "nautilus" because of messenger v2 support.)
We would like to have the test cluster get the "1c" flags and see if we can reproduce the issue. How can we achieve that?
You can't set 0x1c directly. These correspond to reserved feature bits for unspecified older Ceph releases. Suggest you just set the min_compat_client to jewel. In any case, I think what you're asking is about the file system flags and not the required_client_features. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 12/17/20 5:54 PM, Patrick Donnelly wrote:
file system flags are not the same as the "feature" flags. See this doc for the feature flags:
https://docs.ceph.com/en/latest/cephfs/administration/#minimum-client-versio...
Thanks for making that clear.
Note that the new "fs feature" and "fs required_client_features" commands will be new in Pacific. They provide better control on the exact features you want to require. The old style of specifying the minimum release was inflexible and made it difficult to require specific features the kernel client supports. (For example, the kernel client is only now just about to support "nautilus" because of messenger v2 support.)
Ah, nice. Both the fs feature part as v2 support for kernel.
We would like to have the test cluster get the "1c" flags and see if we can reproduce the issue. How can we achieve that?
You can't set 0x1c directly. These correspond to reserved feature bits for unspecified older Ceph releases. Suggest you just set the min_compat_client to jewel.
Up to now we have never set a "min_compat_client". But I guess we can enforce jewel nowadays (thats the lowest ceph features gives for clients).
In any case, I think what you're asking is about the file system flags and not the required_client_features.
That's correct. So I checked the file system flags on different clusters (some installed luminous, some mimic, some nautilus) and for the clusters that started as luminous the file sytems flags are either "1c" or "1e". The ones with "1e" have been installed with newer luminous releases. So does the filesystem flags bit ever change during the lifetime of a cluster? What exactly is the purpose of the filesystem flags bit? Thanks, Gr. Stefan
On Thu, Dec 17, 2020 at 10:27 AM Stefan Kooman <stefan@bit.nl> wrote:
In any case, I think what you're asking is about the file system flags and not the required_client_features.
That's correct. So I checked the file system flags on different clusters (some installed luminous, some mimic, some nautilus) and for the clusters that started as luminous the file sytems flags are either "1c" or "1e". The ones with "1e" have been installed with newer luminous releases. So does the filesystem flags bit ever change during the lifetime of a cluster? What exactly is the purpose of the filesystem flags bit?
When a file system is newly created, it's assumed you want all the stable features on, including multiple MDS, directory fragmentation, snapshots, etc. That's what those flags are for. If you've been upgrading your cluster, you need to turn those on yourself. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 12/17/20 7:45 PM, Patrick Donnelly wrote:
When a file system is newly created, it's assumed you want all the stable features on, including multiple MDS, directory fragmentation, snapshots, etc. That's what those flags are for. If you've been upgrading your cluster, you need to turn those on yourself.
OK, fair enough. So I tried adding "allow_dirfrags" which gives me: ceph fs set cephfs allow_dirfrags true Directory fragmentation is now permanently enabled. This command is DEPRECATED and will be REMOVED from future releases. And I enabled snapshot support: ceph fs set cephfs allow_new_snaps true enabled new snapshots However, this has not changed the "flags" of the filesystem in any way. So I guess there are still features not enabled that are enabled on newly installed clusters. Where can I find a list of features that I can enable? I have searched through documentation but I don't see anything related. It's also not described / suggested in the part about upgrading the MDS cluster (IMHO that would be a logical place) [1]. Gr. Stefan [1]: https://docs.ceph.com/en/latest/cephfs/upgrading/
On Thu, Dec 17, 2020 at 11:35 AM Stefan Kooman <stefan@bit.nl> wrote:
On 12/17/20 7:45 PM, Patrick Donnelly wrote:
When a file system is newly created, it's assumed you want all the stable features on, including multiple MDS, directory fragmentation, snapshots, etc. That's what those flags are for. If you've been upgrading your cluster, you need to turn those on yourself.
OK, fair enough. So I tried adding "allow_dirfrags" which gives me:
ceph fs set cephfs allow_dirfrags true Directory fragmentation is now permanently enabled. This command is DEPRECATED and will be REMOVED from future releases.
And I enabled snapshot support:
ceph fs set cephfs allow_new_snaps true enabled new snapshots
However, this has not changed the "flags" of the filesystem in any way. So I guess there are still features not enabled that are enabled on newly installed clusters. Where can I find a list of features that I can enable?
Apologies for linking code: https://github.com/ceph/ceph/blob/master/src/include/ceph_fs.h#L275-L285
I have searched through documentation but I don't see anything related. It's also not described / suggested in the part about upgrading the MDS cluster (IMHO that would be a logical place) [1].
You're the first person I'm aware of asking for this. :) -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi, On 12/17/20 8:57 PM, Patrick Donnelly wrote:
On Thu, Dec 17, 2020 at 11:35 AM Stefan Kooman <stefan@bit.nl> wrote:
On 12/17/20 7:45 PM, Patrick Donnelly wrote:
When a file system is newly created, it's assumed you want all the stable features on, including multiple MDS, directory fragmentation, snapshots, etc. That's what those flags are for. If you've been upgrading your cluster, you need to turn those on yourself.
OK, fair enough. So I tried adding "allow_dirfrags" which gives me:
ceph fs set cephfs allow_dirfrags true Directory fragmentation is now permanently enabled. This command is DEPRECATED and will be REMOVED from future releases.
And I enabled snapshot support:
ceph fs set cephfs allow_new_snaps true enabled new snapshots
However, this has not changed the "flags" of the filesystem in any way. So I guess there are still features not enabled that are enabled on newly installed clusters. Where can I find a list of features that I can enable?
Apologies for linking code: https://github.com/ceph/ceph/blob/master/src/include/ceph_fs.h#L275-L285
No problem.
I have searched through documentation but I don't see anything related. It's also not described / suggested in the part about upgrading the MDS cluster (IMHO that would be a logical place) [1].
You're the first person I'm aware of asking for this. :)
Somehow I'm not suprised :-). But on the other hand, I am. I will try to explain this. Maybe this works best with an example. Nautilus 14.2.4 cluster here (upgraded from luminous, mimic). relevant part of ceph -s: mds: cephfs:1 {0=mds2=up:active} 1 up:standby-replay ^^ Two MDSes in this cluster: one active and one standby-replay. ceph fs get cephfs |grep flags flags 1e Let's try to enable standby replay support: ceph fs set cephfs allow_standby_replay true That worked, did flags change? ceph fs get cephfs |grep flags flags 3e Yes! But why? I would expect the support for standby replay to have been enabled already. How else would it work even without setting this fs feature. But apparently it does, and does not need this feature to be enabled like this. And that might explain that nobody ever wondered how to change "ceph fs flags" in the first place. Is that correct? At this point I ask myself the question: who / what uses the cephfs flags, and what fort. Do I, as a storage admin, need to care about this at all? But hey, here we are, and now I would like to understand it. If, just for the sake of upgrading clusters to have identical features, I would like to "upgrade" the cephfs to support all ceph fs features, I seem not to be able to do that: ceph fs set cephfs Invalid command: missing required parameter var(max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client) fs set <fs_name> max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client <val> {--yes-i-really-mean-it} : set fs parameter <var> to <val> Error EINVAL: invalid command I can choose between these: max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client Most of them I would not like to set at all (i.e down, joinable, max_mds) as they are not "features" but merely a way to set the ceph fs in a certain STATE. So my question is: what do I need to enable to get an upgraded fs to get say "flags 12" (Nautilus with snapshot support enabled AFAIK). Is that at all possible? The reason why I started this whole thread was to eliminate any Ceph config related difference between production and test. But maybe I should ask a different question: Does a (ceph-fuse / kernel) client use the *cephfs flags* bit at all? If not than we don't have to focus on this, and we can conclude we cannot reproduce the issue on our test environment. I hope above makes sense to you ;-). Thanks, Gr. Stefan
On Fri, Dec 18, 2020 at 6:28 AM Stefan Kooman <stefan@bit.nl> wrote:
I have searched through documentation but I don't see anything related. It's also not described / suggested in the part about upgrading the MDS cluster (IMHO that would be a logical place) [1].
You're the first person I'm aware of asking for this. :)
Somehow I'm not suprised :-). But on the other hand, I am. I will try to explain this. Maybe this works best with an example.
Nautilus 14.2.4 cluster here (upgraded from luminous, mimic).
relevant part of ceph -s:
mds: cephfs:1 {0=mds2=up:active} 1 up:standby-replay
^^ Two MDSes in this cluster: one active and one standby-replay.
ceph fs get cephfs |grep flags flags 1e
Let's try to enable standby replay support:
ceph fs set cephfs allow_standby_replay true
That worked, did flags change?
ceph fs get cephfs |grep flags flags 3e
Yes! But why?
Well that's interesting. I don't have an explanation unfortunately. You upgraded the MDS too, right? Only scenario that could cause this I can think of is that the MDS were never restarted/upgraded to nautilus.
I would expect the support for standby replay to have been enabled already. How else would it work even without setting this fs feature. But apparently it does, and does not need this feature to be enabled like this. And that might explain that nobody ever wondered how to change "ceph fs flags" in the first place. Is that correct?
At this point I ask myself the question: who / what uses the cephfs flags, and what fort. Do I, as a storage admin, need to care about this at all?
Operators should only care about "is X flag turned on" but we don't really show that very well in the MDSMap dump. I'll make a note to improve that. We'd really rather not that operators need to do bitwise arithmetic on the flags bitfield to determine what features are turned on. https://tracker.ceph.com/issues/48683
But hey, here we are, and now I would like to undersand it.
If, just for the sake of upgrading clusters to have identical features, I would like to "upgrade" the cephfs to support all ceph fs features, I seem not to be able to do that:
ceph fs set cephfs Invalid command: missing required parameter var(max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client) fs set <fs_name> max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client <val> {--yes-i-really-mean-it} : set fs parameter <var> to <val> Error EINVAL: invalid command
I can choose between these: max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client
Most of them I would not like to set at all (i.e down, joinable, max_mds) as they are not "features" but merely a way to set the ceph fs in a certain STATE.
So my question is: what do I need to enable to get an upgraded fs to get say "flags 12" (Nautilus with snapshot support enabled AFAIK). Is that at all possible?
I'll also add a note to list features that can be turned on: https://tracker.ceph.com/issues/48682
The reason why I started this whole thread was to eliminate any Ceph config related difference between production and test. But maybe I should ask a different question: Does a (ceph-fuse / kernel) client use the *cephfs flags* bit at all? If not than we don't have to focus on this, and we can conclude we cannot reproduce the issue on our test environment.
ceph-fuse/kernel client don't use these flags. Only the MDS.
I hope above makes sense to you ;-).
Thanks,
Gr. Stefan
-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 12/19/20 7:37 PM, Patrick Donnelly wrote:
Well that's interesting. I don't have an explanation unfortunately. You upgraded the MDS too, right? Only scenario that could cause this I can think of is that the MDS were never restarted/upgraded to nautilus.
Yes, the MDSes were upgraded and restarted for sure.
Operators should only care about "is X flag turned on" but we don't really show that very well in the MDSMap dump. I'll make a note to improve that. We'd really rather not that operators need to do bitwise arithmetic on the flags bitfield to determine what features are turned on.
Thanks.
I'll also add a note to list features that can be turned on: https://tracker.ceph.com/issues/48682
OK.
ceph-fuse/kernel client don't use these flags. Only the MDS.
Check. Gr. Stefan
participants (2)
-
Patrick Donnelly
-
Stefan Kooman