On Fri, Dec 18, 2020 at 6:28 AM Stefan Kooman <stefan@bit.nl> wrote:
I have searched through documentation but I don't see anything related. It's also not described / suggested in the part about upgrading the MDS cluster (IMHO that would be a logical place) [1].
You're the first person I'm aware of asking for this. :)
Somehow I'm not suprised :-). But on the other hand, I am. I will try to explain this. Maybe this works best with an example.
Nautilus 14.2.4 cluster here (upgraded from luminous, mimic).
relevant part of ceph -s:
mds: cephfs:1 {0=mds2=up:active} 1 up:standby-replay
^^ Two MDSes in this cluster: one active and one standby-replay.
ceph fs get cephfs |grep flags flags 1e
Let's try to enable standby replay support:
ceph fs set cephfs allow_standby_replay true
That worked, did flags change?
ceph fs get cephfs |grep flags flags 3e
Yes! But why?
Well that's interesting. I don't have an explanation unfortunately. You upgraded the MDS too, right? Only scenario that could cause this I can think of is that the MDS were never restarted/upgraded to nautilus.
I would expect the support for standby replay to have been enabled already. How else would it work even without setting this fs feature. But apparently it does, and does not need this feature to be enabled like this. And that might explain that nobody ever wondered how to change "ceph fs flags" in the first place. Is that correct?
At this point I ask myself the question: who / what uses the cephfs flags, and what fort. Do I, as a storage admin, need to care about this at all?
Operators should only care about "is X flag turned on" but we don't really show that very well in the MDSMap dump. I'll make a note to improve that. We'd really rather not that operators need to do bitwise arithmetic on the flags bitfield to determine what features are turned on. https://tracker.ceph.com/issues/48683
But hey, here we are, and now I would like to undersand it.
If, just for the sake of upgrading clusters to have identical features, I would like to "upgrade" the cephfs to support all ceph fs features, I seem not to be able to do that:
ceph fs set cephfs Invalid command: missing required parameter var(max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client) fs set <fs_name> max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client <val> {--yes-i-really-mean-it} : set fs parameter <var> to <val> Error EINVAL: invalid command
I can choose between these: max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client
Most of them I would not like to set at all (i.e down, joinable, max_mds) as they are not "features" but merely a way to set the ceph fs in a certain STATE.
So my question is: what do I need to enable to get an upgraded fs to get say "flags 12" (Nautilus with snapshot support enabled AFAIK). Is that at all possible?
I'll also add a note to list features that can be turned on: https://tracker.ceph.com/issues/48682
The reason why I started this whole thread was to eliminate any Ceph config related difference between production and test. But maybe I should ask a different question: Does a (ceph-fuse / kernel) client use the *cephfs flags* bit at all? If not than we don't have to focus on this, and we can conclude we cannot reproduce the issue on our test environment.
ceph-fuse/kernel client don't use these flags. Only the MDS.
I hope above makes sense to you ;-).
Thanks,
Gr. Stefan
-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D