28 May
2020
28 May
'20
4:58 p.m.
On Thu, 28 May 2020, John Mulligan wrote:
Both librados and libcephfs have functions that are
named like *_command that
generally take json formatted input "commands".
Across all the functions the json `{"prefix: "get_command_descriptions"}`
appears to work everywhere, returning a json formatted listing of the
available commands.
Related to this I have a few questions that I have not been able to answer
myself just by looking at the sources:
There appears to be a distinction between 'tell' commands and those that are
not 'tell' commands but I couldn't quite figure it all out on my own.
What's
the difference and how can I determine which commands in the listing are tell
commands or not?
'tell' commands are sent to a specific daemon, and (starting with
octopus) map 1:1 to the admin socket ('ceph daemon <name> help')
commands. Non-tell commands are normal CLI commands that are routed to
any mon (or, if appropriate, the active mgr).
Sometimes certain commands appear in more than one
listing. An example that is
relevant to some upcoming work for the ceph-csi project is subvolume creation
for cephfs. By tracing/instrumenting the `ceph` cli tool I can see that (on my
system) it sends the command to create the subvolume via mon_command, but the
command ("fs subvolume create") appears for both mon_command and mgr_command.
I would like some advice about which command function we should prefer to use
even if both work. I tend to prefer doing what the cli does, as I assume that
gets a lot of real work testing. However, if someone on the list who
understands the fundamentals of this system better could provide some advice
I'd greatly appreciate it.
If the FLAG_MGR
static const uint64_t FLAG_MGR = 1 << 3;
is set on the command, then you should use mgr_command. Using mon_command
will still generally work, as the mon will forward to the mgr on your
behalf (for compatibility with old CLI clients and lazy API users), but
best to use mgr_command based on FLAG_MGR.
Somewhat related to the previous item, is it safe to
"hard code" the json
structure or should we always prefer to dynmically fetch the parameters of the
commands via get_command_descriptions like the cli does? What are the risks
if a code base were to "hard code" the json for a command?
IMO it's generally safe to hard-code the arguments. We try to avoid
changing the structure of commands, and make notes of it if we do. And
in any case I'm not sure that fetching the command description will help
you automagically adapt to a change.
When I was testing the rados_mon_command_target
function, I noticed some
varying behavior when the server side was running binaries based on the
octopus release vs. client libraries from nautilus. When I use nautilus libs
to query the octopus-based server things seem to look like many of the other
results of "get_command_descriptions". However, using octopus client libs to
query the octopus-based server I see output that looks a bit different. The
json returned has extra white space and there are odd signatures like "0" and
"1". Example:
"cmd000": {
"sig": [
"0"
],
"help": ""
},
Poking around I saw that this might be related to the following items:
https://github.com/ceph/ceph/pull/30217
https://github.com/ceph/ceph/pull/30859
https://github.com/ceph/ceph/pull/31138
But I couldn't say for sure. The formatting of the JSON does not bother me but
the "0", "1", etc signatures don't look right and I'm
thinking that it might
be a bug but didn't want to file something without a hint that this isn't
desired behavior.
The 0, 1 commands are for compatibility with ancient (bobtail-era?) admin
socket commands. We can should probably hide them from the command
descriptions.
There is were a few subtle changes in behavior in octopus when the admin
socket and tell command sets were unified... this is probably what you're
seeing.
As a relative newcomer to ceph I'm doing more
research into how to use this
family of functions works (is expected to work) versus having absorbed
institutional knowledge. So, I'd be happy to record some of what I'm learning
somewhere more permanent for future folks in my current situation. If I were
to contribute documentation about these functions what's the preferred place
to do so? I could see this going into the comments in librados.h or somewhere
in the sphinx based docs but I'd appreciate a few pointers before I make
patches/PRs. FWIW, If there's already docs for it, I've managed to overlook it
despite looking a number of times for such a thing so apologies if it is
noise.
Lastly, and possibly leastly, does the team have a better name/label for this
collection of functions/general approach? I've been calling them *_command
functions or command json functions. I would be happy to more widely known
name for them if one exists. :-)
I was thinking the same thing myself. I tend to call them CLI commands,
but that's not very precise..
sage