On Wed, Nov 27, 2024 at 8:14 AM John Mulligan <phlogistonjohn@asynchrono.us> wrote:
On Wednesday, November 27, 2024 6:48:03 AM EST Neeraj Pratap Singh wrote:
Hey everyone,
Recently, we came across a ticket which mentions about xml format not working properly for some of ceph commands.
After looking into the code, I found that we are not following any standard to use the formats for the ceph commands across manager modules, and especially xml format is highly ignored. Though xml and xml-pretty is being accepted by 'src/ceph.in' , the ceph's command management tool.
Apart from this, each and every module have their own way of using the formats, some have hardcoded for json, plain etc. and rest in other way.
I too encountered a file named `object_format.py` which resides in `src/pybind/mgr/` but this also had been used by a few modules (mgr/nfs).
Thoughts?
Hi Neeraj, you're right that every ceph mgr module can do things independently with the format value. The C++ mgr modules might be a tad bit more consistent with XML handling but I don't know for sure. Certainly, many of the python modules are probably not handling xml at all. The `ceph` command line tool can send various format strings to the mgr module backend but the backend is free to ignore them.
The object_format.py code is currently in use by the nfs module and the smb module (in main). We have had discussions of moving the cephadm module to use it with some frequency but we have not done so yet. I'd love to see more modules adapt the object formatter approach. It is designed to try and separate concerns so that most python code in the mgr can be written in a typical python style and only a very limited set of functions (typically in the module class) need to be aware it's part of the mgr. However, I have not had the time or energy to convert "random" mgr module so it's up to the mgr module maintainers to adopt the object formatter style if desired. (Also, I'm sure there are cases it doesn't cover - I am happy to discuss design tweaks and features if you'd like to use it but it's not covering your use cases)
What are you looking at in terms of XML support? With XML vs JSON/YAML it's much more typical to have a pre-defined xml schema versus the free use of lists and maps. Therefore most of our objects can be automatically mapped to JSON/ YAML without much additional work. We could come up with our own schema or adopt something out there... or we could create some sort of JSON-but-in-XML, but I am unclear how much value that would be. I'm also curious if the existing JSON/YAML support is not covering a use case you have?
This isn't so much something Neeraj is having trouble with. The problem is: * Our CLI tools and documentation suggest that you can choose to get responses in plain text, JSON, or XML formats. * Our C++ implementations use a Formatter class that can output each of those (and there is some special handling for "plain text" v "programmatic" in places) * Our manager's Python implementations are a random grab bag and pretty much all support only plain text and JSON.[1] Our downstream QE filed a bug against mgr/volumes for not outputting XML when it was expected, but this is going to be true of just about every mgr module's commands if anybody gets around to running them. So from my perspective we need to make a choice. Either 1) We stop supporting XML, since we...don't actually support it 2) We start supporting XML robustly in the mgr. (2) probably means adopting something like our C++ Formatter infrastructure (presumably via object_format.py?) to make sure we're consistent about it. I'm down with this plan but we need to agree on it as a community. (1) has the advantage of not requiring work from every mgr component, and since lack of XML support hasn't come up before it doesn't seem likely that people are actually using it? -Greg [1]: This has also led to bugs, where one output schema provides different data than the other due to calling different variables!