Speaking abstractly, I can see 3 possible approaches. 1. You can create a separate container and invoke it from the mgr container as a micro-service. As to how, I don't know. This is likely the cleanest approach. 2. You can create a Dockerfile based on the stock mgr but with your extensions added. The main problem with this is that from what I can see, the cephadm tool has the names and repositories of the stock containers hard-wired in. Which ensures quality (getting the right versions) and integrity (makes it hard for a bad agent to swap in a malware module). So more information is needed at least. 3. You can inject your code into the stock container image by packaging it as an RPM, adding a local RPM repository to the stock container, and installing the extra code something like "docker exec -it [cephadmn- container-name] /usr/bin/dnf install mycode". The third option does require that the infrastructure to run dnf/yum hasn't been removed from the container image. Also not that if you're running a dynamic container launch, you might have to deal with having to re-install your code every time the container launches because there would be no persistent image.. However, option 3 would, if the stars are right, be something that Ansible could easily handle. As for testing, I'd look at the source for the mgr module and its regression tests. Plus of course testing your own code is something you'd have to do yourself. Tom On Wed, 2024-10-30 at 18:00 +0000, Darrell Enns wrote:

Is there a simple way to deploy a custom (in-house) mgr module to an orchestrator managed cluster? I assume the module code would need to be included in the mgr container image. However, there doesn't seem to be a straightforward way to do this without having the module merged to upstream ceph (not possible for a custom/in-house solution) or maintaining an in-house container repository and custom container images (a lot of extra maintenance overhead).

Also, what's the best way to handle testing during development? Custom scripts to push the code into to a mgr container in a dev cluster? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On Wednesday, October 30, 2024 2:00:56 PM EDT Darrell Enns wrote:

Is there a simple way to deploy a custom (in-house) mgr module to an orchestrator managed cluster? I assume the module code would need to be included in the mgr container image. However, there doesn't seem to be a straightforward way to do this without having the module merged to upstream ceph (not possible for a custom/in-house solution) or maintaining an in-house container repository and custom container images (a lot of extra maintenance overhead).

Also, what's the best way to handle testing during development? Custom scripts to push the code into to a mgr container in a dev cluster?

There's a tool in the ceph tree designed for this: src/script/cpatch.py (there's also an older shell based version in the same dir but that is not maintained WRT python changes as far as I know). There are some downsides to how this script works, as it creates many layers, but it's intended for development and in these cases having extra container layers is not usually a big deal. If you are working on a python based mgr module there's flags you can pass to have the script only create images with your local version of src/pybind/mgr (or whatnot, see the --help for more info). If you need extra help with the script, just ask here on the list. I use it frequently and have been maintaining it. When you build an image you can then push it to a private or public registry and configure ceph(adm) to use it. ( See https://docs.ceph.com/en/latest/ cephadm/install/#deployment-in-an-isolated-environment for some hints)