cephadm without containers
Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine) If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way? I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm Thank you! Adrian
When you say cephadm, do you mean all the commands under “ceph orch” / enabling the cephadm mgr module, without deploying the containers? If so then I don’t think so. Cephadm assumes it is handling containers. I can go into more detail if this is the case. On Wed, Nov 19, 2025 at 10:18 PM Adrian Sevcenco <Adrian.Sevcenco@cern.ch> wrote:
Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine)
If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way?
I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm
Thank you! Adrian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 11/20/25 07:17, Adrian Sevcenco wrote:
Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine)
If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way?
I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm
Just my 0.02 euro: I'm not exactly a fan of containers for critical infrastructure, and adding another layer to an already complex system is not making your life easier. Our clusters are still running on bare metal and manual / ansible based deployment, but we will probably change this with Tentacle. There's a simple reason for it: choice. With a container based approach, it doesn't matter which linux distribution and which exact release you want to use. And the subtle differences between distribution, slightly varying library packages etc. makes the life of a developer really hard. This is were containers shine. You build them once with the distribution that fits your needs, and do not care about the deployment on the hosts. Container just works[tm]. Even within the red hat ecosystem you will find enough divergence that a single set of packages won't be sufficient, especially if you also have to support older releases. And speaking of older releases: these often to not provide the necessary recent libraries version you need for ceph. So containers also solve this problem. So you already answered your own question: container bvased deployment is probably the only best way to do it.....given the heterogeneous linux ecosystems. Best regards, Burkhard
On 11/20/25 7:03 AM, Burkhard Linke wrote:
Hi,
On 11/20/25 07:17, Adrian Sevcenco wrote:
Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine)
If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way?
I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm
Just my 0.02 euro:
I'm not exactly a fan of containers for critical infrastructure, and adding another layer to an already complex system is not making your life easier. Our clusters are still running on bare metal and manual / ansible based deployment, but we will probably change this with Tentacle.
There's a simple reason for it: choice.
With a container based approach, it doesn't matter which linux distribution and which exact release you want to use. And the subtle differences between distribution, slightly varying library packages etc. makes the life of a developer really hard. This is were containers shine. You build them once with the distribution that fits your needs, and do not care about the deployment on the hosts. Container just works[tm].
Thus displacing the problem from developers to the admins as now admins have to maintain the whole container support environment, including the "slightly varying library packages etc." This whole exercise is pointless as Ceph cluster storage nodes are natively dedicated to Ceph, thus the bare-metal machine is the real Ceph container. Just install the specific OS supported by Ceph and you've got your Ceph "container" made out of metal. You can make an argument that monitor and/or manager daemons could run in containers to allow for migration to alternative nodes. Properly sized Ceph clusters come with a number of dedicated monitor nodes anyway, therefore migration usually means add a new monitor node to the cluster and remove the old one, thus making the monitor containers point moot, too. Milan -- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing
Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine)
If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way?
I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm
I am sticking for now to bare metal. As you wrote, containers is an extra layer again and you need to be proficient in using containers. Docker is already out of the question because if that daemon dies all tasks are killed.
Podman FTW.
On Nov 20, 2025, at 8:48 AM, Marc <Marc@f1-outsourcing.eu> wrote:
Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine)
If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way?
I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm
I am sticking for now to bare metal. As you wrote, containers is an extra layer again and you need to be proficient in using containers. Docker is already out of the question because if that daemon dies all tasks are killed. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine) If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it
On Thursday, November 20, 2025 1:17:57 AM Eastern Standard Time Adrian Sevcenco wrote: therefore cephadm will only use it this way?
I ask this in the context of finding out that ceph-ansible is deprecated and
that it should not be used, but i still need to use ansible for
management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm Thank you! Adrian
This question comes up every so often and so I'll repeat the answer that was formulated previously in the orch weekly discussion as best as I can remember it: Using containers is one of the fundamental building blocks of cephadm and it would be a very large, potentially disruptive, change to alter that. However, Ceph already has some level of support of alternative orch backends. Cephadm is probably the most often used and it's prominence impacts the shape of the API (IMO), but it is possible to add more. One could create a `cephpkgadm` if one wanted to. That said, the majority of the people actively contributing to Cephadm prefer the container based approach and are not actively seeking to make this alternative happen. If it is something folks want strongly enough they will have to contribute the majority of code. Existing orch contributors are willing to provide some advice and guidance but can not stop working on cephadm and related items to help with this new backend. Common APIs & code could be eventually developed and shared between cephadm and this new back- end, but a viable prototype would probably need to be demonstrated first. To also briefly touch on your point WRT ansible: there is already a cephadm- ansible project (see https://docs.ceph.com/projects/cephadm-ansible/en/latest/ index.html ). Personally, I also drive my VM setups using ansible and cephadm together (but I'm just a mere dev and all my clusters are ephemeral :-)).
This topic has been controversial in the past, but ...
(so to use bare-metal, with the packages directly installed on the machine)
Often called “package-based”. When I started wit Dumpling, there was no upstream orchestrator, one reason that various adopters and distributions rolled there own. When cephadm / ceph orch first came out, it was a bit rough, like any newly-hatched system. I’d never touched a container before, so I personally clung to package-based deployments. Both have since changed.
If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way?
There is a nontrivial segment of our community who run package-based installs for various reasons, one of which is existing homebrew orchestration. Certain distributions also have their own orchestration: Proxmox AIUI uses package-based deployments. The key thing to know here is that cephadm is a tool, but not the critical path. Our nomenclature and usage can be a bit confusing here, we often conflate “cephadm” and the “ceph orch” orchestrator.
I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used,
I hear conflicting information re ceph-ansible, but my sense is that usage of and support for it going forward is at most less than it once was.
but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm
If you had difficulty with a cephadm / ceph orch deployment, it may be that your process wasn’t getting something right. It would not be uncommon for an Ansible playbook to take care of setting up SSH keys etc. before an orchestrated deployment. That said, an example of Ansible code using cephadm / ceph orch is vexxhost/atmosphere on GitHub. I would not necessarily recommend adopting / adapting this for Ceph deployment, but it’s a useful example. Ansible’s command and shell modules let you invoke any CLI you like and act based on the return code. Even the asynchronous nature of certain operations is straightforward to accommodate with looping (and I have an RFE in to add a —synchronous switch).
Using containers is one of the fundamental building blocks of cephadm and it would be a very large, potentially disruptive, change to alter that. However, Ceph already has some level of support of alternative orch backends.
My sense is that Rook is an example.
To also briefly touch on your point WRT ansible: there is already a cephadm- ansible project (see https://docs.ceph.com/projects/cephadm-ansible/en/latest/ index.html ). Personally, I also drive my VM setups using ansible and cephadm together (but I'm just a mere dev and all my clusters are ephemeral :-)).
cephadm-ansible is AIUI a collection of utilities / glue that enable just what you’re after: managing the orchestrator with Ansible.
Hi Adrian, just want to add some other facts to cephadm. I started with cephadm together with SUSE around 2017. At this time together with Sage the idea was to develop an orchestrator for ceph that can manage ceph packages and containers. Around 2019 the decision was made to remove this middle layer and only focus on container orchestration. When Red Hat stopped Ceph-Ansible's development and support, there was a long mailing thread here, and no one volunteered to officially continue it. Therefore, currently only Cephadm and Rook are officially supported open-source Ceph orchestrators. Note: I happen to be at the KCD conference at CERN in December, if you have any more questions. Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Do., 20. Nov. 2025 um 07:18 Uhr schrieb Adrian Sevcenco < Adrian.Sevcenco@cern.ch>:
Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine)
If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way?
I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm
Thank you! Adrian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, Nov 20, 2025 at 1:20 AM Adrian Sevcenco <Adrian.Sevcenco@cern.ch> wrote:
Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine)
Package based Ceph deployments, while popular, are not a good choice in general. The very simple reason is that it makes upgrades more dangerous: you can unintentionally upgrade services in the wrong order due to failovers. Furthermore, it's harder to ensure dependencies are aligned with what we test upstream. I'd be interested to hear ways in which we could make containers more attractive (developer tooling, lower overhead, etc.) rather than ways to return to a fundamentally flawed package based orchestration framework. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Package based Ceph deployments, while popular, are not a good choice in general. The very simple reason is that it makes upgrades more dangerous: you can unintentionally upgrade services in the wrong order due to failovers.
Or when a node crashes / reboots during an upgrade. This has happened to me.
I'd be interested to hear ways in which we could make containers more attractive (developer tooling, lower overhead, etc.) rather than ways to return to a fundamentally flawed package based orchestration framework.
^ This
Package based Ceph deployments, while popular, are not a good choice in general. The very simple reason is that it makes upgrades more dangerous: you can unintentionally upgrade services in the wrong order due to failovers.
Or when a node crashes / reboots during an upgrade. This has happened to me.
Hmmm, that is not really nice to read that ceph is so picky that everything can go wrong with just a minor update on a single node. I am not sure if that is a good direction for development. I would expect ceph to be more robust.
On Nov 21, 2025, at 9:55 AM, Marc <Marc@f1-outsourcing.eu> wrote:
Package based Ceph deployments, while popular, are not a good choice in general. The very simple reason is that it makes upgrades more dangerous: you can unintentionally upgrade services in the wrong order due to failovers.
Or when a node crashes / reboots during an upgrade. This has happened to me.
Hmmm, that is not really nice to read that ceph is so picky that everything can go wrong with just a minor update on a single node.
Then if you really want to stick with package installs, deploy every service on a separate node. Your DC runneth over.
I am not sure if that is a good direction for development.
Hence the increasing motivations for container installations, which are by nature immune from this dynamic.
I would expect ceph to be more robust.
It's not a function of Ceph, but robustness is exactly the goal.
Package based Ceph deployments, while popular, are not a good choice in general. The very simple reason is that it makes upgrades more dangerous: you can unintentionally upgrade services in the wrong order due to failovers.
Or when a node crashes / reboots during an upgrade. This has happened to me.
Hmmm, that is not really nice to read that ceph is so picky that everything can go wrong with just a minor update on a single node.
Then if you really want to stick with package installs, deploy every service on a separate node. Your DC runneth over.
I am not sure if that is a good direction for development.
Hence the increasing motivations for container installations, which are by nature immune from this dynamic.
I don't know about that, you just move the issue from ceph daemons to container daemons. If I remember correctly, I even read something here on the list about podman version problems there. And since a lot is going still via hosts configs not volumes assigned to the task, changes between the host and the task image could also complicate things. Didn't I read recently something with lvm.conf or so?
I would expect ceph to be more robust.
It's not a function of Ceph, but robustness is exactly the goal.
So what is then all this nonsense of having a minor upgrade on a node? I have never seen any issues with apache httpd, mariadb, postgres etc. Nor with ceph for that matter.
Hence the increasing motivations for container installations, which are by nature immune from this dynamic.
I don't know about that, you just move the issue from ceph daemons to container daemons.
Daemons have individual container images that are updated individually, which obviates the issue.
If I remember correctly, I even read something here on the list about podman version problems there.
With recent OS, Ceph, and Podman releases this is moot.
And since a lot is going still via hosts configs not volumes assigned to the task, changes between the host and the task image could also complicate things. Didn't I read recently something with lvm.conf or so?
I would expect ceph to be more robust.
It's not a function of Ceph, but robustness is exactly the goal.
So what is then all this nonsense of having a minor upgrade on a node? I have never seen any issues with apache httpd, mariadb, postgres etc. Nor with ceph for that matter.
ymmv.
Thank you so much everyone for helping me view and understand the bigger picture. I do realize (or did realize) that the container model AND cephadm have a goal to both isolate and remove any kind of middle layer between developers (or developing party) and any other build process/repository, system dependencies or other 3rd party tools used for configuration (as ansible is) This create a short circuit from developers to end users which of course increase stability through fast response and independence of underlying environment .. So, understanding this, now i can ask about potential improvements to cephadm .. 1. first of my pain problem is the assumption of lacking of ssh communication between hosts. while there is --skip-ssh for bootstrap, it would be so much easier if there could be some environment variables that could steer and configure the behavior of the whole cluster without having to remember what i choose for the cluster. it would be really useful if all bootstrap options could be automatically read from CEPHENV_<name of option without -- and s/-/_/> so i.e. --skip-ssh would firstly be read/checked for as CEPHENV_skip_ssh this would allow to automatically "seed" the main parameters of cluster to all involved nodes with something like ansible :) there are some other global options that are usually managed by other tools and have nothing to do with cephadm like --skip-firewalld 2. the second pain point is that the cephadm does not act remotely .. it could be really useful to add a --host argument and to act through ssh on the remote host 3. documentation : explanation of arguments is missing for example for cephadm deploy --name NAME where NAME should be type.id i get the type which maybe i can get a list, but what is id? an index, number? if it's an index, cannot cephadm enumerate it automatically and just put it? i.e i want to deploy alert mananager that would be the 6th service in cluster, so automatically the name could be alertmanager.6 ? --config and --config-json can be inferred that they are expecting a filepath but what about --keyring and --key ? some kind of example and structure of names should be shown or linked what content/information is present in the name of the key? is it arbitrary? Thank you for help and info!! Adrian -------- Original Message -------- Subject: [ceph-users] cephadm without containers From: Anthony D'Atri To: Marc Date: 11/21/2025, 5:03:27 PM
On Nov 21, 2025, at 9:55 AM, Marc <Marc@f1-outsourcing.eu> wrote:
Package based Ceph deployments, while popular, are not a good choice in general. The very simple reason is that it makes upgrades more dangerous: you can unintentionally upgrade services in the wrong order due to failovers.
Or when a node crashes / reboots during an upgrade. This has happened to me.
Hmmm, that is not really nice to read that ceph is so picky that everything can go wrong with just a minor update on a single node.
Then if you really want to stick with package installs, deploy every service on a separate node. Your DC runneth over.
I am not sure if that is a good direction for development.
Hence the increasing motivations for container installations, which are by nature immune from this dynamic.
I would expect ceph to be more robust.
It's not a function of Ceph, but robustness is exactly the goal.
On Friday, November 21, 2025 11:01:36 AM Eastern Standard Time Adrian Sevcenco wrote:
Thank you so much everyone for helping me view and understand the bigger picture. I do realize (or did realize) that the container model AND cephadm have a goal to both isolate and remove any kind of middle layer between developers (or developing party) and any other build process/repository, system dependencies or other 3rd party tools used for configuration (as ansible is) This create a short circuit from developers to end users which of course increase stability through fast response and independence of underlying environment .. So, understanding this, now i can ask about potential improvements to cephadm ..
1. first of my pain problem is the assumption of lacking of ssh communication between hosts. while there is --skip-ssh for bootstrap, it would be so much easier if there could be some environment variables that could steer and configure the behavior of the whole cluster without having to remember what i choose for the cluster. it would be really useful if all bootstrap options could be automatically read from CEPHENV_<name of option without -- and s/-/_/> so i.e. --skip-ssh would firstly be read/checked for as CEPHENV_skip_ssh this would allow to automatically "seed" the main parameters of cluster to all involved nodes with something like ansible :)
there are some other global options that are usually managed by other tools and have nothing to do with cephadm like --skip-firewalld
There *are* a lot of options to bootstrap. A quick and dirty grep counts around 47 options that are unique to bootstrap (it didn't count global options). However, environment variables strike me as a bit unwieldy. What would your opinion of a configuration file be instead? Are there benefits to environment variables over a configuration file that I'm overlooking?
2. the second pain point is that the cephadm does not act remotely .. it could be really useful to add a --host argument and to act through ssh on the remote host
The cephadm binary is a bit stretched now (IMO). It acts as both something users are expected to interact with as well as the backend for lower level deployment operations. I think it makes it a bit harder for it to act as a general administration tool. For now I suggest just sticking to `ssh user@host cephadm ...` and/or tools like ansible to invoke the cephadm version installed on the particular host. For tasks that need just the ceph command, I have a toy (protoype) tool that you might find interesting: https://github.com/ceph/ceph/pull/66061 It allows the use of (the equivalent of) cephadm shell outside of the ceph cluster admin nodes. If I hear people are interested in this, it will motivate me to work on it more :-) Do note that it's so much of a prototype you do have to check out a branch and build it yourself if you want to try it in action, but the PR should describe what it can do sufficiently (I hope).
3. documentation : explanation of arguments is missing for example for cephadm deploy --name NAME where NAME should be type.id i get the type which maybe i can get a list, but what is id? an index, number? if it's an index, cannot cephadm enumerate it automatically and just put it? i.e i want to deploy alert mananager that would be the 6th service in cluster, so automatically the name could be alertmanager.6 ?
--config and --config-json can be inferred that they are expecting a filepath but what about --keyring and --key ? some kind of example and structure of names should be shown or linked what content/information is present in the name of the key? is it arbitrary?
Yes there a a lot of help texts that can be improved. Please consider filing individual tracker tickets for these items. Putting them in the tracker makes it easier to find in the future and adding them individually makes it easy for leads to assign them as tasks for Jr devs (good first issue type stuff), for example.
-------- Original Message -------- Subject: [ceph-users] Re: cephadm without containers From: John Mulligan To: ceph-users Date: 11/21/2025, 8:27:43 PM Hi!
There *are* a lot of options to bootstrap. A quick and dirty grep counts around 47 options that are unique to bootstrap (it didn't count global options). However, environment variables strike me as a bit unwieldy. What would your opinion of a configuration file be instead? Are there benefits to environment variables over a configuration file that I'm overlooking? well, easy of use, and easy knowledge intake. e.g --mon-addrv as env var would be enough to have an hypothetical CEPHENV_MON_ADDRV with the clear information of : put as value the list of host:port monitors split-ed by comma as configuration file: where should be this configuration placed? and with what name? what content format does it have? can it be modular? (to enable configuration by dropped-in files)
So, basically it boils down to the easy knowledge to do something .. i'm struggling to understand what get configured where and the only tip that i found so far is https://docs.ceph.com/en/squid/dev/config-key/#configuration I did not find so far a template configuration with content explained and with an example format. i found also https://docs.ceph.com/en/squid/cephadm/services/#service-specification which seems to indicate that for configuration of a service one need to create a temporary yml and then apply it and then https://docs.ceph.com/en/squid/cephadm/services/#retrieving-the-running-serv... show an easy way to get current configuration and then apply it but i'm not sure how can this be used for a client configuration.
2. the second pain point is that the cephadm does not act remotely .. it could be really useful to add a --host argument and to act through ssh on the remote host
The cephadm binary is a bit stretched now (IMO). It acts as both something users are expected to interact with as well as the backend for lower level deployment operations. I think it makes it a bit harder for it to act as a general administration tool.
For now I suggest just sticking to `ssh user@host cephadm ...` and/or tools like ansible to invoke the cephadm version installed on the particular host. i see, got it, i can template something for this ..
For tasks that need just the ceph command, I have a toy (protoype) tool that you might find interesting: https://github.com/ceph/ceph/pull/66061 It allows the use of (the equivalent of) cephadm shell outside of the ceph cluster admin nodes. If I hear people are interested in this, it will motivate me to work on it more :-) Do note that it's so much of a prototype you do have to check out a branch and build it yourself if you want to try it in action, but the PR should describe what it can do sufficiently (I hope). Nice! yeah, it actually check what i felt the need for!!! i will try it out and let you know how it feels or any other feedback Thanks a lot!!
3. documentation : explanation of arguments is missing for example for cephadm deploy --name NAME where NAME should be type.id i get the type which maybe i can get a list, but what is id? an index, number? if it's an index, cannot cephadm enumerate it automatically and just put it? i.e i want to deploy alert mananager that would be the 6th service in cluster, so automatically the name could be alertmanager.6 ?
--config and --config-json can be inferred that they are expecting a filepath but what about --keyring and --key ? some kind of example and structure of names should be shown or linked what content/information is present in the name of the key? is it arbitrary?
Yes there a a lot of help texts that can be improved. Please consider filing individual tracker tickets for these items. Putting them in the tracker makes it easier to find in the future and adding them individually makes it easy for leads to assign them as tasks for Jr devs (good first issue type stuff), for example. tracker tickets mean this? https://github.com/ceph/ceph.io/issues ?
Thanks a lot! Adrian
Not sure what exact issue you have with deploying services (alertmanager in the example). But choosing which service to deploy, how many instances and where should be really easy by using specs: https://docs.ceph.com/en/latest/cephadm/services/ https://docs.ceph.com/en/latest/cephadm/services/monitoring/ In general it's as easy as calling the following command (which will use the default placement):
ceph orch apply my-service
Or use any of the supported placement options: https://docs.ceph.com/en/latest/cephadm/services/#daemon-placement Anyway, in general If you have suggestions for improvements/features I'd recommend opening a ticket on the below tracker providing as much information as possible: https://tracker.ceph.com/projects/orchestrator/issues/new The Cephadm team continuously reviews the list and tries to implement any features that would improve the overall user experience. Best, Redouane. On Fri, Nov 21, 2025 at 9:30 PM Adrian Sevcenco <Adrian.Sevcenco@cern.ch> wrote:
-------- Original Message -------- Subject: [ceph-users] Re: cephadm without containers From: John Mulligan To: ceph-users Date: 11/21/2025, 8:27:43 PM
Hi!
There *are* a lot of options to bootstrap. A quick and dirty grep counts around 47 options that are unique to bootstrap (it didn't count global options). However, environment variables strike me as a bit unwieldy. What would your opinion of a configuration file be instead? Are there benefits to environment variables over a configuration file that I'm overlooking? well, easy of use, and easy knowledge intake. e.g --mon-addrv as env var would be enough to have an hypothetical CEPHENV_MON_ADDRV with the clear information of : put as value the list of host:port monitors split-ed by comma as configuration file: where should be this configuration placed? and with what name? what content format does it have? can it be modular? (to enable configuration by dropped-in files)
So, basically it boils down to the easy knowledge to do something .. i'm struggling to understand what get configured where and the only tip that i found so far is https://docs.ceph.com/en/squid/dev/config-key/#configuration
I did not find so far a template configuration with content explained and with an example format.
i found also https://docs.ceph.com/en/squid/cephadm/services/#service-specification which seems to indicate that for configuration of a service one need to create a temporary yml and then apply it and then https://docs.ceph.com/en/squid/cephadm/services/#retrieving-the-running-serv... show an easy way to get current configuration and then apply it but i'm not sure how can this be used for a client configuration.
2. the second pain point is that the cephadm does not act remotely .. it could be really useful to add a --host argument and to act through ssh on the remote host
The cephadm binary is a bit stretched now (IMO). It acts as both something users are expected to interact with as well as the backend for lower level deployment operations. I think it makes it a bit harder for it to act as a general administration tool.
For now I suggest just sticking to `ssh user@host cephadm ...` and/or tools like ansible to invoke the cephadm version installed on the particular host. i see, got it, i can template something for this ..
For tasks that need just the ceph command, I have a toy (protoype) tool that you might find interesting: https://github.com/ceph/ceph/pull/66061 It allows the use of (the equivalent of) cephadm shell outside of the ceph cluster admin nodes. If I hear people are interested in this, it will motivate me to work on it more :-) Do note that it's so much of a prototype you do have to check out a branch and build it yourself if you want to try it in action, but the PR should describe what it can do sufficiently (I hope). Nice! yeah, it actually check what i felt the need for!!! i will try it out and let you know how it feels or any other feedback Thanks a lot!!
3. documentation : explanation of arguments is missing for example for cephadm deploy --name NAME where NAME should be type.id i get the type which maybe i can get a list, but what is id? an index, number? if it's an index, cannot cephadm enumerate it automatically and just put it? i.e i want to deploy alert mananager that would be the 6th service in cluster, so automatically the name could be alertmanager.6 ?
--config and --config-json can be inferred that they are expecting a filepath but what about --keyring and --key ? some kind of example and structure of names should be shown or linked what content/information is present in the name of the key? is it arbitrary?
Yes there a a lot of help texts that can be improved. Please consider filing individual tracker tickets for these items. Putting them in the tracker makes it easier to find in the future and adding them individually makes it easy for leads to assign them as tasks for Jr devs (good first issue type stuff), for example. tracker tickets mean this? https://github.com/ceph/ceph.io/issues ?
Thanks a lot! Adrian
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Nov 21, 2025 at 5:07 PM Adrian Sevcenco <Adrian.Sevcenco@cern.ch> wrote:
Thank you so much everyone for helping me view and understand the bigger picture. I do realize (or did realize) that the container model AND cephadm have a goal to both isolate and remove any kind of middle layer between developers (or developing party) and any other build process/repository, system dependencies or other 3rd party tools used for configuration (as ansible is) This create a short circuit from developers to end users which of course increase stability through fast response and independence of underlying environment ..
So, understanding this, now i can ask about potential improvements to cephadm ..
1. first of my pain problem is the assumption of lacking of ssh communication between hosts. while there is --skip-ssh for bootstrap, it would be so much easier if there could be some environment variables that could steer and configure the behavior of the whole cluster without having to remember what i choose for the cluster. it would be really useful if all bootstrap options could be automatically read from CEPHENV_<name of option without -- and s/-/_/> so i.e. --skip-ssh would firstly be read/checked for as CEPHENV_skip_ssh this would allow to automatically "seed" the main parameters of cluster to all involved nodes with something like ansible :)
there are some other global options that are usually managed by other tools and have nothing to do with cephadm like --skip-firewalld
[Redo] A John has indicated, as we have so many options, maybe going with a config file here (which you can then build with your favourite config tool) can be more helpful.
2. the second pain point is that the cephadm does not act remotely .. it could be really useful to add a --host argument and to act through ssh on the remote host
3. documentation : explanation of arguments is missing for example for cephadm deploy --name NAME where NAME should be type.id i get the type which maybe i can get a list, but what is id? an index, number? if it's an index, cannot cephadm enumerate it automatically and just put it? i.e i want to deploy alert mananager that would be the 6th service in cluster, so automatically the name could be alertmanager.6 ?
[Redo] I'll try to answer 2) and 3). For 2 the answer is that in general
the user doesn't have to ssh to the hosts to do orchestration. That's something that cephadm orchestrator takes care of automatically. Please correct me if I'm wrong, but I think there's some confusion going on here. In fact, cephadm architecture has two different layers and different components: 1) User-facing layer - The user runs cephadm bootstrap once to create the first cluster. - After that you mostly interact with the cluster via "ceph orch commands (or the dashboard)" --> normally this run by the cephadm mgr-module 2) Internal / orchestrator layer (this normally is executed by the cephadm binary) The mgr’s cephadm orchestrator module calls cephadm on hosts to do the real work: such as cephadm deploy, cephadm rm-daemon, etc.These commands are primarily intended for the orchestrator, not for day-to-day human. So normally, once the user bootstrapped the cluster (and added all the nodes), he doesn't need to ssh into hosts anymore. He just needs to have a copy of the config conf and key (stored in /var/lib/ceph/<fsid>/config/ during the bootstrap) and then run the cephadm shell as:
cephadm shell --fsid <fsid> -c my-ceph.conf -k my-keyring
If the ceph.conf or the keyring exists on the localhost, or the monitor is running, cephadm will infer automatically the config) so you just needs to run
cephadm shell
This will create a shell from where you can run "ceph orch" commands, and any other ceph command. The orch commands should be good enough to orchestrate the whole cluster. There's no need to ssh into any host. Docs: https://docs.ceph.com/en/latest/cephadm/ https://docs.ceph.com/en/latest/cephadm/services/ Plz, let me know if this helps to answer your questions, otherwise just let me know which operations you can't run following the above procedure. Best, Redo.
--config and --config-json can be inferred that they are expecting a filepath but what about --keyring and --key ? some kind of example and structure of names should be shown or linked what content/information is present in the name of the key? is it arbitrary?
Thank you for help and info!! Adrian
-------- Original Message -------- Subject: [ceph-users] cephadm without containers From: Anthony D'Atri To: Marc Date: 11/21/2025, 5:03:27 PM
On Nov 21, 2025, at 9:55 AM, Marc <Marc@f1-outsourcing.eu> wrote:
Package based Ceph deployments, while popular, are not a good choice in general. The very simple reason is that it makes upgrades more dangerous: you can unintentionally upgrade services in the wrong order due to failovers.
Or when a node crashes / reboots during an upgrade. This has happened to me.
Hmmm, that is not really nice to read that ceph is so picky that
everything can go wrong with just a minor update on a single node.
Then if you really want to stick with package installs, deploy every
service on a separate node. Your DC runneth over.
I am not sure if that is a good direction for development.
Hence the increasing motivations for container installations, which are
by nature immune from this dynamic.
I would expect ceph to be more robust.
It's not a function of Ceph, but robustness is exactly the goal.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (11)
-
Adrian Sevcenco
-
Anthony D'Atri
-
Anthony D'Atri
-
Burkhard Linke
-
Joachim Kraftmayer
-
John Mulligan
-
Kirby Haze
-
Marc
-
Milan Kupcevic
-
Patrick Donnelly
-
Redouane Kachach