...

However, in practice, many operations (e.g., using ceph-bluestore-tool

Using that tool, to be fair, should be rare. Notably that tool requires that the OSD on which it operates not be running. I would think it might be possible to enter an OSD container and kill the ceph-osd process without killing the container so that the tool could be run there, but there might not be other processes in OSD containers so that may be a non-starter.

we do have a recommended process for this ( https://docs.ceph.com/en/latest/cephadm/troubleshooting/#running-various-cep...). The high level overview is you stop the daemon and then run `cephadm shell` with `--name <daemon-name>` and it should spin up a container with all the same files and mounts as if we were actually running the daemon, but with an interactive bash session inside instead of the actual daemon process running. I was just messing with this for trying to add a wal device to an OSD earlier today (which wasn't working for another reason related to ceph-volume, but the process for running the tools in general worked) [root@vm-00 ~]# [root@vm-00 ~]# systemctl stop ceph-50327e5e-3196-11f0-8285-52540034d386@osd.5.service [root@vm-00 ~]# [root@vm-00 ~]# [root@vm-00 ~]# cephadm shell --name osd.5 Inferring fsid 50327e5e-3196-11f0-8285-52540034d386 Inferring config /var/lib/ceph/50327e5e-3196-11f0-8285-52540034d386/osd.5/config Creating an OSD daemon form without an OSD FSID value [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5 bluefs-bdev-new-wal --dev-target /dev/vdf inferring bluefs devices from bluestore path WAL device added /dev/vdf [ceph: root@vm-00 /]# On Thu, May 15, 2025 at 12:14 PM Anthony D'Atri <aad@dreamsnake.net> wrote:

...

On May 15, 2025, at 1:22 AM, Florent Carli <fcarli@gmail.com> wrote:

Hello ceph team,

I’ve been working with ceph for some time

Wise choice!

...

1) Declarative workflows and Infrastructure as Code

One of the advantages of ceph-ansible was the ability to define the cluster state declaratively in YAML files, which aligned well with Infrastructure-as-Code principles.

Absolutely. If it isn’t in git, it’s hearsay.

...

With cephadm, the process appears more imperative and CLI-driven, which makes automation and reproducibility harder in comparison. Is there a recommended approach to achieving a fully declarative deployment model with cephadm? Or plans to support this more directly?

Once you have a cluster bootstrapped, it can be 100% declarative. There are various CLI commands so you can perform various tasks surgically, but it’s also entirely possible to maintain almost the entire cluster state in a YAML file:

ceph orch ls —export > myawesomecluster.yaml

# edit the file with your favorite emacs

ceph orch apply -i myawesomecluster.yaml —dry-run ceph orch apply -i myawesomecluster.yaml

This is the best of both worlds: the YAML file readily fits into revision control and peer review, and one can add commit hooks to validate syntax or to even perform a dry run for a sanity check.

There’s also a cephadm-ansible project out there that wraps some cephadm tasks in Ansible for playbook goodness.

...

2) Containerization vs. local dependencies

Cephadm’s move to full containerization makes sense in principle, especially to avoid system-level dependencies.

This is so, so, so nice. It also greatly facilitates the orchestrator’s ability to move daemons around.

...

However, in practice, many operations (e.g., using ceph-bluestore-tool

Using that tool, to be fair, should be rare. Notably that tool requires that the OSD on which it operates not be running. I would think it might be possible to enter an OSD container and kill the ceph-osd process without killing the container so that the tool could be run there, but there might not be other processes in OSD containers so that may be a non-starter.

...

or the python modules for Rados/rbd)

I’m not familiar with those.

...

3) Ceph packages for Debian Trixie on download.ceph.com

Since I'm using debian, I'm also in the process of anticipating the soon to come Debian 13 version (Trixie).

I can’t speak authoritatively for the build folks, but you do note that this is incipient. It’s not unusual for support for a new OS release to take some time for any given software, and for enterprises to let a new major OS release to bake / shakeout for a while before betting the farm on it. As David Lindley wrote, “Rasta soon come”.

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Once you have a cluster bootstrapped, it can be 100% declarative. There are various CLI commands so you can perform various tasks surgically, but it’s also entirely possible to maintain almost the entire cluster state in a YAML file:

ceph orch ls —export > myawesomecluster.yaml

# edit the file with your favorite emacs

ceph orch apply -i myawesomecluster.yaml —dry-run ceph orch apply -i myawesomecluster.yaml

This is the best of both worlds: the YAML file readily fits into revision control and peer review, and one can add commit hooks to validate syntax or to even perform a dry run for a sanity check.

Thanks, but this raises a lot of questions :) Running ceph orch ls --export gives me: service_type: mgr service_name: mgr placement: count: 3 --- service_type: mon service_name: mon placement: count: 3 --- service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore Very light isn't it? One use case that gave me a headache was losing a node (assume the hardware is completely dead). I was able to remove it from the cluster easily using: ceph orch host rm nodeX --force --offline After that, I want my playbook to detect that the cluster is already bootstrapped and then deploy everything needed on the replacement node—whichever node that may be. With ceph-ansible, I could simply run the site.yml playbook without giving it much thought. With cephadm, however, I feel like I have to handle all the logic manually. For instance, if the first node I try isn't in the cluster, I need to determine whether it's because the node was lost or because the cluster hasn't been bootstrapped yet. Then there’s the OSDs. The output of ceph orch ls doesn’t show where the OSDs are or what devices they're using. I also read somewhere that it's normal cephadm consideres them as "unmanaged". In my setup, I use pre-provisioned LVM volumes (e.g., vg_ceph/lv_ceph). My use case requires detecting whether all necessary OSDs are deployed. If not, I want to zap and redeploy the missing ones. I don’t see how ceph orch apply can do this. So, for now, I’m stuck handling all of this myself. You can take a look at what I’ve done here if you're interested: https://github.com/seapath/ansible/blob/main/roles/cephadm/tasks/main.yml My playbook deals with all those use cases based on my ansible inventory: - boostrapping a new cluster from scratch - adding a missing node (monitor) in an already boostrapped cluster - adding missing mgrs - zapping and adding missing OSDs (based on pre-provisionned LVM volumes) - of course: doing nothing if everything is already in place But it's such a pain to maintain.

It depends a bit on what you're doing with these modules. Are you using them interactively in a python prompt or are you building applications on top of them?

Indeed, I'm talking about a client application for tooling. My use case is a python script, that will do some stuff with ceph/rbd and some stuff with pacemaker and act as a wrapper for our admins. For this it using the ceph and pacemaker python modules directly on the system, which require the associated libraries.... which requires to install the associated packages: I need ceph-common because I need python3-ceph-common and librbd1, etc... to do my "import rbd".

Using that tool (ceph-bluestore-tool), to be fair, should be rare

I love this tool to extend an OSDs in seconds :) Once again, I run my OSDs on LVM and so LVM (lvextend) + ceph-bluestore-tool gives me a lot of flexibility with storage :)

we do have a recommended process for this (https://docs.ceph.com/en/latest/cephadm/troubleshooting/#running-various-cep...). The high level overview is you stop the daemon and then run `cephadm shell` with `--name <daemon-name>` and it should spin up a container with all the same files and mounts as if we were actually running the daemon, but with an interactive bash session inside instead of the actual daemon process running. I was just messing with this for trying to add a wal device to an OSD earlier today (which wasn't working for another reason related to ceph-volume, but the process for running the tools in general worked)

Very nice ! I had my own way of hacking this ("cephadm shell -v /dev:/dev -m /var/lib/ceph/e1887cc0-2988-11f0-a805-ead37406f331/osd.0/:/var/lib/ceph/osd/ceph-0/:z -- ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-0/"), but "--name osd.0" is so much nicer ! Thanks.

Containers are an excellent tool for developers and continuous testing in the development cycle but not a good solution for system tool deployment.

Could you please elaborate on the second part? Containers have succeeded not because they worked for developers, but because they solved issues throughout the entire software lifecycle, including deployment and maintenance.

An effort to provide native packages is therefore essential for enterprise ready deployment.

1. Ceph containers are built from distro packages, so containers and packages are not mutually exclusive. 2. That said, core distro packages are generally well tested and supported by major distro vendors such as Canonical or Red Hat. However, all distros also rely on repositories that are maintained by third parties with no guarantees (e.g. Ubuntu Universe, Fedora EPEL, ...). In some cases, these third parties are us, Ceph (and trust me, we're not such a large team). Whenever a bug or CVE is found, we have to drive ourselves all the distro package build process. Often these packages depend in turn on other packages not "maintained" by the Ceph team, which then involves some extra waits, and that slows down the whole process. Compared to that, a container native pipeline could release fixed images with almost zero latency and higher quality (because a vulnerable dependency might only compromise the container, not the whole system).

Regards --martin

Am 15.05.2025 10:22 schrieb Florent Carli <fcarli@gmail.com>:

2) Containerization vs. local dependencies

Cephadm’s move to full containerization makes sense in principle, especially to avoid system-level dependencies. However, in practice, many operations (e.g., using ceph-bluestore-tool, or the python modules for Rados/rbd) still seem to require installing packages on th Is this the expected model : containers for core daemons, but local packages for tooling ? It feels somewhat contradictory, and I wonder if there's a clearer pattern or guidance for those cases.

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io