Den fre 30 aug. 2024 kl 20:43 skrev Milan Kupcevic <milan_kupcevic@harvard.edu>:

On 8/30/24 12:38, Tim Holloway wrote:

...

I believe that the original Ansible installation process is deprecated.

This would be a bad news as I repeatedly hear from admins running large storage deployments that they prefer to stay away from containers.

You have other choices than Ansible or containers. It has always been possible to install using rpm/deb's manually, using any kind of scripts or frameworks. The point is that the ceph people no longer tries to make pre-made ansible scripts that work "everywhere", because they didn't. This does not prevent you in any way from avoiding containers (if that is what you want) but it makes you responsible for figuring out the automation part if you need one. -- May the most significant bit of your life be positive.

Den lör 31 aug. 2024 kl 15:42 skrev Tim Holloway <timh@mousetech.com>:

I would greatly like to know what the rationale is for avoiding containers.

Especially in large shops. From what I can tell, you need to use the containerized Ceph if you want to run multiple Ceph filesystems on a single host. The legacy installations only support dumping everything directly under /var/lib/ceph, so you'd have to invest a lot of effort into installing, maintaining and operating a second fsid under the legacy architecture.

Using two fsids on one machine is far outside our scope for the 10-or-so clusters we run. Not saying no one does it, but it was frowned upon to have multiple clusternames on the same host, so I guess most people took that to also include multiple fsids running in parallel on the same host, even if the cluster name was the same.

The only definite argument I've ever heard in my insular world against containers was based on security. Yet the primary security issues seemed to be more because people were pulling insecure containers from Docker repositories. I'd expect Ceph to have safeguards. Plus Ceph under RHEL 9 (and 8?) will run entirely and preferably under Podman, which allegedly is more secure, and can in fact, run containers under user accounts to allow additional security. I do that myself, although I think the mechanisms could stand some extra polishing.

From what I see on irc and the maillists, the container setup seems to sometimes end up recreating new containers with new/unique fsids at times as if it forgot the old cluster and rather decided to invent a new one. This seems to combine well (sarcastically) with those ceph admins inability to easily enter the containers and/or read out logs from the old/missing containers to figure out what happened and why this new mon container wants to reinvent the cluster instead of joining the existing one. I know bugs are bugs, but wrapping it all into an extra layer is not helping new ceph admins when it breaks. We have a decent page on PG repairs of various kinds, but perhaps not as much on what to do when the orchestrator isn't orchestrating? Containers help them set the initial cluster up tons faster, but it seems as if it leads them into situations where the container's ephemeral state is actively working against their ability to figure out when things go wrong and what the actual cause for that was. Perhaps it is clusters that were adopted into the new style, perhaps they run the containers in the wrong way, but there are a certain amount of posts about "I pressed the button for totally automated (re)deploy of X,Y and Z and it doesn't work". I would not like to end up in this situation while at the same time handling real customers who wonder why our storage is not serving IO at this moment. Doing installs 'manually' is far from optimal, but at least I know the logs end up under /var/log/ceph/<clustername>-<daemon><instance>.log and they stay there even if the OSD disk is totally dead and gone. -- May the most significant bit of your life be positive.

Those are reasonable objections, although some are now dated. In the context of Ceph some of those issues are also further addressed by Ceph itself. So let me present my take. 1. Networking. You can set up some gnarly virtual networks in both container and cloud systems, it's true. Docker has changed some of its original rules as well but more on that in a bit. In my non-ceph containers I've simply used host networking, which has its drawbacks but it's simple and it's all I require. 2. Logs in containerized ceph almost all go straight to the system journal. Specialized subsystems such as Prometheus can be configured in other ways, but everything's filed under /var/lib/ceph/<fsid>/<subsystem> so there's relatively little confusion. 3. I don't understand this, as I never stop all services just to play with firewalls. RHEL 8+ support firewall-cmd and you can can open ceph up with --add-service ceph, -add-service ceph-mon. Make them --permanent and do a --reload and it's all done. 4. Ceph knows exactly the names and locations of its containers (NOTE: a "package" is NOT a "container"). Within Ceph, almost all services actually employ the same container, just with different invocation options. You don't talk to "Docker*" directly, though, as systemd handles that. 5. I've never encountered this, so I can say nothing. But I run containers for about 6 different build base systems 24x7. 6. As I said, Ceph does almost everything via cephadm or ceph orch when running in containers, which actually means you need to learn less. Administration of ceph itself, is, again, done via systemd. *Docker. As I've said elsewhere, Red Hat prefers Podman to Docker these days and even if you install Docker, there's a Podman transparency feature. Now if you really want networking headaches, run Podman containers rootless. I've learned how to account for the differences but Ceph, fortunately hasn't gone that route so far. Nor have they instituted private networks for Ceph internal controls. On 9/1/24 15:54, Anthony D'Atri wrote:

* Docker networking is a hassle * Not always clear how to get logs * Not being able to update iptables without stopping all services * Docker package management when the name changes at random * Docker core leaks and kernel compatibility * When someone isn’t already using containers, or has their own orchestration, going to containers steepens the learning curve.

Containers have advantages including decoupling the applications from the underlying OS

...

I would greatly like to know what the rationale is for avoiding containers

Sorry if that sounds trollish. It wasn't intended to be. Look at it this way. There are two approaches to running an IT installation. One is the free-wheeling idependent aproach. The other is the stuffy corporate approach. Free-wheeling shops run things like Ubuntu. Or even BSD (but that's another matter). They often roll their own solutions. Frequently they were set up by a resident genius who liked some particular platform. And more power to them. Whether they do or don't buy into containerization is basically going to be determined on whether their guiding genius(es) take an interest in such things. They can ignore the points I just made. Said points were aimed at the stuffy business crowd. All that remains to be said on containers for that topic is that as far as I know Ceph never promoted setting up containerization manually. Now Ubuntu tries very hard to support the Red Hat Enterprise products, but there are certain quirks to the Red Hat world that make it extra challenging. Literally from the very start. Kickstart and preseed are not directly compatible, and in general, the RPM-based distros have one crucial difference from the Debian world in that at no time does an RPM ever halt for interactive input. So the non-Red Hat folks do have to work a little harder. Ceph, I believe does have some serious Ubuntu support, but at its core it is a Red Hat/IBM product. And IBM Red Hat thus has significant control over its shaping and direction. So, for the stuffy business crowd. Banks, insurance companies, Fortune corporations, as opposed to public institutions (where they're more likely to be free-wheeling). These organizations generally pay for Red Hat contracts (bless them, Red Hat stock paid for my wife's retirement). They want the "neck to choke", they want someone to cry to (besides this forum!). They want access to the restricted Red Hat Q&A. They're generally going to install RHEL or at a minimum a CentOS-style equivalent. And they're going to go with what their paid support tells them to do. As of Ceph Octopus, the older various methods of deploying Ceph have apparently been deprecated in favor of cephadm, and I expect that when you call for support, they're going to be hoping that's what your using. The cephadm utility collapses a LOT of previous manual work down into a central control program. It also (as far as I know, has no support for the legacy Ceph structure, being entirely based on "administered" (e.g. containerized) servers. Rather like virsh handles almost everything for KVM-based VMs, cephadm is the only program you have to install to ceph-enable a real or virtual machine. The fact that "administered" daemons are container-based is actually incidental, as from the Ceph administrator's point of view, the daemons are black boxes. It does not support roll-your-own Ceph containers; it uses its own and everything is controlled either via cephadm or systemd. Being a container expert is neither required nor expected. And could even potentially get you into trouble (not that I'd ever have had that problem...). The more critical point is that ceph has abandoned older install/maintenance processes. Ansible is no longer supported, and as I said, even though the Octopus docs give examples using ceph-install, that utility is not available from the repository install of Octopus. So at this point, it appears to be a choice between keeping legacy structure and doing everything the hard (manual) way or using cephadm. There is a serious possibility that eventually that choice may no longer exist. Aside from making administration simpler overall, if you want to run multiple fsids on a single machine, the only simple way to do that is via cephadm. Incidentally, the Ceph config does still remain in /etc/ceph and is essentially unchanged, allowing for the items that now get stored in object store instead of in text configs. A final note. The cephadm utility also can convert legacy resources into administered resources with a single command, thus easing migration. You can have both legacy and administered resources concurrently on the same machine so migrations can be gradual and outages are minimized. Ultimately, the choice is yours, for now. But it wouldn't be a bad idea to set up a test cephadm-based system just to get familiar with the concept. Avoid Octopus, though. It had some serious teething issues, as I found to my pain.   Tim On 9/2/24 11:41, Anthony D'Atri wrote:

I should know to not feed the trolls, but here goes. I was answering a question asked to the list, not arguing for or against containers.

...

2. Logs in containerized ceph almost all go straight to the system journal. Specialized subsystems such as Prometheus can be configured in other ways, but everything's filed under /var/lib/ceph/<fsid>/<subsystem> so there's relatively little confusion. I see various other paths, which often aren’t /var/log. And don’t conflate “containerized Ceph” with “cephadm”. There are lots of containerized deployments that don’t use cephadm / ceph orch.

...

3. I don't understand this, as I never stop all services just to play with firewalls. RHEL 8+ support firewall-cmd Lots of people don’t run RHEL, and I did wrote “iptables”, not whatever obscure firewall system RHEL also happens to ship.

...

4. Ceph knows exactly the names and locations of its containers Sometimes. See above.

...

(NOTE: a "package" is NOT a "container") Nobody claimed otherwise.

...

You don't talk to "Docker*" directly, though, as systemd handles that. Not in my experience. Docker is not Podman. I have Ceph clusters *right now* that use Docker and do not have Podman installed. They also aren’t RHEL.

...

6. As I said, Ceph does almost everything via cephadm When deployed with cephadm. You asked about containers, not about cephadm. They are not fungible.

...

or ceph orch when running in containers, which actually means you need to learn less. You assume that everyone already knows how containers roll, including the subtle dynamics of /etc/ceph/ceph.conf being mapped to the container’s filesystem view and potentially containing option settings that perplexing unless one knows how to find and modify them. That isn’t rue. When someone doesn’t know the dynamics of containers, they can add to the learning curve. And yes the docs do not yet pervasively cover the panoply of container scenarios.

...

Administration of ceph itself, is, again, done via systemd. Sorry, but that often isn’t the case.

...

*Docker. As I've said elsewhere, Red Hat prefers Podman to Docker these days Confused look. I know people who prefer using vi or like brussell sprouts. Those aren’t relevant to the question about containerized deployments either. And the question was re containers, not about the organization formerly known as Red Hat.

...

and even if you install Docker, there's a Podman transparency feature. See above.

...

Now if you really want networking headaches, run Podman containers rootless. I've learned how to account for the differences but Ceph, fortunately hasn't gone that route so far. Nor have they instituted private networks for Ceph internal controls.

On 9/1/24 15:54, Anthony D'Atri wrote:

...

* Docker networking is a hassle * Not always clear how to get logs * Not being able to update iptables without stopping all services * Docker package management when the name changes at random * Docker core leaks and kernel compatibility * When someone isn’t already using containers, or has their own orchestration, going to containers steepens the learning curve.

Containers have advantages including decoupling the applications from the underlying OS

...

I would greatly like to know what the rationale is for avoiding containers

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io