Podman containers will not restart due to restart or failure of centralized podman daemon. Container is not synonymous to Docker. This thread reminds me systemd haters threads more and more by I guess it is fine. On Thu, Jun 3, 2021, 2:16 AM Marc <Marc(a)f1-outsourcing.eu> wrote:

Dear Nico, do you think it is sensible and it's a precise statement saying that "we can't reduce complexity by adding a layer of complexity"? Containers are always adding a so-called layer, but people keep using them, and in some cases, they offload complexity from another side. Claiming the "complexity", without explaining examining the details is pure FUD. "complexity, OMG!!!111!!!" is not enough of a statement. You have to explain what complexity you gain and what complexity you reduce. Installing SeaweedFS consists of the following: `cd seaweedfs/weed && make install` This is the type of problem that Ceph is trying to solve, and starting a discussion by saying that everything is bad, without providing any helpful message is useless FUD. Max

My Cephadm deployment on RHEL8 created a service for each container, complete with restarts. And on the host, the processes run under the 'ceph' user account. The biggest issue I had with running as containers is that the unit.run script generated runs podman -rm ... with the -rm, the logs are removed when there is an issue, so it took extra effort to figure out what the issue I was having on one machine was. (Although that turned out to be a bad memory chip, which would have manifested itself anyway). As a manager of a team who develops microservices in containers, I have a mixed attitude towards them - the ability to know the version of Java, and support libraries deployed for a given container, and isolate updates one image at a time can be a bonus, but with my client's CI/CD pipeline and how they want our containers to be built, simple tasks like upgrading a package version, upgrading the version of Java or even a simple replacement of a certificate has become significantly more difficult, because we need to rebuild all of the containers and go through the QA processes rather than update a cert. For my usage, (At home running Ceph on older hardware I converted to servers) I don't want to have to care about ceph dependencies and also isolate things from other things running on the server, so a container infrastructure works well, but I can see where packages can be much better in a well maintained server infrastructure. -----Original Message----- From: Sasha Litvak <alexander.v.litvak(a)gmail.com> Sent: Thursday, June 3, 2021 3:57 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Podman containers will not restart due to restart or failure of centralized podman daemon. Container is not synonymous to Docker. This thread reminds me systemd haters threads more and more by I guess it is fine. On Thu, Jun 3, 2021, 2:16 AM Marc <Marc(a)f1-outsourcing.eu> wrote: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi, this is rather important for us as well. we use systemd-nspawn based containers (that act and are managed like traditional VMs, just without the overhead). cephadm enforces not just containers, but particular ones (granted, docker/podman are the currently most used container-runtimes). it would be nice if cephadm would, like cephdeploy did, support "traditional"/plain systems (which then automatically works perfectly for us with systemd-nspawn without any changes). until now we've still used ceph-deply and stayed away from cephadm, as it would require us to completely change our system deployment. Regards, Daniel

Following up with some general comments on the main container downsides and on the upsides that led us down this path in the first place. Aside from a few minor misunderstandings, it seems like most of the objections to containers boil down to a few major points: I think that part of this comes down to a learning curve and some semi-arbitrary changes to get used to (e.g., systemd unit name has changed; logs now in /var/log/ceph/$fsid instead of /var/log/ceph). Another part of these changes are real hoops to jump through: to inspect process(es) inside a container you have to `cephadm enter --name ...`; ceph CLI may not be automatically installed on every host; stracing or finding coredumps requires extra steps. We're continuing to improve the tools etc so please call these things out as you see them! This feels like the most tangible critique. It's a tradeoff. We have had so many bugs over the years due to varying versions of our dependencies that containers feel like a huge win: we can finally test and distribute something that we know won't break due to some random library on some random distro. But it means the Ceph team is on the hook for rebuilding our containers when the libraries inside the container need to be patched. On the flip side, cephadm's use of containers offer some huge wins: - Package installation hell is gone. Previously, ceph-deploy and ceph-ansible had thousands of lines of code to deal with the myriad ways that packages could be installed and where they could be published. With containers, this now boils down to a single string, which is usually just something like "ceph/ceph:v16". We're grown a handful of complexity there to let you log into private registries, but otherwise things are so much simpler. Not to mention what happens when package dependencies break. - Upgrades/downgrades can be carefully orchestrated. With packages, the version change is by host, with a limbo period (and occasional SIGBUS) before daemons were restarted. Now we can run new or patched code on individual daemons and avoid an accidental upgrade when a daemon restarts. (Also, running e.g. ceph CLI commands no longer error out with a dynamic linker error while the package upgrade itself is in progress, something all of our automated upgrade tests have to carefully avoid to prevent intermittent failures.) - Ceph installations are carefully sandboxed. Removing/scrubbing ceph from a host is trivial as only a handful of directories or configuration files are touched. And we can safely run multiple clusters on the same machine without worry about bad interactions (mostly great for development, but also handy for users experimenting with new features etc). - Cephadm deploys a bunch of non-ceph software as well to provide a complete storage system, including haproxy and keepalived for HA ingress for RGW and NFS, ganesha for NFS service, grafana, prometheus, node-exporter, and (soon) samba for SMB. All neatly containerized to avoid bumping into other software on the host; testing and supporting the huge matrix of packages versions available via various distros would be a huge time sink. Most importantly, cephadm and the orchestrator API vastly improve the overall ceph experience from the CLI and dashboard. Users no longer have to give any thought to where and which daemons run if they don't want to (or they can carefully specify daemon placement if they choose). And users can use commands like 'ceph fs volume create foo' and the fs will get created *and* MDS daemons will be started all in one go. (This would also be possible with a package-based orchestrator implementation if one existed.) We've been beat up for years about how complicated and hard Ceph is. Rook and cephadm represent two of the most successful efforts to address usability (and not just because they enable deployment management via the dashboard!), and taking advantage of containers was one expedient way to get to where we needed to go. If users feel strongly about supporting packages, we can get much of the same experience with another package-based orchestrator module. My view, though, is that we have much higher priority problems to tackle. sage

Hello Sage, usability... Somehow it does not look right to me. There is much more to operate a Ceph cluster than just deploying software. Of course that helps on the short run to avoid that people leave the train right when they started their Ceph journey. But the harder part is what to do if shit hit's the fan and your cluster is down due to some issues and then having additional layers of complexity kicking in and biting your ass. Just saying, that day2 ops is much more important than getting a cluster up&running. In my believe, no admin want to dig around containers and other abstractions when the single most important part of a whole IT infrastructure stops working. But just my thought, maybe I'm wrong. In my opinion, the best possible way to run IT software is KISS, keep it stupid simple. No additional layers, no abstractions of abstractions and good error messages. For example the docker topic here looks like something that can be showcased: containers? As you might see, you need a lot of knowledge about abstraction layers to operate them well. Docker for example provides so called live-restore ( https://docs.docker.com/config/containers/live-restore/) that allows you to stop the daemon without killing your containers. This enables you to update docker daemon without downtimes but you have to know it and of course enable it. This can make operating a Ceph cluster harder, not easier. What about more sophisticated features, for example performance. Ceph already is not a fast storage solution with way to high latency. Does it help to add containers instead of going more direct to the hardware and reduce overhead? Of course you can run SPDK and/or DPDK inside containers, but does it make it better and faster or even easier? If you need high-performance storage today, you can turn to open source alternatives that are massively cheaper per IO and only minimally more expensive per GB. I therefore believe, stripping out overhead is also an important topic for the future of Ceph. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx On Fri, 18 Jun 2021 at 20:43, Sage Weil <sage(a)newdream.net> wrote:

On Sat, Jun 19, 2021 at 3:43 PM Nico Schottelius <nico.schottelius(a)ungleich.ch> wrote: I want to reiterate that while cephadm requirements are systemd+lvm+python3+containers, the orchestration framework does not have any of these limitations, and is designed to allow you to plug in other options. Cephadm is intended to be the primary non-k8s option, since it seems pretty clear that there is a significant (huge?) portion of the user commuity that is not interested in adding kubernetes underneath their storage (take all of the "containers add complex" arguments and * 100). We used containers because, in our view, it simplified the developer AND user experience. But neither rook nor cephadm preclude deploying Ceph the traditional way. The newer capabilities in the dashboard to manage the deployment of Ceph relies on the orchestrator API, so a traditional deployment today cannot make use of these new features, but nothing is preventing a non-container-based orchestrator implementation. sage

Am 18.06.21 um 20:42 schrieb Sage Weil: Thanks, Sage, for the nice and concise summary on the Cephadm benefits, and the reasoning on why the path was chosen! Also thanks for your reply on my question about the modularity of the actual orchestrator. I really appreciate this, and will try to reply in one place here. After the huge activity in this thread, I did take a step back to watch and also make up my mind, trying to condense my main issues with the "containers-only" approach, also taking other replies into account. I hope this is not seen as a rant, but rather a collection of arguments for an additional orchestrator module, or maybe even something different. Unfortunately, it has become a wall of text, but I hope at least some will fight their way through. First of all, I fully agree with the positive points you raised — it's surely a gain for devs and many users to ship something tested and "complete" without having a full and still necessarily incomplete OS test matrix to constantly check and extend. It also eases testing especially when trying out experimental features, and takes away usage complexity e.g. in the upgrade path. Of course, there's also the point that having a large test matrix across OSs tends to uncover actual bugs or issues which may not show up in a reduced test environment[0], so reducing the matrix also comes at a price for reliability which has to be weighed against the time which is saved. The security issue (50 containers -> 50 versions of openssl to patch) also still stands — the earlier question on this list (when to expect patched containers for a CVE affecting a library) is still unreplied to[1], so these are real-life concerns. In general, I don't know any project which ever managed to keep up with the workload caused by the requirement to follow all CVEs of all dependencies, informing about them and patching them, since this is a workload comparable to the one the security teams of Linux distributions have to handle. In addition, you'll also need to address the question of when and how to pull new images when patched containers become available, how / when to inform the administrator, and orchestrate service restarts as-needed (you'd basically need "needs-restarting" and friends). That's still quite a way to go, and will be a constant developer effort from now on. That being said, Ceph may be the first ever project managing to fulfil expectations here due to the close coupling to those guys wearing red hats ;-). Another point raised on this list is that some users are anxious about pushing a "magic" button which upgrades a whole cluster. Sure, this button is super useful, and incorporates developer wisdom, and allows the developers to test the full sequence and ship it to everybody. So these buttons (e.g. "ceph orch upgrade") are something which is useful and, I should say, important. However, by design, it hides the "inner workings" of Ceph, which is a major drawback for some users. After this introduction, let me come to my main personal concern: Loss of integratibility. Our model of operation is to have all machines (anything, be it your off-the-shelf desktop, a laptop, a hypervisor, a compute node or a Ceph node) handled by the very same configuration management. It means all configuration is self-documenting, reinstallation is done with the push of a button, and anybody who understands the configuration management and the services at a basic level can take over operations. It's the only way we _can_ operate, given the huge number of services requested and required in the IT business these days. To give just one example: We mount kerberised NFS on all our desktop nodes, via CephFS exposed via nfs-ganesha. The desktops run Ubuntu/Debian, the file servers CentOS. If I need to change the Kerberos configuration (order of KDCs, roll out new principals etc.), for us this is a change in a single place: We perform the change in Puppet, wait 30 minutes, and all systems run the new configuration[2]. When operating a service with its own orchestrator, this means for me: I have to manually adapt the configuration of this service. I need someone who is able to do that (i.e. two configuration systems have to be learnt), and who then does it for all instances of the service (e.g. all Ceph clusters). Hence, a previously simple change is multiplied in complexity. I can't just replace the OS disk of a Ceph-OSD node and push "reinstall" anymore, letting Puppet install all services and Ceph packages (so I only have to adopt the disks, this is not automated as a safety precaution), but I also have to talk to the Ceph orchestrator. So automation is a must, and we also heavily rely on containers for scientific workloads to offer a large variety of software stacks to our users. Automation works brilliantly on Linux (and likely also similar open platforms), since almost all services can be combined as building blocks, and controlled by our configuration management Puppet (or any other tool). It's basically a consequence of the Unix idea that all programs do their own thing well, and can be "glued" together as-needed for site-specific requirements. While the "cephadm with containers" approach does this kind of glueing for me, it makes it harder (impossible?) to integrate as-is into existing configuration management systems. I think this is also a strong point, and reading through the list, one of the major reasons why larger ceph sites do not want to use cephadm with containers seems to be exactly that: It can't easily be "cut into pieces" and be integrated with an existing system they use for all other infrastructure. So in summary: The orchestrator is a good thing, but my point is that the currently implemented solution is not the right solution for a noticeable fraction of the existing community. In addition to the users you had in mind when designing the current orchestrator, there are also many active users of Ceph who want to have more direct access to the "complexity" of the system for two reasons[3]: - To integrate it into existing automations. - To learn how things work and interact. The latter point is also a strong one, especially for me as an experimental physicist: I learned to love Ceph exactly because I played with the different components and their interactions, tried to break them, found how they react, and got a deeper understanding and knowledge to understand any future issues. This love never develops for me when I use a more "polished" product which has buttons doing things for me. It's a major reason why I'd even choose Ceph(FS) over G*FS if the latter was available for free (of course, there are many more reasons). So my conclusion is: The chosen path is changing the audience of Ceph, affecting both existing users, but also new users. The length of this thread has shown that the community has different opinions on this path forward, for many different reasons. My personal feeling is that the solution embraces new users who want to set something up quickly, and is mainly a problem for existing and future long-term production users with larger clusters who want to understand the full stack and integrate it into their environment. Does it really free developer resources in the long run? I'm not sure about that — the community may shift more to users reporting issues like "I pushed the green upgrade button, and now it is stuck" (similar mails have arrived on the mailing list already), and have less reports from users who report stacktraces or issues in network communication between services (will automated crashdump reports be able to replace experienced bug reporters?). My personal feeling is that the latter type of users are those who stay with Ceph for years or decades, and even though they may seem to be complaining that Ceph is a complicated machine with many nuts and bolts, it's usually not something which disturbs them as much as it may seem to be the case. So finally, what is my idea about a path forward? In addition to keep delivering packages (will they stay?), I see two ways, basically: - Having an additional orchestrator module running on bare-metal. Given the assumptions above and mails on this list, most users who'd use it would in any case install their packages differently, and only use the orchestrator e.g. to distribute cephx keys, set up initial configuration, enable systemd units etc., finally persisting a differing fraction of these things into their configuration management. So the orchestrator may even be useful for them without the actual complex capabilities of adding repositories and installing packages. At least, that's what I'd be happy about: An orchestrator doing all the Ceph-only things for me, ideally telling me what it does. - Having really extensive manual installation instructions. Currently, I find these super useful for the basic first steps, but they basically break off after you have a mon, mgr and osd. In essence, these two points are the same: The modular orchestrator already has all tasks coded inside, i.e. essentially, the orchestrator is the most up-to-date, complete, tested and well-maintained manual documentation we'll get — correct? This is why my personal feeling is that having a "bare metal orchestrator" (i.e. an "SSH orchestrator" like ceph-deploy) even without the features of adding repositories, installing packages, or upgrading at the push of a button will be sufficient for those of us having issues with the current solution. It's essentially about making the manual installation instructions into an orchestrator module (which is probably close to the current cephadm minus containers). Is this a scope which is sufficiently low-hanging to warrant the effort? It may also be less work than having a more extensive manual installation documentation, and maintaining it? Cheers (and congratulations to all who made it to the end of this mail), Oliver [0] As a basic example, g++ gets increasingly better at warning about potentially unintended behaviour caused by common classes of bugs in code, so a newer g++ may find more issues when testing. The same may happen for different library versions, which may point out API usage bugs early on, or reveal issues earlier in testing. [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/PPLJIHT6WK… [2] Of course, you can (and should) use a staged rollout. [3] Well, that's a presumption, but the fact that you mentioned user concerns about this in the survey seems to strengthen that point. -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.047 Nußallee 12 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

Is there a link of the talk I can use as a reference? I would like to look at the pro container points as this post is getting a little bit one sided. I understand that most people prefer things to be stable especially with the underlying storage systems. To me personally, use of containers in general adds a great flexibility because it detaches underlying OS from the running software. All points are fair about adding complexity to the complex system but one thing is missing. Every time developers decide to introduce some new more efficient libraries or frameworks we hit a distribution dependency hell. Because of that, ceph sometimes abandons entire OS versions before their actual lifetime is over. My resources are limited and I don't want to debug / troubleshoot / upgrade OS in addition to ceph itself, hence the containers. Yes it took me a while to warm up to the idea in general but now I don't even think too much about it. I went from Nautilus to Pacific (Centos 7 to Centos 8) within a few hours without needing to upgrade my Ubuntu bare metal nodes. This said, I am for giving people a choice to use packages + ansible / manual install and also allowing manual install of containers. Forcing users' hands too much may make people avoid upgrading their ceph clusters. On Wed, Jun 2, 2021 at 9:27 AM Dave Hall <kdhall(a)binghamton.edu> wrote:

Only if you also look at why containers are bad in general, as that also applies to ceph as well. Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com -----Original Message----- From: Fox, Kevin M [mailto:Kevin.Fox@pnnl.gov] Sent: Wednesday, June 2, 2021 9:27 AM To: Sasha Litvak; ceph-users(a)ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Debating containers vs packages is like debating systemd vs initrd. There are lots of reasons why containers (and container orchestration) are good for deploying things, including ceph. Repeating them in each project every time it comes up is not really productive. I'd recommend looking at why containers are good in general. It applies to ceph too. ________________________________________ From: Sasha Litvak <alexander.v.litvak(a)gmail.com> Sent: Wednesday, June 2, 2021 7:56 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. Is there a link of the talk I can use as a reference? I would like to look at the pro container points as this post is getting a little bit one sided. I understand that most people prefer things to be stable especially with the underlying storage systems. To me personally, use of containers in general adds a great flexibility because it detaches underlying OS from the running software. All points are fair about adding complexity to the complex system but one thing is missing. Every time developers decide to introduce some new more efficient libraries or frameworks we hit a distribution dependency hell. Because of that, ceph sometimes abandons entire OS versions before their actual lifetime is over. My resources are limited and I don't want to debug / troubleshoot / upgrade OS in addition to ceph itself, hence the containers. Yes it took me a while to warm up to the idea in general but now I don't even think too much about it. I went from Nautilus to Pacific (Centos 7 to Centos 8) within a few hours without needing to upgrade my Ubuntu bare metal nodes. This said, I am for giving people a choice to use packages + ansible / manual install and also allowing manual install of containers. Forcing users' hands too much may make people avoid upgrading their ceph clusters. On Wed, Jun 2, 2021 at 9:27 AM Dave Hall <kdhall(a)binghamton.edu> wrote: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

On 6/2/21 2:28 PM, Phil Regnauld wrote: I think there's great value in ceph devs doing QA and testing docker images, releasing them as a 'known good thing'.  Why? Doing that avoids dependency hell inducing fragility-- fragility which I've experienced in other multi-host / multi-master packages.  Wherein one distro's maintainer decides some new rev ought be pushed out as 'security update' while another distro's maintainer decides it's a feature change, another calls it a backport, etc.  There's no way to QA 'upgrades' across so many grains of shifting sand. While the devs and the rest of the bleeding-edge folks should enjoy the benefits that come with tolerating and managing dependency hell, having the orchestrator upgrade in a known good sequence from a known base to a known release reduces fragility. Thanks for ceph! Harry

Ultimately, that's what a container image is. From the outside, its a statically linked binary. From the inside, it can be assembled using modular techniques. The best thing about it, is you can use container scanners and other techniques to gain a lot of the benefits of that modularity still. Plus, you don't need special compilers, whole dependency chains compiled with static options, etc. So an interesting data point in the statically linked binary vs dynamic debate. Its kind of both. ________________________________________ From: Rok Jaklič <rjaklic(a)gmail.com> Sent: Wednesday, June 2, 2021 1:18 PM To: Harry G. Coin Cc: ceph-users Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. In this giga, tera byte times all this dependency hell can now be avoided with some static linking. For example, we do use statically linked mysql binaries and it saved us numerous times. https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyoutu.be%… Rok On Wed, Jun 2, 2021 at 9:57 PM Harry G. Coin <hgcoin(a)gmail.com> wrote: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi, We operate a few Ceph hyperconverged clusters with Proxmox, that provides a custom ceph package repository. They do a great work; and deployment is a brezee. So, even as currently we would rely on Proxmox packages/distribution and not upstream, we have a number of other projects deployed with containers and we even distribute some of our own development in deb and container packages, so I will comment on our view: El 2/6/21 a las 23:26, Oliver Freyermuth escribió: [...] These are the two main issues I find with containers really: - Keeping hosts uptodate is more complex (apt-get update+apt-get dist-upgrade and also some kind of docker pull+docker restart/docker-compose up ...). Much of the time the second part is not standard (just deployed a Harbor service, upgrade is quite simple but I have to know how to do it as it's speciffic, maintenance would be much easier if it was packaged in Debian). I won't say it's more difficult, but it will be more diverse and complex. - Container image quality and security support quality, that will vary from upstream to upstream. You have to research each of them to know were they stand. A distro (specially a good one like Debian, Ubuntu, RHEL or SUSE) has known, quality security support for the repositories. They will even fix issues not fixed by upstream (o backport them to distro's version...). This is more an upstream vs distro issue, really. About debugging issues reported with Ceph containers, I think those are things waiting for a fix: why are logs writen in container image (or an ephemeral volume, I don't know really how is that done right now) instead of an external name volume o a local mapped dir in /var/log/ceph ? All that said, I think that it makes sense for an upstream project like Ceph, to distribute container images, as it is the most generic way to distribute (you can deploy on any system/distro supporting container images) and eases development. But only distributing container images could make more users depend on third party distribution (global or specific distros), which would delay feeback/bugreport to upstream. Cheers and thanks for the great work! Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/

I'm arriving late to this thread, but a few things stood out that I wanted to clarify. On Wed, Jun 2, 2021 at 4:28 PM Oliver Freyermuth <freyermuth(a)physik.uni-bonn.de> wrote: The orchestrator layer is an abstraction of the tool(s) used to provision ceph. Cephadm is just one implementation of that abstraction (there is another implementation that talks to rook). The abstraction is not designed to be 'container-only,' although currently both implementations do use containers. To add orchestrator support for a traditional deployment using packages, we have two options: (1) implement a third orchestrator module that handles package installation etc, or (2) modify cephadm to handle both container-based and package-based deployments. I suspect that (2) is less work, but I'll be honest that the cephadm team isn't yet swayed by the anti-container arguments, so there would be some lobbying and discussion to be done first! For either option, there are also some minor orchestrator interface adjustments that maybe needed. Cephadm can deploy an arbitrary ceph version/container image for individual daemons, but with packages it's per-host. With rook it's a per-cluster option (rook currently handles any upgrade/downgrade), so there is already some ambiguity and cleanup opportunity here. sage

Podman uses systemd to manage containers so that it is daemonless, contrast with Docker where one has to maintain a separate daemon and use docker-specific tools to mange containers. If you assert that Podman should not exist, please take that up with the Podman folks. Bundling all the daemons together into one container is *genius*. Much simpler to build and maintain, one artifact vs a bunch. I wouldn’t be surprised if there are memory usage efficiencies too. This seems like a non-sequitor. As many have written, CPU and memory limits aren’t the reason for containerizing Ceph daemons. If there are other container applications where doing so makes sense, that’s fine for those applications. I suspect that artificial CPU limiting of Ceph daemons would have a negative impact on latency, peering/flapping, and slow requests. Ceph is a distributed system, not a massively parallel one. OSDs already have a memory target that can be managed natively, vs an external mechanism that arbitrarily cuts them off at the knees when they need it the most. That approach would be addressing the symptoms, not the root cause. Having been through a multi-day outage that was substantially worsened by the OOMkiller (*), I personally want nothing to do with blind external mechanisms deciding that they know better than Ceph daemons whether or not they should be running. If your availability and performance needs favor rigidly defined areas of doubt and uncertainty, that’s your own lookout. * A release that didn’t have OSD memory target setting yet. Having that would have helped dramatically. With all possible respect, that’s another non-sequitor, or at the very least, an assumption that your needs are everyone’s needs. Centralized logging makes sense in some contexts. But not always, and not to everyone. Back around the Hammer or Jewel releases there was a bug where logging to syslog resulted in daemon crashes. I haven’t tried it with newer releases, but assume that’s long been fixed. I’m not a an [r]syslog[ng] expert by far, but I suspect that central-only logging might not deal well with situations like an OSD spewing entries when the debug subsystem level is elevated. Moreover, many of the issues one sees with Ceph clusters are network related. So when there’s a network problem, I want to rely on the network to be able to see logs? I’ve seen syslog drop entries under load, something I personally wouldn’t want for Ceph daemons. There are of course many strategies between the extremes. I’m guessing you’ve never updated between major releases. That process tends to have additional steps and nuances, which is one of the compelling arguments in favor of orchestration: when it’s done well, most operators don’t need to rev their own homebrew orchestration to set the right flags at the right time, etc. But one of the great things about OSS is that you have the flexibility to roll you own if you so choose. So you want to do all the things that an orchestrated rolling upgrade does for you. Check. Careful with those inferences. * Operators who submit telemetry may not be a representative sample * Sites may have many more than one cluster, If one has a 20 OSD lab cluster and a 1000 OSD production cluster, perspectives and processes are going to be different than someone with a single 60 OSD cluster. * Average != median That’s more than a little harsh. A lot of people get a lot of value out of those tools. They don’t make julienne fries either. What of it? Nobody’s stopping you from rolling your own containers, using traditional packages, or heck even deploying with tarballs. That’s the beauty of OSS. Let’s leave Orange County out of it though. Asked and answered, sir. Different ones than you’re trying to solve, which doesn’t mean that either you or upstream are wrong. There’s always room for improvement with any approach, and again the beauty of OSS is that you *can* go your own way if you so choose.

I think the primary goal of a container environments are resource isolation. At least when I read about the history I never read anything about a tool for people to skip to learn something. But you clearly portray the situation here. Nobody here is against container environment, because every tool has its purpose. But the container solution proposed here is not for the purpose of utilizing container benefits, but for creating a tool so 'I do not know what to do' people can use ceph. And because this is the development perspective and ceph is not really adapted for being used in containers, you get the current friction with accepting cephadm. Eg. this command is wrong in my opinion ceph orch device ls What has a device list to do with having an OC or not? I do not even know why ceph is working on something that uses Kubernetes, let the Kubernetes platform implement a ceph solution. If ceph wants be used in a container environment, start making the daemons ready to run in container environments. So my question still stands: What problem is it actually that the cephadm dev team is trying to solve? That is not clear to me.

I think 2 things need to be clarified here: If a daemon is running on an operating system, it does not reload shared libraries or binaries. So upgrading a system != every daemon at the same time gets new executable content loaded. If a daemon restarts that particular moment of OS upgrade, then yes, then it loads the new binary. But are you able to hit that spot of a few minutes? Unlikely. Especially if you tested your upgrade on a staging environment before. Not biting on this one. If we talk platforms, you are basically fine for a lot of projects if you deliver a set of .rpms and a set of .debs. Or if you work closely together with the distros, then often that distro can actually help you building (and maintaining) that package. As far as I know the core ceph team is working at Redhat, so you are basically getting 1 distro/package for free. And while we are at claiming "on a lot more platforms", you are at the same time EXCLUDING a lot of platforms by saying "Linux based container" (remember Ceph on FreeBSD? [0]). I am aware that centralisation is in fashion, but coming from an Open Source background, there is one big reason to build Open Source software: to be portable and inclusive. I've experienced the time where the "only platform" was MS-DOS/Windows and I can assure anyone in here that "setting on one platform" instead of "being portable" is a shot in your own foot. History repeats. Cheers, Nico [0] https://docs.ceph.com/en/latest/install/manual-freebsd-deployment/ -- Sustainable and modern Infrastructures by ungleich.ch

On 6/21/21 7:37 PM, Marc wrote: I advise to read the whole thread again, especially Sage his comments, as there are other benefits. It would free up resources that can be dedicated to (arguably) more pressing issues. Containers are not being used as they should be. There is no "should be", there is no one answer to that, other than 42. Containers have been there before Docker, but Docker made them popular, exactly for the same reason as why Ceph wants to use them: ship a known good version (CI tests) of the software with all dependencies, that can be run "as is" on any supported platform. Gr. Stefan

On 6/22/21 6:56 PM, Martin Verges wrote: To see the bigger picture: if indeed resources are freed up they can be used to fix bugs faster / create new features, add more tests, etc. There are differences when running a docker As far as I understood there are a few container runtimes that are supported, i.e docker, podman. And that this might be extended in the future. So, at least in theory, it should work on those. There are even problems with containers if you Future will tell what container runtimes gives least amount of problems / most benefits. I guess natural selection will do it's thing here. But you got a valid point: there still might be minor version compatibility issues in container runtimes themselves (although I wonder how often those impact Ceph). And even that is something that could be checked by the orchestrator. Still I would recommend to test container deployments just like non-containerized deploys, just to spot these kind of issues. I have used rancher, but not on debian, and only for tests. Gr. Stefan

Hey Sage, thanks for the reply. Sage Weil <sage(a)newdream.net> writes: let me come back to a particular part of your message: " our experience so far has been a big improvement over the complexity of managing package dependencies across even just a handful of distros. " This is something I cannot understand at all. This is a process that for most mid sized Open Source projects have automated years ago. And you can even use containers for that! Does it break on distro upgrade? Maybe. Is it trival? Probably in 90% of the cases. Is it easy to spot? Almost always if you use CI. *What* is exactly the complexity that you deal with? Where are the problems? Is there an open issue that says "Reduce time and complexity of package build" open anywhere? Can we join that discussion? The message I get here is: - Building packages is too hard for us - let's make the life of everyone else more complex for a build case that works easier for us I am not sure whether that's the right direction. *If* this this really about complexity of package building, why did you not shout out to the community and ask for help? I assume that one or the other party on this mailing list is open for helping out. Nico -- Sustainable and modern Infrastructures by ungleich.ch

What I am getting from reading 'between the lines'. Is that they want to create something that is easier to install for a broader target audience. And instead of just saying this, for whatever reasoning other arguments have been put forward which are questionable and therefore raise a discussion here. Thanks for the link and info!

I see or hear none of that in cephadm presentations. Are you sure you are not using some future never going to exist feature to substantiate the current state? The expected general ambitionless reply. To me a should be is that container images should run on any container platform. To me a should be is that you ship your container having only the relevant and necessary binaries. Sort of like haproxy (9MB) or java/python where you only have some execution environment included. To me a should be is that the task in the container, is just one process and does not spawn anything else. To me a should be is that the task can be scaled to multiple instances by the OC. Yes yes this is constantly been written. As if this was a huge problem the last decade? Please show me a roadmap of ceph development where this is a high priority. I rather see resources go to the crimson development and testing before a releasing. Do you know your argument of shipping code was quite similar to the senior software engineer of redhat. Who thought it was not a problem that accidentally the root fs of node was mounted in a container. Because the driver should have been ran in a namespace. Is the short version of this not 'hide our crappy code'? This is incorrect, afaik I need to have a platform that has podman not? I would like to run ceph containers on mesos without podman. Where is this man page???? And to be honest how long is docker going to supply these huge images for free? It is not like there is distributed network where you can get your container images. To me this development just looks wrong. It just adds complexity while the beauty is to keep things simple. Today I heard some agent is going to be developed and added to the nodes, I heard something about port conflicts and some issues with ip addressing support. It looks like this OC is developed from the ground up or this podman is seriously lacking functionality. Does it publish automatically assigned ports via srv records, do tasks get dns entries? How is it with dhcp? In any case my current OC does not have such issues, and I am quite happy with it. Why would any company like to administer 2nd OC just for ceph? If ceph goes containers, it should be in the OC the client chooses to use. Ceph just needs to make this work regardless of what OC the client has, not develop something custom. From a security perspective there is also some doubt think twice here. Do you really want to run Grafana, Prometheus and haproxy on the OC that is running your osd's??? The ceph OC has to allow CAP_SYS_ADMIN[1] to tasks to be able to run drivers to mount lvm's. (I can't imagine there are lot of people disabling this like I have done, and run these as external drivers). So if there is any exploit in your ceph OC a single task could destroy a node (and maybe worse, get automatically started on a different node, so it can destroy that one as well) So how is this going to look like when you have a ceph cluster with 100 osd, 3 monitors and 3 rgw, prometheus, prafana. From a performance perspective others have already been writing that containerizing osd's could very well influence latency. I have already seen this with a dns server on a macvtap interface. However I have to admit I do not have the resources to test and give a proper judgement on this. If people here are talking about fixing c-states and frequencies. I wonder how far the containerizing of osd's even goes. Say these are not fully containerized because of performance issues, then you have custom OC from ceph that runs 8 processes containerized from the 108. I would think maybe add this "To me a should be is " list? This ceph containerizing is almost a joke. I will bet you can not even run Prometheus as a stateful task on a ceph rbd image in this CO. Do you know why? Because the csi-ceph driver development team do not grasp the concept of csi and code a driver that can only be used to Kubernetes. Ceph and containers sure, if I may quote a red haired film star "Kill it in the utero" 😉 [1] https://lwn.net/Articles/486306/

On Tue, Jun 22, 2021 at 1:25 PM Stefan Kooman <stefan(a)bit.nl> wrote: Thank you for bringing this up. This is in fact a key reason why the orchestration abstraction works the way it does--to allow other runtime environments to be supported (FreeBSD! sysvinit/Devuan/whatever for systemd haters!) while ALSO allowing an integrated, user-friendly experience in which users workflow for adding/removing hosts, replacing failed OSDs, managing services (MDSs, RGWs, load balancers, etc) can be consistent across all platforms. For 10+ years we basically said "out of scope" to these pesky deployment details and left this job to Puppet, Chef, Ansible, ceph-deploy, rook, etc., but the result of that strategy was pretty clear: ceph was hard to use and the user experience dismal when compared to an integrated product from any half-decent enterprise storage company, or products like Martin's that capitalize on core ceph's bad UX. The question isn't whether we support other environments, but how. As I mentioned in one of my first messages, we can either (1) generalize cephadm to work in other environments (break the current systemd+container requirement), or (2) add another orchestrator backend that supports a new environment. I don't have any well-formed opinion here. There is a lot of pretty generic "orchestration" logic in cephadm right now that isn't related to systemd or containers that could either be pulled out of cephadm into the mgr/ochestrator layer or a library. Or an independent, fresh orch backend implementation could opt for a very different approach or set of opinions. Either way, my assumption has been that these other environments would probably not be docker|podman-based. In the case of FreeBSD we'd probably want to use jails or whatever. But anything is possible. s

This thread would not be so long if docker/containers solved the problems, but it did not. It solved some, but introduced new ones. So we cannot really say its better now. Again, I think focus should more on a working ceph with clean documentation while leaving software management, packages to admins. And staticilly linked binaries would certinly solve dependecy hell and "how to support other environments" for most of the cases. On Thu, 24 Jun 2021, 23:06 Sage Weil, <sage(a)newdream.net> wrote:

Hey Sage, Sage Weil <sage(a)newdream.net> writes: I would like you to stop labeling people who have reasons for not using a specific software as haters. It is not productive to call Ceph developers "GlusterFS haters", nor to call Redhat users Debian haters. It is simple not an accurate representation. Cheers, Nico -- Sustainable and modern Infrastructures by ungleich.ch

So why did you make this a point previously stating something that you were able to use ceph with kubernetes like commands. How can you compare docker/containers to ansible? It is like comparing a drill to fruit mixer or so. Your feelings mislead you, you would understand this if you knew what I am writing about. 1. Is this even being done then cephadm? I still see commands that upgrade a whole node. 2. Has this ever been a problem in the past? I had no problems going from Kraken -> Luminous -> nautilus. There is no significant problem updating and/or upgrading. Yes and let me guess the solution is sending a whole centos/rhel container image? Not!

I've actually had rook-ceph not proceed with something that I would have continued on with. Turns out I was wrong and it was right. Its checking was more through then mine. Thought that was pretty cool. It eventually cleared itself and finished up. For a large ceph cluster, the orchestration is very nice. Thanks, Kevin ________________________________________ From: Sage Weil <sage(a)newdream.net> Sent: Thursday, June 24, 2021 1:46 PM To: Marc Cc: Anthony D'Atri; Nico Schottelius; Matthew Vernon; ceph-users(a)ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. On Sun, Jun 20, 2021 at 9:51 AM Marc <Marc(a)f1-outsourcing.eu> wrote: rgw, grafana, prom, haproxy, etc are all optional components. The monitoring stack is deployed by default but is trivially disabled via a flag to the bootstrap command. We are well aware that not everyone wants these, but we cannot ignore the vast majority of users that wants things to Just Work without figuring out how to properly deploy and manage all of these extraneous integrated components. Early on the team building the container images opted for a single image that includes all of the daemons for simplicity. We could build stripped down images for each daemon type, but that's an investment in developer time and complexity and we haven't heard any complaints about the container size. (Usually a few hundred MB on a large scale storage server isn't a problem.) Something needs to start the ceph daemon containers when the system reboots. We integrated with systemd since all major distros adopted it. Cephadm could be extended to support other init systems with pretty minimal effort... we aren't doing anything fancy with systemd. The orchestrator automates the creation and cleanup of credentials for each rgw instance. (It also trivially scales them up/down, ala k8s.) If you have an autoscaler, you just need to tell cephadm how many you want and it will add/remove daemons. If you are using cephadm's ingress (haproxy) capability, the LB configuration will be adjusted for you. If you are using an external LB, you can query cephadm for a description of the current daemons and their endpoints and feed that info into your own ingress solution. Only the OSD containers get extra caps to deal with the storage hardware. Memory limits are partially implemented; we haven't gotten to CPU limits yet. It's on the list! By default, we log to stderr and your logs are in journald or whatever alternative your container runtime has set up. You can trivially flip a switch and you get traditional file-based logs with a logrotated.d config, primary to satisfy users (like me!) who aren't comfortable with the newfangled log management style. I humbly contend that most users, especially those with small clusters, would rather issue a single command and have the cluster upgrade itself--with all of the latest and often version-specific safety checks and any special per-release steps implemented for them--than to do it themselves. sage _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Orchestration is hard, especially with every permutation. The devs have implemented what they feel is the right solution for their own needs from the sound of it. The orchestration was made modular to support non containerized deployment. It just takes someone to step up and implement the permutations desired. And ultimately that's what opensource is geared towards. With opensource and some desired feature, you can: 1. Implement it 2. Pay someone else to implement it 3. Convince someone else to implement it in their spare time. The thread seems to be currently focused around #3 but no developer seems to be interested in implementing it. So that leaves options 1 and 2? To move this forward, is anyone interested in developing package support in the orchestration system or paying to have it implemented? ________________________________________ From: Oliver Freyermuth <freyermuth(a)physik.uni-bonn.de> Sent: Wednesday, June 2, 2021 2:26 PM To: Matthew Vernon; ceph-users(a)ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. Hi, that's also a +1 from me — we also use containers heavily for scientific workflows, and know their benefits well. But they are not the "best", or rather, the most fitting tool in every situation. You have provided a great summary and I agree with all points, and thank you a lot for this very competent and concise write-up. Since in this lengthy thread, static linking and solving the issue of many inter-dependencies for production services with containers have been mentioned as solutions, I'd like to add another point to your list of complexities: * Keeping production systems secure may be a lot more of a hassle. Even though the following article is long and many may regard it as controversial, I'd like to link to a concise write-up from a packager discussing this topic in a quite generic way: https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblogs.gen… While the article discusses the issues of static linking and package management performed in language-specific domains, it applies all the same to containers. If I operate services in containers built by developers, of course this ensures the setup works, and dependencies are well tested, and even upgrades work well — but it also means that, at the end of the day, if I run 50 services in 50 different containers from 50 different upstreams, I'll have up to 50 different versions of OpenSSL floating around my production servers. If a security issue is found in any of the packages used in all the container images, I now need to trust the security teams of all the 50 developer groups building these containers (and most FOSS projects won't have the ressources, understandably...), instead of the one security team of the disto I use. And then, I also have to re-pull all these containers, after finding out that a security fix has become available. Or I need to build all these containers myself, and effectively take over the complete job, and have my own security team. This may scale somewhat well, if you have a team of 50 people, and every person takes care of one service. Containers are often your friend in this case[1], since it allows to isolate the different responsibilities along with the service. But this is rarely the case outside of industry, and especially not in academics. So the approach we chose for us is to have one common OS everywhere, and automate all of our deployment and configuration management with Puppet. Of course, that puts is in one of the many corners out there, but it scales extremely well to all services we operate, and I can still trust the distro maintainers to keep the base OS safe on all our servers, automate reboots etc. For Ceph, we've actually seen questions about security issues already on the list[0] (never answered AFAICT). To conclude, I strongly believe there's no one size fits all here. That was why I was hopeful when I first heard about the Ceph orchestrator idea, when it looked to be planned out to be modular, with the different tasks being implementable in several backends, so one could imagine them being implemented with containers, with classic SSH on bare-metal (i.e. ceph-deploy-like), ansible, rook or maybe others. Sadly, it seems it ended up being "container-only". Containers certainly have many uses, and we run thousands of them daily, but neither do they fit each and every existing requirement, nor are they a magic bullet to solve all issues. Cheers, Oliver [0] https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.cep… [1] But you may also just have a very well structured configuration management system fitting your organizational structure. Am 02.06.21 um 11:36 schrieb Matthew Vernon: -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.047 Nußallee 12 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --