Debating containers vs packages is like debating systemd vs initrd. There are lots of
reasons why containers (and container orchestration) are good for deploying things,
including ceph. Repeating them in each project every time it comes up is not really
productive. I'd recommend looking at why containers are good in general. It applies to
ceph too.
________________________________________
From: Sasha Litvak <alexander.v.litvak(a)gmail.com>
Sent: Wednesday, June 2, 2021 7:56 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments
Check twice before you click! This email originated from outside PNNL.
Is there a link of the talk I can use as a reference? I would like to
look at the pro container points as this post is getting a little bit one
sided. I understand that most people prefer things to be stable especially
with the underlying storage systems. To me personally, use of
containers in general adds a great flexibility because it
detaches underlying OS from the running software. All points are fair
about adding complexity to the complex system but one thing is missing.
Every time developers decide to introduce some new more efficient libraries
or frameworks we hit a distribution dependency hell. Because of that, ceph
sometimes abandons entire OS versions before their actual lifetime is
over. My resources are limited and I don't want to debug / troubleshoot
/ upgrade OS in addition to ceph itself, hence the containers. Yes it
took me a while to warm up to the idea in general but now I don't even
think too much about it. I went from Nautilus to Pacific (Centos 7 to
Centos 8) within a few hours without needing to upgrade my Ubuntu bare
metal nodes.
This said, I am for giving people a choice to use packages + ansible /
manual install and also allowing manual install of containers. Forcing
users' hands too much may make people avoid upgrading their ceph clusters.
On Wed, Jun 2, 2021 at 9:27 AM Dave Hall <kdhall(a)binghamton.edu> wrote:
I'd like to pick up on something that Matthew
alluded to, although what I'm
saying may not be popular. I agree that containers are compelling for
cloud deployment and application scaling, and we can all be glad that
container technology has progressed from the days of docker privilege
escalation. I also agree that for Ceph users switching from native Ceph
processes to containers carries a learning curve that could be as
intimidating as learning Ceph itself.
However, here is where I disagree with containerized Ceph: I worked for 19
years as a software developer for a major world-wide company. In that
time, I noticed that most of the usability issues experienced by customers
were due to the natural and understandable tendency for software developers
to program in a way that's easy for the programmer, and in the process to
lose sight of what's easy for the user.
In the case of Ceph, containers make it easier for the developers to
produce and ship releases. It reduces dependency complexities and testing
time. But the developers aren't out in the field with their deployments
when something weird impacts a cluster and the standard approaches don't
resolve it. And let's face it: Ceph is a marvelously robust solution for
large scale storage, but it is also an amazingly intricate matrix of
layered interdependent processes, and you haven't got all of the bugs
worked out yet.
Just note that the beauty of software (or really of anything that is
'designed') is that a few people (developers) can produce something that a
large number of people (storage administrators, or 'users') will want to
use.
Please remember the ratio of users (cluster administrators) to developers
and don't lose sight of the users in working to ease and simplify
development.
-Dave
--
Dave Hall
Binghamton University
kdhall(a)binghamton.edu
On Wed, Jun 2, 2021 at 5:37 AM Matthew Vernon <mv3(a)sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit
of chat about cephadm / containers / packages. IIRC, Sage observed that
a common reason in the recent user survey for not using cephadm was that
it only worked on containerised deployments. I think he then went on to
say that he hadn't heard any compelling reasons why not to use
containers, and suggested that resistance was essentially a user
education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not
simply a matter of familiarity
* reducing the complexity of systems makes admins' lives easier
* the trade-off of the pros and cons of containers vs packages is not
obvious, and will depend on deployment needs
* Ceph users will benefit from both approaches being supported into the
future
We make extensive use of containers at Sanger, particularly for
scientific workflows, and also for bundling some web apps (e.g.
Grafana). We've also looked at a number of container runtimes (Docker,
singularity, charliecloud). They do have advantages - it's easy to
distribute a complex userland in a way that will run on (almost) any
target distribution; rapid "cloud" deployment; some separation (via
namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of
dedicated hardware and then run for a long time), I'm not sure any of
these benefits are particularly relevant and/or compelling - Ceph
upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud
Archive) provide .debs of a couple of different Ceph releases per Ubuntu
LTS - meaning we can easily separate out OS upgrade from Ceph upgrade.
And upgrading the Ceph packages _doesn't_ restart the daemons[1],
meaning that we maintain control over restart order during an upgrade.
And while we might briefly install packages from a PPA or similar to
test a bugfix, we roll those (test-)cluster-wide, rather than trying to
run a mixed set of versions on a single cluster - and I understand this
single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've
found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd
* docker updates (via distribution unattended-upgrades) have an
unfortunate habit of rudely restarting everything
* docker squats on a chunk of RFC 1918 space (and telling it not to can
be a bore), which coincides with our internal network...
* there is more friction if you need to look inside containers
(particularly if you have a lot running on a host and are trying to find
out what's going on)
* you typically need to be root to build docker containers (unlike
packages)
* we already have package deployment infrastructure (which we'll need
regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units
(e.g. to do some network sanity checks before bringing up an OSD), and
have some tools to pair OSD / journal / LVM / disk device up; I think
these would be more fiddly in a containerised deployment. I'd accept
that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying
"don't ship containers". But I think there is added complexity to your
deployment from going the containers route, and that is not simply a
"learn how to use containers" learning curve. I do think it is
reasonable for an admin to want to reduce the complexity of what they're
dealing with - after all, much of my job is trying to automate or
simplify the management of complex systems!
I can see from a software maintainer's point of view that just building
one container and shipping it everywhere is easier than building
packages for a number of different distributions (one of my other hats
is a Debian developer, and I have a bunch of machinery for doing this
sort of thing). But it would be a bit unfortunate if the general thrust
of "let's make Ceph easier to set up and manage" was somewhat derailed
with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based
deployment (and I'm sure there are plenty of setups where it's an
obvious win), but if I were advising someone who wanted to set up and
use a 'boring' Ceph cluster for the medium term, I'd still advise on
using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary!
[1] This hasn't always been true...
[2] Simple (sic.) Matter of Programming
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io