Hi,
Correct me if I'm wrong but cephadm uses containers, isn't it? So that one comes
together.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo(a)agoda.com
---------------------------------------------------
-----Original Message-----
From: Matthew Vernon <mv3(a)sanger.ac.uk>
Sent: Wednesday, June 2, 2021 4:36 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Why you might want packages not containers for Ceph deployments
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about
cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent
user survey for not using cephadm was that it only worked on containerised deployments. I
think he then went on to say that he hadn't heard any compelling reasons why not to
use containers, and suggested that resistance was essentially a user education
question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of
familiarity
* reducing the complexity of systems makes admins' lives easier
* the trade-off of the pros and cons of containers vs packages is not obvious, and will
depend on deployment needs
* Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and
also for bundling some web apps (e.g.
Grafana). We've also looked at a number of container runtimes (Docker, singularity,
charliecloud). They do have advantages - it's easy to distribute a complex userland in
a way that will run on (almost) any target distribution; rapid "cloud"
deployment; some separation (via
namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated
hardware and then run for a long time), I'm not sure any of these benefits are
particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical
(via their Ubuntu Cloud
Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we
can easily separate out OS upgrade from Ceph upgrade.
And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we
maintain control over restart order during an upgrade.
And while we might briefly install packages from a PPA or similar to test a bugfix, we
roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a
single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger
(not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd
* docker updates (via distribution unattended-upgrades) have an unfortunate habit of
rudely restarting everything
* docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which
coincides with our internal network...
* there is more friction if you need to look inside containers (particularly if you have a
lot running on a host and are trying to find out what's going on)
* you typically need to be root to build docker containers (unlike packages)
* we already have package deployment infrastructure (which we'll need regardless of
deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some
network sanity checks before bringing up an OSD), and have some tools to pair OSD /
journal / LVM / disk device up; I think these would be more fiddly in a containerised
deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't
ship containers". But I think there is added complexity to your deployment from going
the containers route, and that is not simply a "learn how to use containers"
learning curve. I do think it is reasonable for an admin to want to reduce the complexity
of what they're dealing with - after all, much of my job is trying to automate or
simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container
and shipping it everywhere is easier than building packages for a number of different
distributions (one of my other hats is a Debian developer, and I have a bunch of machinery
for doing this sort of thing). But it would be a bit unfortunate if the general thrust of
"let's make Ceph easier to set up and manage" was somewhat derailed with
"you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and
I'm sure there are plenty of setups where it's an obvious win), but if I were
advising someone who wanted to set up and use a 'boring' Ceph cluster for the
medium term, I'd still advise on using packages. I don't think this makes me a
luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary!
[1] This hasn't always been true...
[2] Simple (sic.) Matter of Programming
--
The Wellcome Sanger Institute is operated by Genome Research Limited, a charity
registered in England with number 1021457 and a company registered in England with number
2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.