Not using cephadm, I would also question other things
like:
- If it uses docker and docker daemon fails what happens to you containers?
- I assume the ceph-osd containers need linux capability sysadmin. So
if you have to allow this via your OC, all your tasks have potentially
access to this permission. (That is why I chose not to allow the OC
access to it)
- cephadm only runs with docker?
-----Original Message-----
From: Martin Verges <martin.verges(a)croit.io>
Sent: 02 June 2021 13:29
To: Matthew Vernon <mv3(a)sanger.ac.uk>
Cc: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Why you might want packages not containers
for Ceph deployments
Hello,
I agree to Matthew, here at croit we work a lot with containers all
day long. No problem with that and enough knowledge to say for sure
it's not about getting used to it.
For us and our decisions here, Storage is the most valuable piece of
IT equipment in a company. If you have problems with your storage,
most likely you have a huge pain, costs, problems, downtime,
whatever. Therefore, your storage solution must be damn simple, you
switch it on, it has to work.
If you take a short look into Ceph documentation about how to deploy
a cephadm cluster vs croit. We strongly believe it's much easier as
we take away all the pain from OS up to Ceph while keeping it simple
behind the scene. You still can always login to a node, kill a
process, attach some strace or whatever you like as you know it from
years of linux administration without any complexity layers like
docker/podman/... It's just friction less. In the end, what do you
need? A kernel, an initramfs, some systemd, a bit of libs and
tooling, and the Ceph packages.
In addition, we help lot's of Ceph users on a regular basis with
their hand made setups, but we don't really wanna touch the cephadm
ones, as they are often harder to debug. But of course we do it
anyways :).
To have a perfect storage, strip away anything unneccessary. Avoid
any complexity, avoid anything that might affect your system. Keep
it simply stupid.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat:
https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht
Munich HRB 231263
Web:
https://croit.io
YouTube:
https://goo.gl/PGE1Bx
On Wed, 2 Jun 2021 at 11:38, Matthew Vernon <mv3(a)sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was
a
bit
of chat about cephadm / containers / packages.
IIRC, Sage observed
that
a common reason in the recent user survey for not
using cephadm
was
that
it only worked on containerised deployments. I
think he then went
on
to
say that he hadn't heard any compelling
reasons why not to use
containers, and suggested that resistance was essentially a user
education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this
is
not
simply a matter of familiarity
* reducing the complexity of systems makes admins' lives easier
* the trade-off of the pros and cons of containers vs packages is
not obvious, and will depend on deployment needs
* Ceph users will benefit from both approaches being supported
into
the
future
We make extensive use of containers at Sanger, particularly for
scientific workflows, and also for bundling some web apps (e.g.
Grafana). We've also looked at a number of container runtimes
(Docker, singularity, charliecloud). They do have advantages -
it's easy to distribute a complex userland in a way that will run
on (almost) any target distribution; rapid "cloud" deployment;
some separation (via
namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a
set
of
dedicated hardware and then run for a long time),
I'm not sure any
of these benefits are particularly relevant and/or compelling -
Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu
Cloud
Archive) provide .debs of a couple of different Ceph releases per
Ubuntu
LTS - meaning we can easily separate out OS
upgrade from Ceph upgrade.
And upgrading the Ceph packages _doesn't_ restart the daemons[1],
meaning that we maintain control over restart order during an upgrade.
And while we might briefly install packages from a PPA or similar
to test a bugfix, we roll those (test-)cluster-wide, rather than
trying
to
run a mixed set of versions on a single cluster -
and I understand
this
single-version approach is best practice.
Deployment via containers does bring complexity; some examples
we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd
* docker updates (via distribution unattended-upgrades) have an
unfortunate habit of rudely restarting everything
* docker squats on a chunk of RFC 1918 space (and telling it not
to
can
be a bore), which coincides with our internal
network...
* there is more friction if you need to look inside containers
(particularly if you have a lot running on a host and are trying
to
find
out what's going on)
* you typically need to be root to build docker containers (unlike
packages)
* we already have package deployment infrastructure (which we'll
need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph
units
(e.g. to do some network sanity checks before
bringing up an OSD),
and have some tools to pair OSD / journal / LVM / disk device up;
I think these would be more fiddly in a containerised deployment.
I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not
saying "don't ship containers". But I think there is added
complexity to your deployment from going the containers route, and
that is not simply a "learn how to use containers" learning curve.
I do think it is reasonable for an admin to want to reduce the
complexity of what
they're
dealing with - after all, much of my job is
trying to automate or
simplify the management of complex systems!
I can see from a software maintainer's point of view that just
building
one container and shipping it everywhere is
easier than building
packages for a number of different distributions (one of my other
hats is a Debian developer, and I have a bunch of machinery for
doing this sort of thing). But it would be a bit unfortunate if
the general
thrust
of "let's make Ceph easier to set up and
manage" was somewhat
derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a
container-based deployment (and I'm sure there are plenty of
setups where it's an obvious win), but if I were advising someone
who wanted to set up and use a 'boring' Ceph cluster for the
medium term, I'd still advise on using packages. I don't think
this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary!
[1] This hasn't always been true...
[2] Simple (sic.) Matter of Programming
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose
registered office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send
an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io