Why you might want packages not containers for Ceph deployments
Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
I agree, simplifying "deployment" by adding another layer of complexity does bring much more problems and hard times when something goes wrong in the runtime. Few additional steps at "install phase" and better understanding of underlying architecture, commands, whatever ... have much more pros than cons at the end. Even in the most simplified installation, deployment, systems are complex by itself enough so I have no idea why under the recommended installation method https://docs.ceph.com/en/latest/cephadm/install/ there are lvm, systemd, docker or podman. I suggest that much effort should be put into better and more understandable documentation. Rok On Wed, Jun 2, 2021 at 11:36 AM Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, I agree to Matthew, here at croit we work a lot with containers all day long. No problem with that and enough knowledge to say for sure it's not about getting used to it. For us and our decisions here, Storage is the most valuable piece of IT equipment in a company. If you have problems with your storage, most likely you have a huge pain, costs, problems, downtime, whatever. Therefore, your storage solution must be damn simple, you switch it on, it has to work. If you take a short look into Ceph documentation about how to deploy a cephadm cluster vs croit. We strongly believe it's much easier as we take away all the pain from OS up to Ceph while keeping it simple behind the scene. You still can always login to a node, kill a process, attach some strace or whatever you like as you know it from years of linux administration without any complexity layers like docker/podman/... It's just friction less. In the end, what do you need? A kernel, an initramfs, some systemd, a bit of libs and tooling, and the Ceph packages. In addition, we help lot's of Ceph users on a regular basis with their hand made setups, but we don't really wanna touch the cephadm ones, as they are often harder to debug. But of course we do it anyways :). To have a perfect storage, strip away anything unneccessary. Avoid any complexity, avoid anything that might affect your system. Keep it simply stupid. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx On Wed, 2 Jun 2021 at 11:38, Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Not using cephadm, I would also question other things like: - If it uses docker and docker daemon fails what happens to you containers? - I assume the ceph-osd containers need linux capability sysadmin. So if you have to allow this via your OC, all your tasks have potentially access to this permission. (That is why I chose not to allow the OC access to it) - cephadm only runs with docker?
-----Original Message----- From: Martin Verges <martin.verges@croit.io> Sent: 02 June 2021 13:29 To: Matthew Vernon <mv3@sanger.ac.uk> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments
Hello,
I agree to Matthew, here at croit we work a lot with containers all day long. No problem with that and enough knowledge to say for sure it's not about getting used to it. For us and our decisions here, Storage is the most valuable piece of IT equipment in a company. If you have problems with your storage, most likely you have a huge pain, costs, problems, downtime, whatever. Therefore, your storage solution must be damn simple, you switch it on, it has to work.
If you take a short look into Ceph documentation about how to deploy a cephadm cluster vs croit. We strongly believe it's much easier as we take away all the pain from OS up to Ceph while keeping it simple behind the scene. You still can always login to a node, kill a process, attach some strace or whatever you like as you know it from years of linux administration without any complexity layers like docker/podman/... It's just friction less. In the end, what do you need? A kernel, an initramfs, some systemd, a bit of libs and tooling, and the Ceph packages.
In addition, we help lot's of Ceph users on a regular basis with their hand made setups, but we don't really wanna touch the cephadm ones, as they are often harder to debug. But of course we do it anyways :).
To have a perfect storage, strip away anything unneccessary. Avoid any complexity, avoid anything that might affect your system. Keep it simply stupid.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
On Wed, 2 Jun 2021 at 11:38, Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Podman containers will not restart due to restart or failure of centralized podman daemon. Container is not synonymous to Docker. This thread reminds me systemd haters threads more and more by I guess it is fine. On Thu, Jun 3, 2021, 2:16 AM Marc <Marc@f1-outsourcing.eu> wrote:
Not using cephadm, I would also question other things like:
- If it uses docker and docker daemon fails what happens to you containers? - I assume the ceph-osd containers need linux capability sysadmin. So if you have to allow this via your OC, all your tasks have potentially access to this permission. (That is why I chose not to allow the OC access to it) - cephadm only runs with docker?
-----Original Message----- From: Martin Verges <martin.verges@croit.io> Sent: 02 June 2021 13:29 To: Matthew Vernon <mv3@sanger.ac.uk> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments
Hello,
I agree to Matthew, here at croit we work a lot with containers all day long. No problem with that and enough knowledge to say for sure it's not about getting used to it. For us and our decisions here, Storage is the most valuable piece of IT equipment in a company. If you have problems with your storage, most likely you have a huge pain, costs, problems, downtime, whatever. Therefore, your storage solution must be damn simple, you switch it on, it has to work.
If you take a short look into Ceph documentation about how to deploy a cephadm cluster vs croit. We strongly believe it's much easier as we take away all the pain from OS up to Ceph while keeping it simple behind the scene. You still can always login to a node, kill a process, attach some strace or whatever you like as you know it from years of linux administration without any complexity layers like docker/podman/... It's just friction less. In the end, what do you need? A kernel, an initramfs, some systemd, a bit of libs and tooling, and the Ceph packages.
In addition, we help lot's of Ceph users on a regular basis with their hand made setups, but we don't really wanna touch the cephadm ones, as they are often harder to debug. But of course we do it anyways :).
To have a perfect storage, strip away anything unneccessary. Avoid any complexity, avoid anything that might affect your system. Keep it simply stupid.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
On Wed, 2 Jun 2021 at 11:38, Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dera Sasha and for everyone else as well, Sasha Litvak <alexander.v.litvak@gmail.com> writes:
Podman containers will not restart due to restart or failure of centralized podman daemon. Container is not synonymous to Docker. This thread reminds me systemd haters threads more and more by I guess it is fine.
calling people "haters" who have reservations or critique is simply wrong, harmful and disrespectful. I ask you to retract above statement immediately for the sake of the ceph and open source community. Nico -- Sustainable and modern Infrastructures by ungleich.ch
Dear Nico, do you think it is sensible and it's a precise statement saying that "we can't reduce complexity by adding a layer of complexity"? Containers are always adding a so-called layer, but people keep using them, and in some cases, they offload complexity from another side. Claiming the "complexity", without explaining examining the details is pure FUD. "complexity, OMG!!!111!!!" is not enough of a statement. You have to explain what complexity you gain and what complexity you reduce. Installing SeaweedFS consists of the following: `cd seaweedfs/weed && make install` This is the type of problem that Ceph is trying to solve, and starting a discussion by saying that everything is bad, without providing any helpful message is useless FUD. Max
"complexity, OMG!!!111!!!" is not enough of a statement. You have to explain what complexity you gain and what complexity you reduce. Installing SeaweedFS consists of the following: `cd seaweedfs/weed && make install` This is the type of problem that Ceph is trying to solve, and starting a discussion by saying that everything is bad, without providing any helpful message is useless FUD.
Max
Max stop eating seaweed
My Cephadm deployment on RHEL8 created a service for each container, complete with restarts. And on the host, the processes run under the 'ceph' user account. The biggest issue I had with running as containers is that the unit.run script generated runs podman -rm ... with the -rm, the logs are removed when there is an issue, so it took extra effort to figure out what the issue I was having on one machine was. (Although that turned out to be a bad memory chip, which would have manifested itself anyway). As a manager of a team who develops microservices in containers, I have a mixed attitude towards them - the ability to know the version of Java, and support libraries deployed for a given container, and isolate updates one image at a time can be a bonus, but with my client's CI/CD pipeline and how they want our containers to be built, simple tasks like upgrading a package version, upgrading the version of Java or even a simple replacement of a certificate has become significantly more difficult, because we need to rebuild all of the containers and go through the QA processes rather than update a cert. For my usage, (At home running Ceph on older hardware I converted to servers) I don't want to have to care about ceph dependencies and also isolate things from other things running on the server, so a container infrastructure works well, but I can see where packages can be much better in a well maintained server infrastructure. -----Original Message----- From: Sasha Litvak <alexander.v.litvak@gmail.com> Sent: Thursday, June 3, 2021 3:57 AM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Podman containers will not restart due to restart or failure of centralized podman daemon. Container is not synonymous to Docker. This thread reminds me systemd haters threads more and more by I guess it is fine. On Thu, Jun 3, 2021, 2:16 AM Marc <Marc@f1-outsourcing.eu> wrote:
Not using cephadm, I would also question other things like:
- If it uses docker and docker daemon fails what happens to you containers? - I assume the ceph-osd containers need linux capability sysadmin. So if you have to allow this via your OC, all your tasks have potentially access to this permission. (That is why I chose not to allow the OC access to it) - cephadm only runs with docker?
-----Original Message----- From: Martin Verges <martin.verges@croit.io> Sent: 02 June 2021 13:29 To: Matthew Vernon <mv3@sanger.ac.uk> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments
Hello,
I agree to Matthew, here at croit we work a lot with containers all day long. No problem with that and enough knowledge to say for sure it's not about getting used to it. For us and our decisions here, Storage is the most valuable piece of IT equipment in a company. If you have problems with your storage, most likely you have a huge pain, costs, problems, downtime, whatever. Therefore, your storage solution must be damn simple, you switch it on, it has to work.
If you take a short look into Ceph documentation about how to deploy a cephadm cluster vs croit. We strongly believe it's much easier as we take away all the pain from OS up to Ceph while keeping it simple behind the scene. You still can always login to a node, kill a process, attach some strace or whatever you like as you know it from years of linux administration without any complexity layers like docker/podman/... It's just friction less. In the end, what do you need? A kernel, an initramfs, some systemd, a bit of libs and tooling, and the Ceph packages.
In addition, we help lot's of Ceph users on a regular basis with their hand made setups, but we don't really wanna touch the cephadm ones, as they are often harder to debug. But of course we do it anyways :).
To have a perfect storage, strip away anything unneccessary. Avoid any complexity, avoid anything that might affect your system. Keep it simply stupid.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
On Wed, 2 Jun 2021 at 11:38, Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, Jun 3, 2021 at 2:18 AM Marc <Marc@f1-outsourcing.eu> wrote:
Not using cephadm, I would also question other things like:
- If it uses docker and docker daemon fails what happens to you containers?
This is an obnoxious feature of docker; podman does not have this problem.
- I assume the ceph-osd containers need linux capability sysadmin. So if you have to allow this via your OC, all your tasks have potentially access to this permission. (That is why I chose not to allow the OC access to it)
The --privileged flag and other caps are only passed on an as-needed basis, based on the daemon (each daemon runs in its own container). (There is only one container image, though.) sage
I totally agree - we use a management system to manage all our Linux machines. Adding containers just makes that a lot more complex, especially since our management system does not support containers. Regards magnus On Wed, 2021-06-02 at 10:36 +0100, Matthew Vernon wrote:
This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe.
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container- based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
Hi, Correct me if I'm wrong but cephadm uses containers, isn't it? So that one comes together. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com --------------------------------------------------- -----Original Message----- From: Matthew Vernon <mv3@sanger.ac.uk> Sent: Wednesday, June 2, 2021 4:36 PM To: ceph-users@ceph.io Subject: [ceph-users] Why you might want packages not containers for Ceph deployments Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hi,
* Ceph users will benefit from both approaches being supported into the future
this is rather important for us as well. we use systemd-nspawn based containers (that act and are managed like traditional VMs, just without the overhead). cephadm enforces not just containers, but particular ones (granted, docker/podman are the currently most used container-runtimes). it would be nice if cephadm would, like cephdeploy did, support "traditional"/plain systems (which then automatically works perfectly for us with systemd-nspawn without any changes). until now we've still used ceph-deply and stayed away from cephadm, as it would require us to completely change our system deployment. Regards, Daniel
On Wed, Jun 2, 2021 at 9:01 AM Daniel Baumann <daniel.baumann@bfh.ch> wrote:
* Ceph users will benefit from both approaches being supported into the future
this is rather important for us as well.
we use systemd-nspawn based containers (that act and are managed like traditional VMs, just without the overhead).
cephadm enforces not just containers, but particular ones (granted, docker/podman are the currently most used container-runtimes).
I'm not familiar with systemd-nspawn, but maybe it could be supported. I suspect the primary challenge would be making it consume the OCI container image, or producing some alternative build artifact that takes its place. cephadm controls everything via systemd units, so in that regard the experience should be consistent either way. sage
Following up with some general comments on the main container downsides and on the upsides that led us down this path in the first place. Aside from a few minor misunderstandings, it seems like most of the objections to containers boil down to a few major points:
Containers are more complicated than packages, making debugging harder.
I think that part of this comes down to a learning curve and some semi-arbitrary changes to get used to (e.g., systemd unit name has changed; logs now in /var/log/ceph/$fsid instead of /var/log/ceph). Another part of these changes are real hoops to jump through: to inspect process(es) inside a container you have to `cephadm enter --name ...`; ceph CLI may not be automatically installed on every host; stracing or finding coredumps requires extra steps. We're continuing to improve the tools etc so please call these things out as you see them!
Security (50 containers -> 50 versions of openssl to patch)
This feels like the most tangible critique. It's a tradeoff. We have had so many bugs over the years due to varying versions of our dependencies that containers feel like a huge win: we can finally test and distribute something that we know won't break due to some random library on some random distro. But it means the Ceph team is on the hook for rebuilding our containers when the libraries inside the container need to be patched. On the flip side, cephadm's use of containers offer some huge wins: - Package installation hell is gone. Previously, ceph-deploy and ceph-ansible had thousands of lines of code to deal with the myriad ways that packages could be installed and where they could be published. With containers, this now boils down to a single string, which is usually just something like "ceph/ceph:v16". We're grown a handful of complexity there to let you log into private registries, but otherwise things are so much simpler. Not to mention what happens when package dependencies break. - Upgrades/downgrades can be carefully orchestrated. With packages, the version change is by host, with a limbo period (and occasional SIGBUS) before daemons were restarted. Now we can run new or patched code on individual daemons and avoid an accidental upgrade when a daemon restarts. (Also, running e.g. ceph CLI commands no longer error out with a dynamic linker error while the package upgrade itself is in progress, something all of our automated upgrade tests have to carefully avoid to prevent intermittent failures.) - Ceph installations are carefully sandboxed. Removing/scrubbing ceph from a host is trivial as only a handful of directories or configuration files are touched. And we can safely run multiple clusters on the same machine without worry about bad interactions (mostly great for development, but also handy for users experimenting with new features etc). - Cephadm deploys a bunch of non-ceph software as well to provide a complete storage system, including haproxy and keepalived for HA ingress for RGW and NFS, ganesha for NFS service, grafana, prometheus, node-exporter, and (soon) samba for SMB. All neatly containerized to avoid bumping into other software on the host; testing and supporting the huge matrix of packages versions available via various distros would be a huge time sink. Most importantly, cephadm and the orchestrator API vastly improve the overall ceph experience from the CLI and dashboard. Users no longer have to give any thought to where and which daemons run if they don't want to (or they can carefully specify daemon placement if they choose). And users can use commands like 'ceph fs volume create foo' and the fs will get created *and* MDS daemons will be started all in one go. (This would also be possible with a package-based orchestrator implementation if one existed.) We've been beat up for years about how complicated and hard Ceph is. Rook and cephadm represent two of the most successful efforts to address usability (and not just because they enable deployment management via the dashboard!), and taking advantage of containers was one expedient way to get to where we needed to go. If users feel strongly about supporting packages, we can get much of the same experience with another package-based orchestrator module. My view, though, is that we have much higher priority problems to tackle. sage
Thanks, Sage. This is a terrific distilation of the challenges and benefits. FWIW here are a few of my own perspectives, as someone experienced with Ceph but with limited container experience. To be very clear, these are *perceptions* not *assertions*; my goal is discussion not argument. For context, I have not used a release newer than Nautilus in production, in large part due to containers and cephadm.
Containers are more complicated than packages, making debugging harder.
I think that part of this comes down to a learning curve and some semi-arbitrary changes to get used to (e.g., systemd unit name has changed; logs now in /var/log/ceph/$fsid instead of /var/log/ceph).
Indeed, if there are logs at all. It seems that (by default?) one has to (know to) use journalctl to extract daemon or cluster logs, which is rather awkward compared to having straight files. And those go away when the daemon restarts or is redeployed, losing data continuity? Is logrotate used as usual, such that it can be adjusted? If running multiple clusters on a set of hardware is deprecated, why include the fsid in the pathname? This complicates scripting and monitoring / metrics collection. Or have we retconned multiple clusters? The admin sockets are under a similar path in /var/run. I have yet to discover an incantation of `ceph daemon mon.foo` eg. that works, indeed specifying the whole path to the asok yields an error about the path being too long, so I’ve had to make a symlink to it. This isn’t great usability, unless of course I’m missing something.
Security (50 containers -> 50 versions of openssl to patch)
This feels like the most tangible critique. It's a tradeoff. We have had so many bugs over the years due to varying versions of our dependencies that containers feel like a huge win: we can finally test and distribute something that we know won't break due to some random library on some random distro. But it means the Ceph team is on the hook for rebuilding our containers when the libraries inside the container need to be patched.
This seems REALLY congruent with the tradeoffs that accompanied shared/dynamic linking years ago. Shared linking saves on binary size and facilitates sharing of address space among processes; dynamic shared linking lets one update dependencies (notably openssl for sure since it has had lots of exploits over time, but others too). But that also means that changes to those libraries can break applications. So we’ve long seen commercial / pre-built binaries statically linked to avoid regression and breakage. Kind of a rock and a hard place situation. Some assert that Ceph daemon systems should mostly or entirely inaccessible from the Internet, and usually don’t have a large set of users — or any customers — logging into them. Thus it can be argued that they are less exposed to attacks which would favor somewhat containerization. One might say that containerization and orchestration make updates for security fixes trivial, but remember that in most cases such an upgrade is not against the immediately prior Ceph dot release, which means exposure to regressions and other unanticipated changes in behavior. Which is one reason why enterprises especially may stick with a given specific dot release that works until compelled. Updating upstream containers for security fixes is right back into the dependency hell situation too.
On the flip side, cephadm's use of containers offer some huge wins:
- Package installation hell is gone.
As a user I never experienced much of this, but then I was mostly installing packages outside of ceph-deploy et al. With at least 3 different container technologies in play, though, are we substiuting one complexity for another?
- Upgrades/downgrades can be carefully orchestrated. With packages, the version change is by host, with a limbo period (and occasional SIGBUS) before daemons were restarted. Now we can run new or patched code on individual daemons and avoid an accidental upgrade when a daemon restarts.
Fair enough - that limbo period was never a problem for me, but re careful orchestration, we see people on this list all the time experiencing orchestration failures. Is the list a nonrepresentative sample of people’s experience? The opacity of said orchestration also complicates troubleshooting.
- Ceph installations are carefully sandboxed. Removing/scrubbing ceph from a host is trivial as only a handful of directories or configuration files are touched.
Plus of course any ancillary tools. This seems like it would be advantageous in labs. In production it’s not uncommon to reimage the entire box anyway.
And we can safely run multiple clusters on the same machine without worry about bad interactions
Wasn’t it observed a few years ago that almost nobody actually did that, hence the deprecation of custom cluster names?
- Cephadm deploys a bunch of non-ceph software as well to provide a complete storage system, including haproxy and keepalived for HA ingress for RGW and NFS, ganesha for NFS service, grafana, prometheus, node-exporter, and (soon) samba for SMB. All neatly containerized to avoid bumping into other software on the host; testing and supporting the huge matrix of packages versions available via various distros would be a huge time sink.
One size fits all? None? Many? Some? Does that get in the way of sites that, eg. choose nginux for LB/HA, to run their own Prometheus / Grafana infra for various reasons? Is this more of the
We've been beat up for years about how complicated and hard Ceph is.
True. I was told in an interview once that one needs a PhD in Ceph. Over the years operators have had to rework tooling with every release, so the substantial retoolings that come with containers and cephadm / ceph orch can be daunting. Midstream changes and changes made for no apparent reason contribute to the perception. JSON output is supposed to be invariant, or at least backward compatible, yet we saw mon clock skew move for no apparent reason, and there have been other breaking changes. cf. the ceph_exporter source for more examples.
Rook and cephadm represent two of the most successful efforts to address usability (and not just because they enable deployment management via the dashboard!),
The goals here are totally worthy, to make things more turnkey. I get that, I really do. There are some wrinkles though: * Are they successful, though? I’m not saying they aren’t, I’m asking. The frequency of cephadm / ceph orch SNAFUs posted to this list is daunting. It seemed at one point that Rook would become the party line, but now it’s heterodox? * Removing other complexity by introducing new complexity (containers). There seems to have been assumption here that operators already grok containers? In any of the three+ flavors in play? It’s easy to just dismiss this as a learning curve, but it’s a rather significant one, and assuming that the operator will do that in their Copious Free Time isn’t IMHO reasonable. * Dashboard operation by pushing buttons can make it dead simple to deploy a single dead simple configuration, but revision-controlled management of dozens of clusters is a different story. Centralized config is one example (assuming the subtree limit bug has been fixed). Absolutely mangaging ceph.conf across system types and multiple clusters is a pain — brittle ERB or J2 templates, inscrutable Ansible errors. But how does one link CLI-based centralized config with revision control and peer review of changes? One thing about turnkey solutions is that in generaly they usually are unreasonably simplistic or rigid in ways that are a bad fit for manageable enterprise deployment, and if we’re going to do everything for the user *and* make it diffcult for them to dig deep or customize, then the bar is *very* high for success. * Some might add ceph-ansible to that list.
Hello Sage,
...I think that part of this comes down to a learning curve... ...cephadm represent two of the most successful efforts to address usability...
Somehow it does not look right to me. There is much more to operate a Ceph cluster than just deploying software. Of course that helps on the short run to avoid that people leave the train right when they started their Ceph journey. But the harder part is what to do if shit hit's the fan and your cluster is down due to some issues and then having additional layers of complexity kicking in and biting your ass. Just saying, that day2 ops is much more important than getting a cluster up&running. In my believe, no admin want to dig around containers and other abstractions when the single most important part of a whole IT infrastructure stops working. But just my thought, maybe I'm wrong. In my opinion, the best possible way to run IT software is KISS, keep it stupid simple. No additional layers, no abstractions of abstractions and good error messages. For example the docker topic here looks like something that can be showcased:
Question: If it uses docker and docker daemon fails what happens to you containers? Answer: This is an obnoxious feature of docker
As you might see, you need a lot of knowledge about abstraction layers to operate them well. Docker for example provides so called live-restore ( https://docs.docker.com/config/containers/live-restore/) that allows you to stop the daemon without killing your containers. This enables you to update docker daemon without downtimes but you have to know it and of course enable it. This can make operating a Ceph cluster harder, not easier. What about more sophisticated features, for example performance. Ceph already is not a fast storage solution with way to high latency. Does it help to add containers instead of going more direct to the hardware and reduce overhead? Of course you can run SPDK and/or DPDK inside containers, but does it make it better and faster or even easier? If you need high-performance storage today, you can turn to open source alternatives that are massively cheaper per IO and only minimally more expensive per GB. I therefore believe, stripping out overhead is also an important topic for the future of Ceph. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx On Fri, 18 Jun 2021 at 20:43, Sage Weil <sage@newdream.net> wrote:
Following up with some general comments on the main container downsides and on the upsides that led us down this path in the first place.
Aside from a few minor misunderstandings, it seems like most of the objections to containers boil down to a few major points:
Containers are more complicated than packages, making debugging harder.
I think that part of this comes down to a learning curve and some semi-arbitrary changes to get used to (e.g., systemd unit name has changed; logs now in /var/log/ceph/$fsid instead of /var/log/ceph). Another part of these changes are real hoops to jump through: to inspect process(es) inside a container you have to `cephadm enter --name ...`; ceph CLI may not be automatically installed on every host; stracing or finding coredumps requires extra steps. We're continuing to improve the tools etc so please call these things out as you see them!
Security (50 containers -> 50 versions of openssl to patch)
This feels like the most tangible critique. It's a tradeoff. We have had so many bugs over the years due to varying versions of our dependencies that containers feel like a huge win: we can finally test and distribute something that we know won't break due to some random library on some random distro. But it means the Ceph team is on the hook for rebuilding our containers when the libraries inside the container need to be patched.
On the flip side, cephadm's use of containers offer some huge wins:
- Package installation hell is gone. Previously, ceph-deploy and ceph-ansible had thousands of lines of code to deal with the myriad ways that packages could be installed and where they could be published. With containers, this now boils down to a single string, which is usually just something like "ceph/ceph:v16". We're grown a handful of complexity there to let you log into private registries, but otherwise things are so much simpler. Not to mention what happens when package dependencies break. - Upgrades/downgrades can be carefully orchestrated. With packages, the version change is by host, with a limbo period (and occasional SIGBUS) before daemons were restarted. Now we can run new or patched code on individual daemons and avoid an accidental upgrade when a daemon restarts. (Also, running e.g. ceph CLI commands no longer error out with a dynamic linker error while the package upgrade itself is in progress, something all of our automated upgrade tests have to carefully avoid to prevent intermittent failures.) - Ceph installations are carefully sandboxed. Removing/scrubbing ceph from a host is trivial as only a handful of directories or configuration files are touched. And we can safely run multiple clusters on the same machine without worry about bad interactions (mostly great for development, but also handy for users experimenting with new features etc). - Cephadm deploys a bunch of non-ceph software as well to provide a complete storage system, including haproxy and keepalived for HA ingress for RGW and NFS, ganesha for NFS service, grafana, prometheus, node-exporter, and (soon) samba for SMB. All neatly containerized to avoid bumping into other software on the host; testing and supporting the huge matrix of packages versions available via various distros would be a huge time sink.
Most importantly, cephadm and the orchestrator API vastly improve the overall ceph experience from the CLI and dashboard. Users no longer have to give any thought to where and which daemons run if they don't want to (or they can carefully specify daemon placement if they choose). And users can use commands like 'ceph fs volume create foo' and the fs will get created *and* MDS daemons will be started all in one go. (This would also be possible with a package-based orchestrator implementation if one existed.)
We've been beat up for years about how complicated and hard Ceph is. Rook and cephadm represent two of the most successful efforts to address usability (and not just because they enable deployment management via the dashboard!), and taking advantage of containers was one expedient way to get to where we needed to go. If users feel strongly about supporting packages, we can get much of the same experience with another package-based orchestrator module. My view, though, is that we have much higher priority problems to tackle.
sage _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Good evening, as an operator running Ceph clusters based on Debian and later Devuan for years and recently testing ceph in rook, I would like to chime in to some of the topics mentioned here with short review: Devuan/OS package: - Over all the years changing from Debian to Devuan, changing the Devuan versions, dist-upgrading - we did not encounter a single issues on the OS basis. The only real problems where, when ceph version incompatibilities between the major versions happened. However this will not change with containers. I do see the lack of proper packages for Alpine Linux, which would be an amazing lean target for running ceph. The biggest problem I see is that ceph/cephadm is the longer the more relying on systemd and that actually locks out folks. Cephadm/Containers: - I am sceptical about this approach to be honest. On one hand yes, putting it into a controlled environment, great. But there are two things that make me really wonder if this is a good approach: - ceph-deploy and friends have been quite a pain over the years and we had to reverse engineer scripts and data to find out how to create OSDs, because in many situations ceph-deploy would fail doing the right thing [tm]. So what is different now with cephadm that the story does not repeat again? - Re-inventing the deployment/distribution system which is basically given by kubernetes. I see many overlaps here and I guess the motivation was "do not force people going into kubernetes" (... but bind them to systemd only operating systems) rook/kubernetes: - If we have to go towards containers, rook seems to be the most plausible solution out there, as it is k8s native, it has solved things like autodetection of disks, restart of daemons by k8s is given, etc. - So if going full blown containers, you might as well go down the road and settle on rook only. Both rook and cephadm come with the obvious disadvantage that the underlying complexity grows quite a lot. And that is a problem, the entrance barrier for ceph just goes up. It's not about understanding concepts of mon, mgr, mds, osd anymore, it's about "how to debug a containerisation deployment framework" and on top of that you begin to understand ceph, effectively adding resources on top of running a ceph cluster. Don't get me wrong, there are good use cases for k8s and we are the longer the more going towards that, but I really don't see the advantage of the in-between approach. Thus my suggestion for the ceph team is to focus on 2 out of the three variants: - Keep providing a native, even manual deployment mode. Let people get an understanding of ceph, develop even their own tooling around it. This enables distros, SMEs, Open Source communities, hackers, developers. Low entrance barrier, easy access, low degree of automation. - For those who are into containers, advise them how to embrace k8s. How to use k8s on bare metal. Is it potentially even smarter to run ceph on IPv6 only clusters? What does the architecture look like with k8s? How does rook do autodetection, what metrics can the kube-prometheus grafana help with? etc. etc. The whole shebang that you'll need to develop and create over time anyway. Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch
On Sat, Jun 19, 2021 at 3:43 PM Nico Schottelius <nico.schottelius@ungleich.ch> wrote:
Good evening,
as an operator running Ceph clusters based on Debian and later Devuan for years and recently testing ceph in rook, I would like to chime in to some of the topics mentioned here with short review:
Devuan/OS package:
- Over all the years changing from Debian to Devuan, changing the Devuan versions, dist-upgrading - we did not encounter a single issues on the OS basis. The only real problems where, when ceph version incompatibilities between the major versions happened. However this will not change with containers.
I do see the lack of proper packages for Alpine Linux, which would be an amazing lean target for running ceph.
The biggest problem I see is that ceph/cephadm is the longer the more relying on systemd and that actually locks out folks.
I want to reiterate that while cephadm requirements are systemd+lvm+python3+containers, the orchestration framework does not have any of these limitations, and is designed to allow you to plug in other options.
[...]
Thus my suggestion for the ceph team is to focus on 2 out of the three variants:
- Keep providing a native, even manual deployment mode. Let people get an understanding of ceph, develop even their own tooling around it. This enables distros, SMEs, Open Source communities, hackers, developers. Low entrance barrier, easy access, low degree of automation.
- For those who are into containers, advise them how to embrace k8s. How to use k8s on bare metal. Is it potentially even smarter to run ceph on IPv6 only clusters? What does the architecture look like with k8s? How does rook do autodetection, what metrics can the kube-prometheus grafana help with? etc. etc. The whole shebang that you'll need to develop and create over time anyway.
Cephadm is intended to be the primary non-k8s option, since it seems pretty clear that there is a significant (huge?) portion of the user commuity that is not interested in adding kubernetes underneath their storage (take all of the "containers add complex" arguments and * 100). We used containers because, in our view, it simplified the developer AND user experience. But neither rook nor cephadm preclude deploying Ceph the traditional way. The newer capabilities in the dashboard to manage the deployment of Ceph relies on the orchestrator API, so a traditional deployment today cannot make use of these new features, but nothing is preventing a non-container-based orchestrator implementation. sage
On 6/18/21 8:42 PM, Sage Weil wrote:
We've been beat up for years about how complicated and hard Ceph is. Rook and cephadm represent two of the most successful efforts to address usability (and not just because they enable deployment management via the dashboard!), and taking advantage of containers was one expedient way to get to where we needed to go. If users feel strongly about supporting packages, we can get much of the same experience with another package-based orchestrator module. My view, though, is that we have much higher priority problems to tackle.
Regardless of the way Ceph is deployed, it will still stay complicated / hard, not? Ceph is a complex piece of many different pieces of software, working together in a distributed fashion. At least from my non-dev / admin / user point of view. So it might also become a huge trap for (naive) users: I can deploy Ceph in a handful of clicks (project aquarium from SUSE for example)*, but if it breaks, you still need to know how to fix it. And therefore it often helps to understand the workings of Ceph, and the OS, and containers, and networking and what not. Something they have not needed to learn beforehand. So ... maybe we also need to make sure that besides making easier to deploy, it should also be easier to troubleshoot Ceph. As long as the container runtime / orchestrator still works, containers might make "self healing" easier. If it does not ... it might be (way) harder. * don't get me wrong, I think it's an awesome project. Gr. Stefan
Am 18.06.21 um 20:42 schrieb Sage Weil:
Following up with some general comments on the main container downsides and on the upsides that led us down this path in the first place. [...]
Thanks, Sage, for the nice and concise summary on the Cephadm benefits, and the reasoning on why the path was chosen! Also thanks for your reply on my question about the modularity of the actual orchestrator. I really appreciate this, and will try to reply in one place here. After the huge activity in this thread, I did take a step back to watch and also make up my mind, trying to condense my main issues with the "containers-only" approach, also taking other replies into account. I hope this is not seen as a rant, but rather a collection of arguments for an additional orchestrator module, or maybe even something different. Unfortunately, it has become a wall of text, but I hope at least some will fight their way through. First of all, I fully agree with the positive points you raised — it's surely a gain for devs and many users to ship something tested and "complete" without having a full and still necessarily incomplete OS test matrix to constantly check and extend. It also eases testing especially when trying out experimental features, and takes away usage complexity e.g. in the upgrade path. Of course, there's also the point that having a large test matrix across OSs tends to uncover actual bugs or issues which may not show up in a reduced test environment[0], so reducing the matrix also comes at a price for reliability which has to be weighed against the time which is saved. The security issue (50 containers -> 50 versions of openssl to patch) also still stands — the earlier question on this list (when to expect patched containers for a CVE affecting a library) is still unreplied to[1], so these are real-life concerns. In general, I don't know any project which ever managed to keep up with the workload caused by the requirement to follow all CVEs of all dependencies, informing about them and patching them, since this is a workload comparable to the one the security teams of Linux distributions have to handle. In addition, you'll also need to address the question of when and how to pull new images when patched containers become available, how / when to inform the administrator, and orchestrate service restarts as-needed (you'd basically need "needs-restarting" and friends). That's still quite a way to go, and will be a constant developer effort from now on. That being said, Ceph may be the first ever project managing to fulfil expectations here due to the close coupling to those guys wearing red hats ;-). Another point raised on this list is that some users are anxious about pushing a "magic" button which upgrades a whole cluster. Sure, this button is super useful, and incorporates developer wisdom, and allows the developers to test the full sequence and ship it to everybody. So these buttons (e.g. "ceph orch upgrade") are something which is useful and, I should say, important. However, by design, it hides the "inner workings" of Ceph, which is a major drawback for some users. After this introduction, let me come to my main personal concern: Loss of integratibility. Our model of operation is to have all machines (anything, be it your off-the-shelf desktop, a laptop, a hypervisor, a compute node or a Ceph node) handled by the very same configuration management. It means all configuration is self-documenting, reinstallation is done with the push of a button, and anybody who understands the configuration management and the services at a basic level can take over operations. It's the only way we _can_ operate, given the huge number of services requested and required in the IT business these days. To give just one example: We mount kerberised NFS on all our desktop nodes, via CephFS exposed via nfs-ganesha. The desktops run Ubuntu/Debian, the file servers CentOS. If I need to change the Kerberos configuration (order of KDCs, roll out new principals etc.), for us this is a change in a single place: We perform the change in Puppet, wait 30 minutes, and all systems run the new configuration[2]. When operating a service with its own orchestrator, this means for me: I have to manually adapt the configuration of this service. I need someone who is able to do that (i.e. two configuration systems have to be learnt), and who then does it for all instances of the service (e.g. all Ceph clusters). Hence, a previously simple change is multiplied in complexity. I can't just replace the OS disk of a Ceph-OSD node and push "reinstall" anymore, letting Puppet install all services and Ceph packages (so I only have to adopt the disks, this is not automated as a safety precaution), but I also have to talk to the Ceph orchestrator. So automation is a must, and we also heavily rely on containers for scientific workloads to offer a large variety of software stacks to our users. Automation works brilliantly on Linux (and likely also similar open platforms), since almost all services can be combined as building blocks, and controlled by our configuration management Puppet (or any other tool). It's basically a consequence of the Unix idea that all programs do their own thing well, and can be "glued" together as-needed for site-specific requirements. While the "cephadm with containers" approach does this kind of glueing for me, it makes it harder (impossible?) to integrate as-is into existing configuration management systems. I think this is also a strong point, and reading through the list, one of the major reasons why larger ceph sites do not want to use cephadm with containers seems to be exactly that: It can't easily be "cut into pieces" and be integrated with an existing system they use for all other infrastructure. So in summary: The orchestrator is a good thing, but my point is that the currently implemented solution is not the right solution for a noticeable fraction of the existing community. In addition to the users you had in mind when designing the current orchestrator, there are also many active users of Ceph who want to have more direct access to the "complexity" of the system for two reasons[3]: - To integrate it into existing automations. - To learn how things work and interact. The latter point is also a strong one, especially for me as an experimental physicist: I learned to love Ceph exactly because I played with the different components and their interactions, tried to break them, found how they react, and got a deeper understanding and knowledge to understand any future issues. This love never develops for me when I use a more "polished" product which has buttons doing things for me. It's a major reason why I'd even choose Ceph(FS) over G*FS if the latter was available for free (of course, there are many more reasons). So my conclusion is: The chosen path is changing the audience of Ceph, affecting both existing users, but also new users. The length of this thread has shown that the community has different opinions on this path forward, for many different reasons. My personal feeling is that the solution embraces new users who want to set something up quickly, and is mainly a problem for existing and future long-term production users with larger clusters who want to understand the full stack and integrate it into their environment. Does it really free developer resources in the long run? I'm not sure about that — the community may shift more to users reporting issues like "I pushed the green upgrade button, and now it is stuck" (similar mails have arrived on the mailing list already), and have less reports from users who report stacktraces or issues in network communication between services (will automated crashdump reports be able to replace experienced bug reporters?). My personal feeling is that the latter type of users are those who stay with Ceph for years or decades, and even though they may seem to be complaining that Ceph is a complicated machine with many nuts and bolts, it's usually not something which disturbs them as much as it may seem to be the case. So finally, what is my idea about a path forward? In addition to keep delivering packages (will they stay?), I see two ways, basically: - Having an additional orchestrator module running on bare-metal. Given the assumptions above and mails on this list, most users who'd use it would in any case install their packages differently, and only use the orchestrator e.g. to distribute cephx keys, set up initial configuration, enable systemd units etc., finally persisting a differing fraction of these things into their configuration management. So the orchestrator may even be useful for them without the actual complex capabilities of adding repositories and installing packages. At least, that's what I'd be happy about: An orchestrator doing all the Ceph-only things for me, ideally telling me what it does. - Having really extensive manual installation instructions. Currently, I find these super useful for the basic first steps, but they basically break off after you have a mon, mgr and osd. In essence, these two points are the same: The modular orchestrator already has all tasks coded inside, i.e. essentially, the orchestrator is the most up-to-date, complete, tested and well-maintained manual documentation we'll get — correct? This is why my personal feeling is that having a "bare metal orchestrator" (i.e. an "SSH orchestrator" like ceph-deploy) even without the features of adding repositories, installing packages, or upgrading at the push of a button will be sufficient for those of us having issues with the current solution. It's essentially about making the manual installation instructions into an orchestrator module (which is probably close to the current cephadm minus containers). Is this a scope which is sufficiently low-hanging to warrant the effort? It may also be less work than having a more extensive manual installation documentation, and maintaining it? Cheers (and congratulations to all who made it to the end of this mail), Oliver [0] As a basic example, g++ gets increasingly better at warning about potentially unintended behaviour caused by common classes of bugs in code, so a newer g++ may find more issues when testing. The same may happen for different library versions, which may point out API usage bugs early on, or reveal issues earlier in testing. [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/PPLJIHT6WKY... [2] Of course, you can (and should) use a staged rollout. [3] Well, that's a presumption, but the fact that you mentioned user concerns about this in the survey seems to strengthen that point. -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.047 Nußallee 12 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --
The security issue (50 containers -> 50 versions of openssl to patch) also still stands — the earlier question on this list (when to expect patched containers for a CVE affecting a library)
I assume they use the default el7/el8 as a base layer, so when that is updated, you will get the updates. However redeploying tasks is not the same as just giving them a restart.
is still unreplied to[1], so these are real-life concerns. In general, I don't know any project which ever managed to keep up with the workload caused by the requirement to follow all CVEs of all dependencies, informing about them and patching them, since this is a workload comparable to the one the security teams of Linux distributions have to handle.
Indeed this is the core business of a distro that you choose. No software solution should ever make it theirs. Eg. this DCOS is just a binary blob of a centos release, from which you have no idea if it is up to date or not, I do not get why people install it.
Cheers (and congratulations to all who made it to the end of this mail),
I think your text clearly summarizes the point of view of many here.
Consider this a +1 for pretty much every point that Matthew made, and articulated much better than I would have. I wholly understand the goal with making ceph more approachable for people learning it, and I think that is a great cause and deserves attention. I also think Tim Serong's Linux.conf.au talk "Containers are hideously undebuggable black boxes and we never should have invented them <https://www.youtube.com/watch?v=pPZsN_urpqw>" was spot on with this problem scenario, in that debugging is a very different path with multiple layers of abstraction. He obviously details the solution(s), but I think it highlights a lot of the friction with moving to container based ceph as compared to package based ceph. Reed
On Jun 2, 2021, at 4:36 AM, Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'd like to pick up on something that Matthew alluded to, although what I'm saying may not be popular. I agree that containers are compelling for cloud deployment and application scaling, and we can all be glad that container technology has progressed from the days of docker privilege escalation. I also agree that for Ceph users switching from native Ceph processes to containers carries a learning curve that could be as intimidating as learning Ceph itself. However, here is where I disagree with containerized Ceph: I worked for 19 years as a software developer for a major world-wide company. In that time, I noticed that most of the usability issues experienced by customers were due to the natural and understandable tendency for software developers to program in a way that's easy for the programmer, and in the process to lose sight of what's easy for the user. In the case of Ceph, containers make it easier for the developers to produce and ship releases. It reduces dependency complexities and testing time. But the developers aren't out in the field with their deployments when something weird impacts a cluster and the standard approaches don't resolve it. And let's face it: Ceph is a marvelously robust solution for large scale storage, but it is also an amazingly intricate matrix of layered interdependent processes, and you haven't got all of the bugs worked out yet. Just note that the beauty of software (or really of anything that is 'designed') is that a few people (developers) can produce something that a large number of people (storage administrators, or 'users') will want to use. Please remember the ratio of users (cluster administrators) to developers and don't lose sight of the users in working to ease and simplify development. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Wed, Jun 2, 2021 at 5:37 AM Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is there a link of the talk I can use as a reference? I would like to look at the pro container points as this post is getting a little bit one sided. I understand that most people prefer things to be stable especially with the underlying storage systems. To me personally, use of containers in general adds a great flexibility because it detaches underlying OS from the running software. All points are fair about adding complexity to the complex system but one thing is missing. Every time developers decide to introduce some new more efficient libraries or frameworks we hit a distribution dependency hell. Because of that, ceph sometimes abandons entire OS versions before their actual lifetime is over. My resources are limited and I don't want to debug / troubleshoot / upgrade OS in addition to ceph itself, hence the containers. Yes it took me a while to warm up to the idea in general but now I don't even think too much about it. I went from Nautilus to Pacific (Centos 7 to Centos 8) within a few hours without needing to upgrade my Ubuntu bare metal nodes. This said, I am for giving people a choice to use packages + ansible / manual install and also allowing manual install of containers. Forcing users' hands too much may make people avoid upgrading their ceph clusters. On Wed, Jun 2, 2021 at 9:27 AM Dave Hall <kdhall@binghamton.edu> wrote:
I'd like to pick up on something that Matthew alluded to, although what I'm saying may not be popular. I agree that containers are compelling for cloud deployment and application scaling, and we can all be glad that container technology has progressed from the days of docker privilege escalation. I also agree that for Ceph users switching from native Ceph processes to containers carries a learning curve that could be as intimidating as learning Ceph itself.
However, here is where I disagree with containerized Ceph: I worked for 19 years as a software developer for a major world-wide company. In that time, I noticed that most of the usability issues experienced by customers were due to the natural and understandable tendency for software developers to program in a way that's easy for the programmer, and in the process to lose sight of what's easy for the user.
In the case of Ceph, containers make it easier for the developers to produce and ship releases. It reduces dependency complexities and testing time. But the developers aren't out in the field with their deployments when something weird impacts a cluster and the standard approaches don't resolve it. And let's face it: Ceph is a marvelously robust solution for large scale storage, but it is also an amazingly intricate matrix of layered interdependent processes, and you haven't got all of the bugs worked out yet.
Just note that the beauty of software (or really of anything that is 'designed') is that a few people (developers) can produce something that a large number of people (storage administrators, or 'users') will want to use.
Please remember the ratio of users (cluster administrators) to developers and don't lose sight of the users in working to ease and simplify development.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Wed, Jun 2, 2021 at 5:37 AM Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Debating containers vs packages is like debating systemd vs initrd. There are lots of reasons why containers (and container orchestration) are good for deploying things, including ceph. Repeating them in each project every time it comes up is not really productive. I'd recommend looking at why containers are good in general. It applies to ceph too. ________________________________________ From: Sasha Litvak <alexander.v.litvak@gmail.com> Sent: Wednesday, June 2, 2021 7:56 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. Is there a link of the talk I can use as a reference? I would like to look at the pro container points as this post is getting a little bit one sided. I understand that most people prefer things to be stable especially with the underlying storage systems. To me personally, use of containers in general adds a great flexibility because it detaches underlying OS from the running software. All points are fair about adding complexity to the complex system but one thing is missing. Every time developers decide to introduce some new more efficient libraries or frameworks we hit a distribution dependency hell. Because of that, ceph sometimes abandons entire OS versions before their actual lifetime is over. My resources are limited and I don't want to debug / troubleshoot / upgrade OS in addition to ceph itself, hence the containers. Yes it took me a while to warm up to the idea in general but now I don't even think too much about it. I went from Nautilus to Pacific (Centos 7 to Centos 8) within a few hours without needing to upgrade my Ubuntu bare metal nodes. This said, I am for giving people a choice to use packages + ansible / manual install and also allowing manual install of containers. Forcing users' hands too much may make people avoid upgrading their ceph clusters. On Wed, Jun 2, 2021 at 9:27 AM Dave Hall <kdhall@binghamton.edu> wrote:
I'd like to pick up on something that Matthew alluded to, although what I'm saying may not be popular. I agree that containers are compelling for cloud deployment and application scaling, and we can all be glad that container technology has progressed from the days of docker privilege escalation. I also agree that for Ceph users switching from native Ceph processes to containers carries a learning curve that could be as intimidating as learning Ceph itself.
However, here is where I disagree with containerized Ceph: I worked for 19 years as a software developer for a major world-wide company. In that time, I noticed that most of the usability issues experienced by customers were due to the natural and understandable tendency for software developers to program in a way that's easy for the programmer, and in the process to lose sight of what's easy for the user.
In the case of Ceph, containers make it easier for the developers to produce and ship releases. It reduces dependency complexities and testing time. But the developers aren't out in the field with their deployments when something weird impacts a cluster and the standard approaches don't resolve it. And let's face it: Ceph is a marvelously robust solution for large scale storage, but it is also an amazingly intricate matrix of layered interdependent processes, and you haven't got all of the bugs worked out yet.
Just note that the beauty of software (or really of anything that is 'designed') is that a few people (developers) can produce something that a large number of people (storage administrators, or 'users') will want to use.
Please remember the ratio of users (cluster administrators) to developers and don't lose sight of the users in working to ease and simplify development.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Wed, Jun 2, 2021 at 5:37 AM Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Only if you also look at why containers are bad in general, as that also applies to ceph as well. Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Fox, Kevin M [mailto:Kevin.Fox@pnnl.gov] Sent: Wednesday, June 2, 2021 9:27 AM To: Sasha Litvak; ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Debating containers vs packages is like debating systemd vs initrd. There are lots of reasons why containers (and container orchestration) are good for deploying things, including ceph. Repeating them in each project every time it comes up is not really productive. I'd recommend looking at why containers are good in general. It applies to ceph too. ________________________________________ From: Sasha Litvak <alexander.v.litvak@gmail.com> Sent: Wednesday, June 2, 2021 7:56 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. Is there a link of the talk I can use as a reference? I would like to look at the pro container points as this post is getting a little bit one sided. I understand that most people prefer things to be stable especially with the underlying storage systems. To me personally, use of containers in general adds a great flexibility because it detaches underlying OS from the running software. All points are fair about adding complexity to the complex system but one thing is missing. Every time developers decide to introduce some new more efficient libraries or frameworks we hit a distribution dependency hell. Because of that, ceph sometimes abandons entire OS versions before their actual lifetime is over. My resources are limited and I don't want to debug / troubleshoot / upgrade OS in addition to ceph itself, hence the containers. Yes it took me a while to warm up to the idea in general but now I don't even think too much about it. I went from Nautilus to Pacific (Centos 7 to Centos 8) within a few hours without needing to upgrade my Ubuntu bare metal nodes. This said, I am for giving people a choice to use packages + ansible / manual install and also allowing manual install of containers. Forcing users' hands too much may make people avoid upgrading their ceph clusters. On Wed, Jun 2, 2021 at 9:27 AM Dave Hall <kdhall@binghamton.edu> wrote:
I'd like to pick up on something that Matthew alluded to, although what I'm saying may not be popular. I agree that containers are compelling for cloud deployment and application scaling, and we can all be glad that container technology has progressed from the days of docker privilege escalation. I also agree that for Ceph users switching from native Ceph processes to containers carries a learning curve that could be as intimidating as learning Ceph itself.
However, here is where I disagree with containerized Ceph: I worked for 19 years as a software developer for a major world-wide company. In that time, I noticed that most of the usability issues experienced by customers were due to the natural and understandable tendency for software developers to program in a way that's easy for the programmer, and in the process to lose sight of what's easy for the user.
In the case of Ceph, containers make it easier for the developers to produce and ship releases. It reduces dependency complexities and testing time. But the developers aren't out in the field with their deployments when something weird impacts a cluster and the standard approaches don't resolve it. And let's face it: Ceph is a marvelously robust solution for large scale storage, but it is also an amazingly intricate matrix of layered interdependent processes, and you haven't got all of the bugs worked out yet.
Just note that the beauty of software (or really of anything that is 'designed') is that a few people (developers) can produce something that a large number of people (storage administrators, or 'users') will want to use.
Please remember the ratio of users (cluster administrators) to developers and don't lose sight of the users in working to ease and simplify development.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Wed, Jun 2, 2021 at 5:37 AM Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dave Hall (kdhall) writes:
But the developers aren't out in the field with their deployments when something weird impacts a cluster and the standard approaches don't resolve it. And let's face it: Ceph is a marvelously robust solution for large scale storage, but it is also an amazingly intricate matrix of layered interdependent processes, and you haven't got all of the bugs worked out yet.
I think you hit a very important point here: the concern with containerized deployments is that they may be a barrier to efficient troubleshooting and bug reporting by traditional methods (strace et al) -- unless a well documented debugging and analysis toolset/methodolgy is provided. Paradoxically, containerized deployments certainly sound like they'd free up lots of cycles from the developer side of things (no more building packages for N distributions as was pointed out, easier upgrade and regression testing), but it might make it more difficult initially for the community to contribute (well, at least for us dinosaurs that aren't born with docker brains). Cheers, Phil
On 6/2/21 2:28 PM, Phil Regnauld wrote:
Dave Hall (kdhall) writes:
But the developers aren't out in the field with their deployments when something weird impacts a cluster and the standard approaches don't resolve it. And let's face it: Ceph is a marvelously robust solution for large scale storage, but it is also an amazingly intricate matrix of layered interdependent processes, and you haven't got all of the bugs worked out yet. I think you hit a very important point here: the concern with containerized deployments is that they may be a barrier to efficient troubleshooting and bug reporting by traditional methods (strace et al) -- unless a well documented debugging and analysis toolset/methodolgy is provided.
Paradoxically, containerized deployments certainly sound like they'd free up lots of cycles from the developer side of things (no more building packages for N distributions as was pointed out, easier upgrade and regression testing), but it might make it more difficult initially for the community to contribute (well, at least for us dinosaurs that aren't born with docker brains).
Cheers, Phil
I think there's great value in ceph devs doing QA and testing docker images, releasing them as a 'known good thing'. Why? Doing that avoids dependency hell inducing fragility-- fragility which I've experienced in other multi-host / multi-master packages. Wherein one distro's maintainer decides some new rev ought be pushed out as 'security update' while another distro's maintainer decides it's a feature change, another calls it a backport, etc. There's no way to QA 'upgrades' across so many grains of shifting sand. While the devs and the rest of the bleeding-edge folks should enjoy the benefits that come with tolerating and managing dependency hell, having the orchestrator upgrade in a known good sequence from a known base to a known release reduces fragility. Thanks for ceph! Harry
In this giga, tera byte times all this dependency hell can now be avoided with some static linking. For example, we do use statically linked mysql binaries and it saved us numerous times. https://youtu.be/5PmHRSeA2c8?t=490 Rok On Wed, Jun 2, 2021 at 9:57 PM Harry G. Coin <hgcoin@gmail.com> wrote:
Dave Hall (kdhall) writes:
But the developers aren't out in the field with their deployments when something weird impacts a cluster and the standard approaches don't resolve it. And let's face it: Ceph is a marvelously robust solution for large scale storage, but it is also an amazingly intricate matrix of layered interdependent processes, and you haven't got all of the bugs worked out yet. I think you hit a very important point here: the concern with containerized deployments is that they may be a barrier to efficient troubleshooting and bug reporting by traditional methods (strace et al) -- unless a well documented debugging and analysis toolset/methodolgy is provided.
Paradoxically, containerized deployments certainly sound like
On 6/2/21 2:28 PM, Phil Regnauld wrote: they'd
free up lots of cycles from the developer side of things (no more building packages for N distributions as was pointed out, easier upgrade and regression testing), but it might make it more
difficult
initially for the community to contribute (well, at least for us dinosaurs that aren't born with docker brains).
Cheers, Phil
I think there's great value in ceph devs doing QA and testing docker images, releasing them as a 'known good thing'. Why? Doing that avoids dependency hell inducing fragility-- fragility which I've experienced in other multi-host / multi-master packages. Wherein one distro's maintainer decides some new rev ought be pushed out as 'security update' while another distro's maintainer decides it's a feature change, another calls it a backport, etc. There's no way to QA 'upgrades' across so many grains of shifting sand.
While the devs and the rest of the bleeding-edge folks should enjoy the benefits that come with tolerating and managing dependency hell, having the orchestrator upgrade in a known good sequence from a known base to a known release reduces fragility.
Thanks for ceph!
Harry
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ultimately, that's what a container image is. From the outside, its a statically linked binary. From the inside, it can be assembled using modular techniques. The best thing about it, is you can use container scanners and other techniques to gain a lot of the benefits of that modularity still. Plus, you don't need special compilers, whole dependency chains compiled with static options, etc. So an interesting data point in the statically linked binary vs dynamic debate. Its kind of both. ________________________________________ From: Rok Jaklič <rjaklic@gmail.com> Sent: Wednesday, June 2, 2021 1:18 PM To: Harry G. Coin Cc: ceph-users Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. In this giga, tera byte times all this dependency hell can now be avoided with some static linking. For example, we do use statically linked mysql binaries and it saved us numerous times. https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyoutu.be%2... Rok On Wed, Jun 2, 2021 at 9:57 PM Harry G. Coin <hgcoin@gmail.com> wrote:
Dave Hall (kdhall) writes:
But the developers aren't out in the field with their deployments when something weird impacts a cluster and the standard approaches don't resolve it. And let's face it: Ceph is a marvelously robust solution for large scale storage, but it is also an amazingly intricate matrix of layered interdependent processes, and you haven't got all of the bugs worked out yet. I think you hit a very important point here: the concern with containerized deployments is that they may be a barrier to efficient troubleshooting and bug reporting by traditional methods (strace et al) -- unless a well documented debugging and analysis toolset/methodolgy is provided.
Paradoxically, containerized deployments certainly sound like
On 6/2/21 2:28 PM, Phil Regnauld wrote: they'd
free up lots of cycles from the developer side of things (no more building packages for N distributions as was pointed out, easier upgrade and regression testing), but it might make it more
difficult
initially for the community to contribute (well, at least for us dinosaurs that aren't born with docker brains).
Cheers, Phil
I think there's great value in ceph devs doing QA and testing docker images, releasing them as a 'known good thing'. Why? Doing that avoids dependency hell inducing fragility-- fragility which I've experienced in other multi-host / multi-master packages. Wherein one distro's maintainer decides some new rev ought be pushed out as 'security update' while another distro's maintainer decides it's a feature change, another calls it a backport, etc. There's no way to QA 'upgrades' across so many grains of shifting sand.
While the devs and the rest of the bleeding-edge folks should enjoy the benefits that come with tolerating and managing dependency hell, having the orchestrator upgrade in a known good sequence from a known base to a known release reduces fragility.
Thanks for ceph!
Harry
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, I figured it might be worth chiming in with an example from users (even though we're not quite novice users anymore). On the one hand we really like cephadm - it's beautiful with central reporting of versions, status, default ability to perform rolling updates, placing services, etc. A minor drawback is that we can no longer easily handle things in salt, but I think that's a matter of having to accept that we can't both eat the cake and have it (since both are different management tools). However, we recently had another snafu - suddenly cephadm upgrade doesn't do anything - it just waits. I'm sure this is a trivial bug/config setting somewhere, but there are pretty much no log messages about, and what would have been trivial with standard packages (just force an upgrade, one node at a time, if necessary) suddenly becomes a bug-hunt that might take a day or two. This is most definitely not an argument against Cephadm. I think it's a great solution we'll stick with, and I'm sure it will only get more stable. Still, I think the challenge for non-expert users is that *default* deployments and maintenance can be easier with container solutions, but when you need to step out of that and debug/customize things, the additional layer(s) create a barrier. Cheers, Erik
While there are many reasons containerization helps, I'll just touch on one real quick that is relevant to the conversation. Orchestration. Implementing orchestration of an entire clustered software with many different: * package managers * dependency chains * init systems * distro specific quirks * configuration management systems has over the years proven intractable. Each project cant implement all permutations of each of those with high quality. Which is especially important in a data storage system. Good orchestration of complex systems before containers was very rare. Finding two that you wanted to use, that supported the same base systems was even more rare. Standardizing on containers allow orchestration to be performed without concern to those things, making it significantly easier to write good orchestration that is usable by a broad audience. Such as cephadm. rook-ceph takes it to another level IMO and only requires managing just the orchestration bits that are specific to ceph over the base orchestration layer, k8s. So even more orchestration layer can be shared between the devs and the admins. This allows admins to generalize their skills even more. Yes, its another thing to learn. but its worth it in the long term IMO. Being able to quickly deploy ceph, elasticsearch, and greenplum clusters and deal with the day two issues with ease is something that hasn't been a thing before. Yes, its a big shift from the traditional distro philosophy. But the advanced orchestration abilities possible once we step outside the traditional linux distro really is game changing. ________________________________________ From: Matthew Vernon <mv3@sanger.ac.uk> Sent: Wednesday, June 2, 2021 2:36 AM To: ceph-users@ceph.io Subject: [ceph-users] Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, that's also a +1 from me — we also use containers heavily for scientific workflows, and know their benefits well. But they are not the "best", or rather, the most fitting tool in every situation. You have provided a great summary and I agree with all points, and thank you a lot for this very competent and concise write-up. Since in this lengthy thread, static linking and solving the issue of many inter-dependencies for production services with containers have been mentioned as solutions, I'd like to add another point to your list of complexities: * Keeping production systems secure may be a lot more of a hassle. Even though the following article is long and many may regard it as controversial, I'd like to link to a concise write-up from a packager discussing this topic in a quite generic way: https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-nig... While the article discusses the issues of static linking and package management performed in language-specific domains, it applies all the same to containers. If I operate services in containers built by developers, of course this ensures the setup works, and dependencies are well tested, and even upgrades work well — but it also means that, at the end of the day, if I run 50 services in 50 different containers from 50 different upstreams, I'll have up to 50 different versions of OpenSSL floating around my production servers. If a security issue is found in any of the packages used in all the container images, I now need to trust the security teams of all the 50 developer groups building these containers (and most FOSS projects won't have the ressources, understandably...), instead of the one security team of the disto I use. And then, I also have to re-pull all these containers, after finding out that a security fix has become available. Or I need to build all these containers myself, and effectively take over the complete job, and have my own security team. This may scale somewhat well, if you have a team of 50 people, and every person takes care of one service. Containers are often your friend in this case[1], since it allows to isolate the different responsibilities along with the service. But this is rarely the case outside of industry, and especially not in academics. So the approach we chose for us is to have one common OS everywhere, and automate all of our deployment and configuration management with Puppet. Of course, that puts is in one of the many corners out there, but it scales extremely well to all services we operate, and I can still trust the distro maintainers to keep the base OS safe on all our servers, automate reboots etc. For Ceph, we've actually seen questions about security issues already on the list[0] (never answered AFAICT). To conclude, I strongly believe there's no one size fits all here. That was why I was hopeful when I first heard about the Ceph orchestrator idea, when it looked to be planned out to be modular, with the different tasks being implementable in several backends, so one could imagine them being implemented with containers, with classic SSH on bare-metal (i.e. ceph-deploy-like), ansible, rook or maybe others. Sadly, it seems it ended up being "container-only". Containers certainly have many uses, and we run thousands of them daily, but neither do they fit each and every existing requirement, nor are they a magic bullet to solve all issues. Cheers, Oliver [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/PPLJIHT6WKY... [1] But you may also just have a very well structured configuration management system fitting your organizational structure. Am 02.06.21 um 11:36 schrieb Matthew Vernon:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.047 Nußallee 12 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --
Hi, We operate a few Ceph hyperconverged clusters with Proxmox, that provides a custom ceph package repository. They do a great work; and deployment is a brezee. So, even as currently we would rely on Proxmox packages/distribution and not upstream, we have a number of other projects deployed with containers and we even distribute some of our own development in deb and container packages, so I will comment on our view: El 2/6/21 a las 23:26, Oliver Freyermuth escribió: [...]
If I operate services in containers built by developers, of course this ensures the setup works, and dependencies are well tested, and even upgrades work well — but it also means that, at the end of the day, if I run 50 services in 50 different containers from 50 different upstreams, I'll have up to 50 different versions of OpenSSL floating around my production servers. If a security issue is found in any of the packages used in all the container images, I now need to trust the security teams of all the 50 developer groups building these containers (and most FOSS projects won't have the ressources, understandably...), instead of the one security team of the disto I use. And then, I also have to re-pull all these containers, after finding out that a security fix has become available. Or I need to build all these containers myself, and effectively take over the complete job, and have my own security team.
This may scale somewhat well, if you have a team of 50 people, and every person takes care of one service. Containers are often your friend in this case[1], since it allows to isolate the different responsibilities along with the service.
But this is rarely the case outside of industry, and especially not in academics. So the approach we chose for us is to have one common OS everywhere, and automate all of our deployment and configuration management with Puppet. Of course, that puts is in one of the many corners out there, but it scales extremely well to all services we operate, and I can still trust the distro maintainers to keep the base OS safe on all our servers, automate reboots etc.
For Ceph, we've actually seen questions about security issues already on the list[0] (never answered AFAICT).
These are the two main issues I find with containers really: - Keeping hosts uptodate is more complex (apt-get update+apt-get dist-upgrade and also some kind of docker pull+docker restart/docker-compose up ...). Much of the time the second part is not standard (just deployed a Harbor service, upgrade is quite simple but I have to know how to do it as it's speciffic, maintenance would be much easier if it was packaged in Debian). I won't say it's more difficult, but it will be more diverse and complex. - Container image quality and security support quality, that will vary from upstream to upstream. You have to research each of them to know were they stand. A distro (specially a good one like Debian, Ubuntu, RHEL or SUSE) has known, quality security support for the repositories. They will even fix issues not fixed by upstream (o backport them to distro's version...). This is more an upstream vs distro issue, really. About debugging issues reported with Ceph containers, I think those are things waiting for a fix: why are logs writen in container image (or an ephemeral volume, I don't know really how is that done right now) instead of an external name volume o a local mapped dir in /var/log/ceph ? All that said, I think that it makes sense for an upstream project like Ceph, to distribute container images, as it is the most generic way to distribute (you can deploy on any system/distro supporting container images) and eases development. But only distributing container images could make more users depend on third party distribution (global or specific distros), which would delay feeback/bugreport to upstream. Cheers and thanks for the great work! Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
在 2021年6月4日，21:51，Eneko Lacunza <elacunza@binovo.es> 写道：
Hi,
We operate a few Ceph hyperconverged clusters with Proxmox, that provides a custom ceph package repository. They do a great work; and deployment is a brezee.
So, even as currently we would rely on Proxmox packages/distribution and not upstream, we have a number of other projects deployed with containers and we even distribute some of our own development in deb and container packages, so I will comment on our view:
El 2/6/21 a las 23:26, Oliver Freyermuth escribió: [...]
If I operate services in containers built by developers, of course this ensures the setup works, and dependencies are well tested, and even upgrades work well — but it also means that, at the end of the day, if I run 50 services in 50 different containers from 50 different upstreams, I'll have up to 50 different versions of OpenSSL floating around my production servers. If a security issue is found in any of the packages used in all the container images, I now need to trust the security teams of all the 50 developer groups building these containers (and most FOSS projects won't have the ressources, understandably...), instead of the one security team of the disto I use. And then, I also have to re-pull all these containers, after finding out that a security fix has become available. Or I need to build all these containers myself, and effectively take over the complete job, and have my own security team.
This may scale somewhat well, if you have a team of 50 people, and every person takes care of one service. Containers are often your friend in this case[1], since it allows to isolate the different responsibilities along with the service.
But this is rarely the case outside of industry, and especially not in academics. So the approach we chose for us is to have one common OS everywhere, and automate all of our deployment and configuration management with Puppet. Of course, that puts is in one of the many corners out there, but it scales extremely well to all services we operate, and I can still trust the distro maintainers to keep the base OS safe on all our servers, automate reboots etc.
For Ceph, we've actually seen questions about security issues already on the list[0] (never answered AFAICT).
These are the two main issues I find with containers really:
- Keeping hosts uptodate is more complex (apt-get update+apt-get dist-upgrade and also some kind of docker pull+docker restart/docker-compose up ...). Much of the time the second part is not standard (just deployed a Harbor service, upgrade is quite simple but I have to know how to do it as it's speciffic, maintenance would be much easier if it was packaged in Debian). I won't say it's more difficult, but it will be more diverse and complex.
- Container image quality and security support quality, that will vary from upstream to upstream. You have to research each of them to know were they stand. A distro (specially a good one like Debian, Ubuntu, RHEL or SUSE) has known, quality security support for the repositories. They will even fix issues not fixed by upstream (o backport them to distro's version...). This is more an upstream vs distro issue, really.
About debugging issues reported with Ceph containers, I think those are things waiting for a fix: why are logs writen in container image (or an ephemeral volume, I don't know really how is that done right now) instead of an external name volume o a local mapped dir in /var/log/ceph ?
You could find the logs with “journalctl” outside of containers.
All that said, I think that it makes sense for an upstream project like Ceph, to distribute container images, as it is the most generic way to distribute (you can deploy on any system/distro supporting container images) and eases development. But only distributing container images could make more users depend on third party distribution (global or specific distros), which would delay feeback/bugreport to upstream.
Cheers and thanks for the great work!
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.binov... Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtu... https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linke... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm arriving late to this thread, but a few things stood out that I wanted to clarify. On Wed, Jun 2, 2021 at 4:28 PM Oliver Freyermuth <freyermuth@physik.uni-bonn.de> wrote:
To conclude, I strongly believe there's no one size fits all here.
That was why I was hopeful when I first heard about the Ceph orchestrator idea, when it looked to be planned out to be modular, with the different tasks being implementable in several backends, so one could imagine them being implemented with containers, with classic SSH on bare-metal (i.e. ceph-deploy-like), ansible, rook or maybe others. Sadly, it seems it ended up being "container-only". Containers certainly have many uses, and we run thousands of them daily, but neither do they fit each and every existing requirement, nor are they a magic bullet to solve all issues.
The orchestrator layer is an abstraction of the tool(s) used to provision ceph. Cephadm is just one implementation of that abstraction (there is another implementation that talks to rook). The abstraction is not designed to be 'container-only,' although currently both implementations do use containers. To add orchestrator support for a traditional deployment using packages, we have two options: (1) implement a third orchestrator module that handles package installation etc, or (2) modify cephadm to handle both container-based and package-based deployments. I suspect that (2) is less work, but I'll be honest that the cephadm team isn't yet swayed by the anti-container arguments, so there would be some lobbying and discussion to be done first! For either option, there are also some minor orchestrator interface adjustments that maybe needed. Cephadm can deploy an arbitrary ceph version/container image for individual daemons, but with packages it's per-host. With rook it's a per-cluster option (rook currently handles any upgrade/downgrade), so there is already some ambiguity and cleanup opportunity here. sage
Thanks for answering these. I have been using ceph since Kraken and are now on Nautilus. I thought before to join this discussion to watch this video[1] on cephadm, but it seems to be more about what console commands to type. So please indulge my rookie comments.
the cephadm team isn't yet swayed by the anti-container arguments, so there would be some lobbying and discussion to be done first!
To be honest, the same goes for seeing the arguments using the cephadm approach. Remarks about your cephadm approach/design: 1. I am not interested in learning podman, rook or kubernetes. I am using mesos which is also on my osd nodes to use the extra available memory and cores. Furthermore your cephadm OC is limited to only ceph nodes. While my mesos OC is spread across a larger cluster and has rules when, and when not to run tasks on the osd nodes. You incorrectly assume that rgw, grafana, prometheus, haproxy are going to be ran on your ceph OC. 2. Nico pointed out that you do not have alpine linux container images. I did not even know you were using container images. So how big are these? Where are these stored. And why are these not as small as they can be? Such an osd container image should be 20MB or so at most. I would even expect statically build binary container image, why even a tiny os? 3. Why is in this cephadm still being talked about systemd? Your orchestrator should handle restarts,namespaces and failed tasks not? There should be no need to have a systemd dependency, at least I have not seen any container images relying on this. 4. Ok found the container images[2] (I think). Sorry but this has ‘nothing’ to do with container thinking. I expected to find container images for osd, msd, rgw separately and smaller. This looks more like an OS deployment. 5. I have been writing this previously on the mailing list here. Is each rgw still requiring its own dedicated client id? Is it still true, that if you want to spawn 3 rgw instances, they need to authorize like client.rgw1, client.rgw2 and client.rgw3? This does not allow for auto scaling. The idea of using an OC is that you launch a task, and that you can scale this task automatically when necessary. So you would get multiple instances of rgw1. If this is still and issue with rgw, mds and mgr etc. Why even bother doing something with an OC and containers? 6. As I wrote before I do not want my rgw or haproxy running in a OC that has the ability to give tasks capability SYSADMIN. So that would mean I have to run my osd daemons/containers separately. 7. If you are not setting cpu and memory limits on your cephadm containers, then again there is an argument why even use containers. 8. I still see lots of comments on the mailing list about accessing logs. I have all my containers log to a remote syslog server, if you still have your ceph daemons that can not do this (correctly). What point is it even going to containers. 9. I am updating my small cluster something like this: ssh root@c01 "ceph osd set noout ; ceph osd set noscrub ; ceph osd set nodeep-scrub" ssh root@c01 "ceph tell osd.* injectargs '--osd_max_scrubs=0'" ssh root@c01 "yum update 'ceph-*' -y" ... ssh root@c01 "service ceph-mon@a restart" ... ssh root@c01 "service ceph-mgr@a restart" ... # wait for up and recovery to finish ssh root@c01 "systemctl restart 'ceph-osd@*'" … I am never going to run a ‘ceph orch upgrade start –ceph-version 16.2.0’. I want to see if everything is ok after each command I issue. I want to see if scrubbing stopped, I want to see if osd have correctly accepted the new config. I have a small cluster so I do not see this procedure as a waste of time. If I look at your telemetry data[3]. I see 600 clusters with 35k osd’s, that is an average of 60 osd per cluster. So these are quite small clusters, I would think these admins have a similar point of view as I have. That leaves these big clusters of >3k osd’s. I wonder what these admins require, are they at CERN really waiting for something like cephadm? I am rather getting the impression you need to have an easy deployment tool for ceph than you want to really utilize containers. First there was this ceph-deploy and ceph-ansible which I luckily skipped both, and now there is cephadm. I am not anti-container and I think a lot here are not anti-container but you are using this as an argument to push cephadm work. The ceph daemons seem to be not prepared for container use, ceph containers can’t use cpu/memory limits, container images are not how they should be. And last but not least you totally bypass that the (ceph) admin should choose the OC platform and not you, because he probably has more than just ceph nodes. So my question to you: What problem is it actually that your cephadm dev team is trying to solve? That is not clear to me. [1] https://www.youtube.com/watch?v=A2LK--zs4Io [2] https://hub.docker.com/r/ceph/ceph/tags?page=1&ordering=last_updated [3] https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?orgId=1
3. Why is in this cephadm still being talked about systemd? Your orchestrator should handle restarts,namespaces and failed tasks not? There should be no need to have a systemd dependency, at least I have not seen any container images relying on this.
Podman uses systemd to manage containers so that it is daemonless, contrast with Docker where one has to maintain a separate daemon and use docker-specific tools to mange containers. If you assert that Podman should not exist, please take that up with the Podman folks.
4. Ok found the container images[2] (I think). Sorry but this has ‘nothing’ to do with container thinking. I expected to find container images for osd, msd, rgw separately and smaller. This looks more like an OS deployment.
Bundling all the daemons together into one container is *genius*. Much simpler to build and maintain, one artifact vs a bunch. I wouldn’t be surprised if there are memory usage efficiencies too.
7. If you are not setting cpu and memory limits on your cephadm containers, then again there is an argument why even use containers.
This seems like a non-sequitor. As many have written, CPU and memory limits aren’t the reason for containerizing Ceph daemons. If there are other container applications where doing so makes sense, that’s fine for those applications. I suspect that artificial CPU limiting of Ceph daemons would have a negative impact on latency, peering/flapping, and slow requests. Ceph is a distributed system, not a massively parallel one. OSDs already have a memory target that can be managed natively, vs an external mechanism that arbitrarily cuts them off at the knees when they need it the most. That approach would be addressing the symptoms, not the root cause. Having been through a multi-day outage that was substantially worsened by the OOMkiller (*), I personally want nothing to do with blind external mechanisms deciding that they know better than Ceph daemons whether or not they should be running. If your availability and performance needs favor rigidly defined areas of doubt and uncertainty, that’s your own lookout. * A release that didn’t have OSD memory target setting yet. Having that would have helped dramatically.
8. I still see lots of comments on the mailing list about accessing logs. I have all my containers log to a remote syslog server, if you still have your ceph daemons that can not do this (correctly). What point is it even going to containers.
With all possible respect, that’s another non-sequitor, or at the very least, an assumption that your needs are everyone’s needs. Centralized logging makes sense in some contexts. But not always, and not to everyone. Back around the Hammer or Jewel releases there was a bug where logging to syslog resulted in daemon crashes. I haven’t tried it with newer releases, but assume that’s long been fixed. I’m not a an [r]syslog[ng] expert by far, but I suspect that central-only logging might not deal well with situations like an OSD spewing entries when the debug subsystem level is elevated. Moreover, many of the issues one sees with Ceph clusters are network related. So when there’s a network problem, I want to rely on the network to be able to see logs? I’ve seen syslog drop entries under load, something I personally wouldn’t want for Ceph daemons. There are of course many strategies between the extremes.
9. I am updating my small cluster something like this:
I’m guessing you’ve never updated between major releases. That process tends to have additional steps and nuances, which is one of the compelling arguments in favor of orchestration: when it’s done well, most operators don’t need to rev their own homebrew orchestration to set the right flags at the right time, etc. But one of the great things about OSS is that you have the flexibility to roll you own if you so choose.
I am never going to run a ‘ceph orch upgrade start –ceph-version 16.2.0’. I want to see if everything is ok after each command I issue. I want to see if scrubbing stopped, I want to see if osd have correctly accepted the new config.
So you want to do all the things that an orchestrated rolling upgrade does for you. Check.
I have a small cluster so I do not see this procedure as a waste of time. If I look at your telemetry data[3]. I see 600 clusters with 35k osd’s, that is an average of 60 osd per cluster. So these are quite small clusters, I would think these admins have a similar point of view as I have.
Careful with those inferences. * Operators who submit telemetry may not be a representative sample * Sites may have many more than one cluster, If one has a 20 OSD lab cluster and a 1000 OSD production cluster, perspectives and processes are going to be different than someone with a single 60 OSD cluster. * Average != median
I am rather getting the impression you need to have an easy deployment tool for ceph than you want to really utilize containers. First there was this ceph-deploy and ceph-ansible which I luckily skipped both
That’s more than a little harsh. A lot of people get a lot of value out of those tools.
The ceph daemons seem to be not prepared for container use, ceph containers can’t use cpu/memory limits
They don’t make julienne fries either. What of it?
And last but not least you totally bypass that the (ceph) admin should choose the OC platform and not you, because he probably has more than just ceph nodes.
Nobody’s stopping you from rolling your own containers, using traditional packages, or heck even deploying with tarballs. That’s the beauty of OSS. Let’s leave Orange County out of it though.
So my question to you: What problem is it actually that your cephadm dev team is trying to solve? That is not clear to me.
Asked and answered, sir. Different ones than you’re trying to solve, which doesn’t mean that either you or upstream are wrong. There’s always room for improvement with any approach, and again the beauty of OSS is that you *can* go your own way if you so choose.
Because all of this reads way to negative regarding containers for me I wanted to give a different perspective. Coming from a day to day job, that heavily utilizes kubernetes for its normal environment, I found cephadm quite like a godsent, instead of having to deal with a lot of pesky details with installations and services, having to learn ansible or ceph-deploy, that tool did like 90% of everything in a way I already feel familiar with it. Additionally, having some pools for not so important data using low replications counts, it is quite nice, that cephadm only concurrently upgrades osd's in matter that no service interruptions happen. There are still some shortcomings (eg. some limitations when moving osd devices between hosts), but as it is still a relatively new tool that is to be expected.
I think the primary goal of a container environments are resource isolation. At least when I read about the history I never read anything about a tool for people to skip to learn something. But you clearly portray the situation here. Nobody here is against container environment, because every tool has its purpose. But the container solution proposed here is not for the purpose of utilizing container benefits, but for creating a tool so 'I do not know what to do' people can use ceph. And because this is the development perspective and ceph is not really adapted for being used in containers, you get the current friction with accepting cephadm. Eg. this command is wrong in my opinion ceph orch device ls What has a device list to do with having an OC or not? I do not even know why ceph is working on something that uses Kubernetes, let the Kubernetes platform implement a ceph solution. If ceph wants be used in a container environment, start making the daemons ready to run in container environments. So my question still stands: What problem is it actually that the cephadm dev team is trying to solve? That is not clear to me.
-----Original Message----- Sent: Monday, 21 June 2021 01:21 To: ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments
Because all of this reads way to negative regarding containers for me I wanted to give a different perspective.
Coming from a day to day job, that heavily utilizes kubernetes for its normal environment, I found cephadm quite like a godsent,
instead of having to deal with a lot of pesky details with installations and services, having to learn ansible or ceph-deploy, that tool did like 90% of everything in a way I already feel familiar with it.
Additionally, having some pools for not so important data using low replications counts, it is quite nice, that cephadm only concurrently upgrades osd's in matter that no service interruptions happen.
There are still some shortcomings (eg. some limitations when moving osd devices between hosts), but as it is still a relatively new tool that is to be expected. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think the primary goal of a container environments are resource isolation. At least when I read about the history I never read anything about a tool for people to skip to learn something. Containers allow to use mixed version of the same dependency, despite being a shared dependency, doing this in a non container envrionment basically means rebuilding containers But you clearly portray the situation here. Nobody here is against container environment, because every tool has its purpose. But the container solution proposed here is not for the purpose of utilizing container benefits, but for creating a tool so 'I do not know what to do' people can use ceph. And because this is the development perspective and ceph is not really adapted for being used in containers, you get the current friction with accepting cephadm.
Is this like an insult? The point is not about not learning something, but I would choose docker/container over ansible every time for this (ansible has its place, but it has no concept of current state, as a ceph orchestrator heavily benefits from). I kinda feel that you have not worked a lot with containers, created a strawman what you think benefits are, and then continue to burn down that strawman while not even recognizing a lot of benefits.
Eg. this command is wrong in my opinion
ceph orch device ls
What has a device list to do with having an OC or not? Well you need to start a small tool to actually find devices on every system, for this you need to first find all nodes that exist in the cluster, if not the orchestrator, who should have this information, especially since it can change at runtime due to hosts being added. I do not even know why ceph is working on something that uses Kubernetes, let the Kubernetes platform implement a ceph solution. If ceph wants be used in a container environment, start making the daemons ready to run in container environments.
So my question still stands: What problem is it actually that the cephadm dev team is trying to solve? That is not clear to me.
Again, clean orchestration, being able to upgrade each deamon without influencing running ones, this is just not possible with the native packages. Being able to run on a lot more platforms, as native packages are just not maintainable for all platforms that could easily run a docker daemon.
-----Original Message----- Sent: Monday, 21 June 2021 01:21 To: ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments
Because all of this reads way to negative regarding containers for me I wanted to give a different perspective.
Coming from a day to day job, that heavily utilizes kubernetes for its normal environment, I found cephadm quite like a godsent,
instead of having to deal with a lot of pesky details with installations and services, having to learn ansible or ceph-deploy, that tool did like 90% of everything in a way I already feel familiar with it.
Additionally, having some pools for not so important data using low replications counts, it is quite nice, that cephadm only concurrently upgrades osd's in matter that no service interruptions happen.
There are still some shortcomings (eg. some limitations when moving osd devices between hosts), but as it is still a relatively new tool that is to be expected. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think 2 things need to be clarified here:
[...] Again, clean orchestration, being able to upgrade each deamon without influencing running ones, this is just not possible with the native packages.
If a daemon is running on an operating system, it does not reload shared libraries or binaries. So upgrading a system != every daemon at the same time gets new executable content loaded. If a daemon restarts that particular moment of OS upgrade, then yes, then it loads the new binary. But are you able to hit that spot of a few minutes? Unlikely. Especially if you tested your upgrade on a staging environment before.
Being able to run on a lot more platforms, as native packages are just not maintainable for all platforms that could easily run a docker daemon.
Not biting on this one. If we talk platforms, you are basically fine for a lot of projects if you deliver a set of .rpms and a set of .debs. Or if you work closely together with the distros, then often that distro can actually help you building (and maintaining) that package. As far as I know the core ceph team is working at Redhat, so you are basically getting 1 distro/package for free. And while we are at claiming "on a lot more platforms", you are at the same time EXCLUDING a lot of platforms by saying "Linux based container" (remember Ceph on FreeBSD? [0]). I am aware that centralisation is in fashion, but coming from an Open Source background, there is one big reason to build Open Source software: to be portable and inclusive. I've experienced the time where the "only platform" was MS-DOS/Windows and I can assure anyone in here that "setting on one platform" instead of "being portable" is a shot in your own foot. History repeats. Cheers, Nico [0] https://docs.ceph.com/en/latest/install/manual-freebsd-deployment/ -- Sustainable and modern Infrastructures by ungleich.ch
I think 2 things need to be clarified here:
[...] Again, clean orchestration, being able to upgrade each deamon without influencing running ones, this is just not possible with the native packages.
If a daemon is running on an operating system, it does not reload shared libraries or binaries. So upgrading a system != every daemon at the same time gets new executable content loaded.
If a daemon restarts that particular moment of OS upgrade, then yes, then it loads the new binary. But are you able to hit that spot of a few minutes? Unlikely. Especially if you tested your upgrade on a staging environment before.
Indeed, as if this has been such a huge problem. I would rather focus on testing the updates. Because every update almost certainly is being followed by regression/fix lately. I am already not installing new releases, and waiting for cowboy cephers to have found all the bugs.
Being able to run on a lot more platforms, as native packages are just not maintainable for all platforms that could easily run a docker daemon.
Not biting on this one. If we talk platforms, you are basically fine for a lot of projects if you deliver a set of .rpms and a set of .debs. Or if you work closely together with the distros, then often that distro can actually help you building (and maintaining) that package.
Again has it ever been a problem? Besides ceph highly relies on kernel versions, so your container environment is creating a dependency there. Furthermore deploying a 350MB (compressed) image for an osd container of 5+17MB? That is like sending an elephant to the Olympics figure skating.
As far as I know the core ceph team is working at Redhat, so you are basically getting 1 distro/package for free.
Yes afaik we should get support for el7, that was always written here. Yet python 3 is messing with this now, and I wonder if the latest release of ceph is even running on el7. I do not know what to believe anymore. I just want to have my packages and I want to keep decent cli tools to work with them. As I have been used to do. I do not want to have a 2nd OC platform just for a ceph cli tool, so rookies are having an easier learning curve.
And while we are at claiming "on a lot more platforms", you are at the same time EXCLUDING a lot of platforms by saying "Linux based container" (remember Ceph on FreeBSD? [0]).
I am aware that centralisation is in fashion, but coming from an Open Source background, there is one big reason to build Open Source software: to be portable and inclusive.
I've experienced the time where the "only platform" was MS-DOS/Windows and I can assure anyone in here that "setting on one platform" instead of "being portable" is a shot in your own foot.
I have seen no arguments why to use containers other than to try and make it "easier" for new ceph people. Containers are not being used as they should be. I wonder even how local persistent volumes are being assigned here. Because that requires then the involvement of (I hope) csi drivers, and if you see how 'buggy' the ceph csi driver is being coded, I do not think at this time it will adhere to a stable solution.
On 6/21/21 7:37 PM, Marc wrote:
I have seen no arguments why to use containers other than to try and make it "easier" for new ceph people.
I advise to read the whole thread again, especially Sage his comments, as there are other benefits. It would free up resources that can be dedicated to (arguably) more pressing issues. Containers are not being used as they should be. There is no "should be", there is no one answer to that, other than 42. Containers have been there before Docker, but Docker made them popular, exactly for the same reason as why Ceph wants to use them: ship a known good version (CI tests) of the software with all dependencies, that can be run "as is" on any supported platform. Gr. Stefan
There is no "should be", there is no one answer to that, other than 42. Containers have been there before Docker, but Docker made them popular, exactly for the same reason as why Ceph wants to use them: ship a known good version (CI tests) of the software with all dependencies, that can be run "as is" on any supported platform.
So ship it tested for container software XXX and run it on YYY. How will that benefit me as a user? There are differences when running a docker container, lxc, nspawn, podman, kubernetes and whatever. So you trade error A for error B. There are even problems with containers if you don't use version X from docker. That's what the past told us, why should it be better in the future with even more container environments. Have you tried running rancher on debian in the past? It breaks apart due to iptables or other stuff. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx On Tue, 22 Jun 2021 at 17:53, Stefan Kooman <stefan@bit.nl> wrote:
On 6/21/21 7:37 PM, Marc wrote:
I have seen no arguments why to use containers other than to try and
make it "easier" for new ceph people.
I advise to read the whole thread again, especially Sage his comments, as there are other benefits. It would free up resources that can be dedicated to (arguably) more pressing issues.
Containers are not being used as they should be.
There is no "should be", there is no one answer to that, other than 42. Containers have been there before Docker, but Docker made them popular, exactly for the same reason as why Ceph wants to use them: ship a known good version (CI tests) of the software with all dependencies, that can be run "as is" on any supported platform.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6/22/21 6:56 PM, Martin Verges wrote: > > There is no "should be", there is no one answer to that, other than > 42. Containers have been there before Docker, but Docker made them > popular, exactly for the same reason as why Ceph wants to use them: ship > a known good version (CI tests) of the software with all dependencies, > that can be run "as is" on any supported platform. > > So ship it tested for container software XXX and run it on YYY. How will > that benefit me as a user? To see the bigger picture: if indeed resources are freed up they can be used to fix bugs faster / create new features, add more tests, etc. There are differences when running a docker > container, lxc, nspawn, podman, kubernetes and whatever. So you trade > error A for error B. As far as I understood there are a few container runtimes that are supported, i.e docker, podman. And that this might be extended in the future. So, at least in theory, it should work on those. There are even problems with containers if you > don't use version X from docker. That's what the past told us, why > should it be better in the future with even more container environments. Future will tell what container runtimes gives least amount of problems / most benefits. I guess natural selection will do it's thing here. But you got a valid point: there still might be minor version compatibility issues in container runtimes themselves (although I wonder how often those impact Ceph). And even that is something that could be checked by the orchestrator. Still I would recommend to test container deployments just like non-containerized deploys, just to spot these kind of issues. > Have you tried running rancher on debian in the past? It breaks apart > due to iptables or other stuff. I have used rancher, but not on debian, and only for tests. Gr. Stefan
On Tue, Jun 22, 2021 at 11:58 AM Martin Verges <martin.verges@croit.io> wrote:
There is no "should be", there is no one answer to that, other than 42. Containers have been there before Docker, but Docker made them popular, exactly for the same reason as why Ceph wants to use them: ship a known good version (CI tests) of the software with all dependencies, that can be run "as is" on any supported platform.
So ship it tested for container software XXX and run it on YYY. How will that benefit me as a user? There are differences when running a docker container, lxc, nspawn, podman, kubernetes and whatever. So you trade error A for error B. There are even problems with containers if you don't use version X from docker. That's what the past told us, why should it be better in the future with even more container environments. Have you tried running rancher on debian in the past? It breaks apart due to iptables or other stuff.
Rook is based on kubernetes, and cephadm on podman or docker. These are well-defined runtimes. Yes, some have bugs, but our experience so far has been a big improvement over the complexity of managing package dependencies across even just a handful of distros. (Podman has been the only real culprit here, tbh, but I give them a partial pass as the tool is relatively new.)
Hey Sage, thanks for the reply. Sage Weil <sage@newdream.net> writes:
Rook is based on kubernetes, and cephadm on podman or docker. These are well-defined runtimes. Yes, some have bugs, but our experience so far has been a big improvement over the complexity of managing package dependencies across even just a handful of distros. (Podman has been the only real culprit here, tbh, but I give them a partial pass as the tool is relatively new.)
let me come back to a particular part of your message: " our experience so far has been a big improvement over the complexity of managing package dependencies across even just a handful of distros. " This is something I cannot understand at all. This is a process that for most mid sized Open Source projects have automated years ago. And you can even use containers for that! Does it break on distro upgrade? Maybe. Is it trival? Probably in 90% of the cases. Is it easy to spot? Almost always if you use CI. *What* is exactly the complexity that you deal with? Where are the problems? Is there an open issue that says "Reduce time and complexity of package build" open anywhere? Can we join that discussion? The message I get here is: - Building packages is too hard for us - let's make the life of everyone else more complex for a build case that works easier for us I am not sure whether that's the right direction. *If* this this really about complexity of package building, why did you not shout out to the community and ask for help? I assume that one or the other party on this mailing list is open for helping out. Nico -- Sustainable and modern Infrastructures by ungleich.ch
Am 25.06.21 um 17:13 schrieb Nico Schottelius:
*If* this this really about complexity of package building, why did you not shout out to the community and ask for help? I assume that one or the other party on this mailing list is open for helping out.
If I may chime in, package building and hosting for open source projects is solved with the openSUSE build service: https://build.opensuse.org/ But I think what Sage meant was e.g. different versions of GCC on the distributions and not being able to use all the latest features needed for compiling Ceph. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
What I am getting from reading 'between the lines'. Is that they want to create something that is easier to install for a broader target audience. And instead of just saying this, for whatever reasoning other arguments have been put forward which are questionable and therefore raise a discussion here. Thanks for the link and info!
*If* this this really about complexity of package building, why did you not shout out to the community and ask for help? I assume that one or the other party on this mailing list is open for helping out.
If I may chime in, package building and hosting for open source projects is solved with the openSUSE build service:
But I think what Sage meant was e.g. different versions of GCC on the distributions and not being able to use all the latest features needed for compiling Ceph.
GCC, the whole toolchain, myriad dependencies, the ways that Python has patterend itself after Java. Add in the way that the major Linux distributions are moving targets and building / running on just one of them is a huge task, not to mention multiple versions of each. And the way that systems running the same nominal release rarely are completely identical, due to midstream package updates. Heck I’ve seen a mostly-well-run operation that nonetheless had 80 different kernels running due to this. Re the community, remember that “they” *are* the community. Ceph is a big complex hunka burning love that many of us get for FREE. It by and large works beautifully and helps us feed our addictions to food and shelter. Stability and performance continually improve, and we’re largely free of the baggage of proprietary solutions. Ceph is simultaneously sliced bread and a cosmic love pulse matrix. So long as packages can be built, those who want to manage the traditional way still can. The things we quibble about are usually minor, which is a testament to just how much we take for granted. So let’s discuss things that can be made *even better*, and try to respect the those who do the heavy lifting. — aad
But I think what Sage meant was e.g. different versions of GCC on the distributions and not being able to use all the latest features needed for compiling Ceph.
I have seen no arguments why to use containers other than to try and
make it "easier" for new ceph people.
I advise to read the whole thread again, especially Sage his comments, as there are other benefits. It would free up resources that can be dedicated to (arguably) more pressing issues.
I see or hear none of that in cephadm presentations. Are you sure you are not using some future never going to exist feature to substantiate the current state?
Containers are not being used as they should be.
There is no "should be", there is no one answer to that, other than 42.
The expected general ambitionless reply. To me a should be is that container images should run on any container platform. To me a should be is that you ship your container having only the relevant and necessary binaries. Sort of like haproxy (9MB) or java/python where you only have some execution environment included. To me a should be is that the task in the container, is just one process and does not spawn anything else. To me a should be is that the task can be scaled to multiple instances by the OC.
Containers have been there before Docker, but Docker made them popular, exactly for the same reason as why Ceph wants to use them: ship a known good version (CI tests) of the software with all dependencies
Yes yes this is constantly been written. As if this was a huge problem the last decade? Please show me a roadmap of ceph development where this is a high priority. I rather see resources go to the crimson development and testing before a releasing. Do you know your argument of shipping code was quite similar to the senior software engineer of redhat. Who thought it was not a problem that accidentally the root fs of node was mounted in a container. Because the driver should have been ran in a namespace. Is the short version of this not 'hide our crappy code'?
, that can be run "as is" on any supported platform.
This is incorrect, afaik I need to have a platform that has podman not? I would like to run ceph containers on mesos without podman. Where is this man page???? And to be honest how long is docker going to supply these huge images for free? It is not like there is distributed network where you can get your container images. To me this development just looks wrong. It just adds complexity while the beauty is to keep things simple. Today I heard some agent is going to be developed and added to the nodes, I heard something about port conflicts and some issues with ip addressing support. It looks like this OC is developed from the ground up or this podman is seriously lacking functionality. Does it publish automatically assigned ports via srv records, do tasks get dns entries? How is it with dhcp? In any case my current OC does not have such issues, and I am quite happy with it. Why would any company like to administer 2nd OC just for ceph? If ceph goes containers, it should be in the OC the client chooses to use. Ceph just needs to make this work regardless of what OC the client has, not develop something custom. From a security perspective there is also some doubt think twice here. Do you really want to run Grafana, Prometheus and haproxy on the OC that is running your osd's??? The ceph OC has to allow CAP_SYS_ADMIN[1] to tasks to be able to run drivers to mount lvm's. (I can't imagine there are lot of people disabling this like I have done, and run these as external drivers). So if there is any exploit in your ceph OC a single task could destroy a node (and maybe worse, get automatically started on a different node, so it can destroy that one as well) So how is this going to look like when you have a ceph cluster with 100 osd, 3 monitors and 3 rgw, prometheus, prafana. From a performance perspective others have already been writing that containerizing osd's could very well influence latency. I have already seen this with a dns server on a macvtap interface. However I have to admit I do not have the resources to test and give a proper judgement on this. If people here are talking about fixing c-states and frequencies. I wonder how far the containerizing of osd's even goes. Say these are not fully containerized because of performance issues, then you have custom OC from ceph that runs 8 processes containerized from the 108. I would think maybe add this "To me a should be is " list? This ceph containerizing is almost a joke. I will bet you can not even run Prometheus as a stateful task on a ceph rbd image in this CO. Do you know why? Because the csi-ceph driver development team do not grasp the concept of csi and code a driver that can only be used to Kubernetes. Ceph and containers sure, if I may quote a red haired film star "Kill it in the utero" 😉 [1] https://lwn.net/Articles/486306/
On 6/21/21 6:19 PM, Nico Schottelius wrote:
And while we are at claiming "on a lot more platforms", you are at the same time EXCLUDING a lot of platforms by saying "Linux based container" (remember Ceph on FreeBSD? [0]).
Indeed, and that is a more fundamental question: how easy it is to make Ceph a first-class citizen on non linux platforms. Was that ever a (design) goal? But then again, if you would be able to port docker natively to say OpenBSD, you should be able to run Ceph on it as well. Gr. Stefan
On Tue, Jun 22, 2021 at 1:25 PM Stefan Kooman <stefan@bit.nl> wrote:
On 6/21/21 6:19 PM, Nico Schottelius wrote:
And while we are at claiming "on a lot more platforms", you are at the same time EXCLUDING a lot of platforms by saying "Linux based container" (remember Ceph on FreeBSD? [0]).
Indeed, and that is a more fundamental question: how easy it is to make Ceph a first-class citizen on non linux platforms. Was that ever a (design) goal? But then again, if you would be able to port docker natively to say OpenBSD, you should be able to run Ceph on it as well.
Thank you for bringing this up. This is in fact a key reason why the orchestration abstraction works the way it does--to allow other runtime environments to be supported (FreeBSD! sysvinit/Devuan/whatever for systemd haters!) while ALSO allowing an integrated, user-friendly experience in which users workflow for adding/removing hosts, replacing failed OSDs, managing services (MDSs, RGWs, load balancers, etc) can be consistent across all platforms. For 10+ years we basically said "out of scope" to these pesky deployment details and left this job to Puppet, Chef, Ansible, ceph-deploy, rook, etc., but the result of that strategy was pretty clear: ceph was hard to use and the user experience dismal when compared to an integrated product from any half-decent enterprise storage company, or products like Martin's that capitalize on core ceph's bad UX. The question isn't whether we support other environments, but how. As I mentioned in one of my first messages, we can either (1) generalize cephadm to work in other environments (break the current systemd+container requirement), or (2) add another orchestrator backend that supports a new environment. I don't have any well-formed opinion here. There is a lot of pretty generic "orchestration" logic in cephadm right now that isn't related to systemd or containers that could either be pulled out of cephadm into the mgr/ochestrator layer or a library. Or an independent, fresh orch backend implementation could opt for a very different approach or set of opinions. Either way, my assumption has been that these other environments would probably not be docker|podman-based. In the case of FreeBSD we'd probably want to use jails or whatever. But anything is possible. s
I bumped into this recently: https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-with-containerd-1... :) Kevin ________________________________________ From: Sage Weil <sage@newdream.net> Sent: Thursday, June 24, 2021 2:06 PM To: Stefan Kooman Cc: Nico Schottelius; Kai Börnert; Marc; ceph-users Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. On Tue, Jun 22, 2021 at 1:25 PM Stefan Kooman <stefan@bit.nl> wrote:
On 6/21/21 6:19 PM, Nico Schottelius wrote:
And while we are at claiming "on a lot more platforms", you are at the same time EXCLUDING a lot of platforms by saying "Linux based container" (remember Ceph on FreeBSD? [0]).
Indeed, and that is a more fundamental question: how easy it is to make Ceph a first-class citizen on non linux platforms. Was that ever a (design) goal? But then again, if you would be able to port docker natively to say OpenBSD, you should be able to run Ceph on it as well.
Thank you for bringing this up. This is in fact a key reason why the orchestration abstraction works the way it does--to allow other runtime environments to be supported (FreeBSD! sysvinit/Devuan/whatever for systemd haters!) while ALSO allowing an integrated, user-friendly experience in which users workflow for adding/removing hosts, replacing failed OSDs, managing services (MDSs, RGWs, load balancers, etc) can be consistent across all platforms. For 10+ years we basically said "out of scope" to these pesky deployment details and left this job to Puppet, Chef, Ansible, ceph-deploy, rook, etc., but the result of that strategy was pretty clear: ceph was hard to use and the user experience dismal when compared to an integrated product from any half-decent enterprise storage company, or products like Martin's that capitalize on core ceph's bad UX. The question isn't whether we support other environments, but how. As I mentioned in one of my first messages, we can either (1) generalize cephadm to work in other environments (break the current systemd+container requirement), or (2) add another orchestrator backend that supports a new environment. I don't have any well-formed opinion here. There is a lot of pretty generic "orchestration" logic in cephadm right now that isn't related to systemd or containers that could either be pulled out of cephadm into the mgr/ochestrator layer or a library. Or an independent, fresh orch backend implementation could opt for a very different approach or set of opinions. Either way, my assumption has been that these other environments would probably not be docker|podman-based. In the case of FreeBSD we'd probably want to use jails or whatever. But anything is possible. s _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This thread would not be so long if docker/containers solved the problems, but it did not. It solved some, but introduced new ones. So we cannot really say its better now. Again, I think focus should more on a working ceph with clean documentation while leaving software management, packages to admins. And staticilly linked binaries would certinly solve dependecy hell and "how to support other environments" for most of the cases. On Thu, 24 Jun 2021, 23:06 Sage Weil, <sage@newdream.net> wrote:
On Tue, Jun 22, 2021 at 1:25 PM Stefan Kooman <stefan@bit.nl> wrote:
On 6/21/21 6:19 PM, Nico Schottelius wrote:
And while we are at claiming "on a lot more platforms", you are at the same time EXCLUDING a lot of platforms by saying "Linux based container" (remember Ceph on FreeBSD? [0]).
Indeed, and that is a more fundamental question: how easy it is to make Ceph a first-class citizen on non linux platforms. Was that ever a (design) goal? But then again, if you would be able to port docker natively to say OpenBSD, you should be able to run Ceph on it as well.
Thank you for bringing this up. This is in fact a key reason why the orchestration abstraction works the way it does--to allow other runtime environments to be supported (FreeBSD! sysvinit/Devuan/whatever for systemd haters!) while ALSO allowing an integrated, user-friendly experience in which users workflow for adding/removing hosts, replacing failed OSDs, managing services (MDSs, RGWs, load balancers, etc) can be consistent across all platforms. For 10+ years we basically said "out of scope" to these pesky deployment details and left this job to Puppet, Chef, Ansible, ceph-deploy, rook, etc., but the result of that strategy was pretty clear: ceph was hard to use and the user experience dismal when compared to an integrated product from any half-decent enterprise storage company, or products like Martin's that capitalize on core ceph's bad UX.
The question isn't whether we support other environments, but how. As I mentioned in one of my first messages, we can either (1) generalize cephadm to work in other environments (break the current systemd+container requirement), or (2) add another orchestrator backend that supports a new environment. I don't have any well-formed opinion here. There is a lot of pretty generic "orchestration" logic in cephadm right now that isn't related to systemd or containers that could either be pulled out of cephadm into the mgr/ochestrator layer or a library. Or an independent, fresh orch backend implementation could opt for a very different approach or set of opinions.
Either way, my assumption has been that these other environments would probably not be docker|podman-based. In the case of FreeBSD we'd probably want to use jails or whatever. But anything is possible.
s _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This thread would not be so long if docker/containers solved the problems, but it did not. It solved some, but introduced new ones. So we cannot really say its better now.
The only thing I can deduct from this thread, is the necessity to create a solution for eg. 'dentists' to install a ceph cluster. Everything really related to container use is moved to the future. The focus of the cephadm development more or less shows this.
Again, I think focus should more on a working ceph with clean documentation while leaving software management, packages to admins. And staticilly linked binaries would certinly solve dependecy hell and "how to support other environments" for most of the cases.
I agree, and I worry that at some point only docker images are going to be available, and/or a CO environment that I do not want.
Hey Sage, Sage Weil <sage@newdream.net> writes:
Thank you for bringing this up. This is in fact a key reason why the orchestration abstraction works the way it does--to allow other runtime environments to be supported (FreeBSD! sysvinit/Devuan/whatever for systemd haters!)
I would like you to stop labeling people who have reasons for not using a specific software as haters. It is not productive to call Ceph developers "GlusterFS haters", nor to call Redhat users Debian haters. It is simple not an accurate representation. Cheers, Nico -- Sustainable and modern Infrastructures by ungleich.ch
On Fri, Jun 25, 2021 at 10:27 AM Nico Schottelius <nico.schottelius@ungleich.ch> wrote:
Hey Sage,
Sage Weil <sage@newdream.net> writes:
Thank you for bringing this up. This is in fact a key reason why the orchestration abstraction works the way it does--to allow other runtime environments to be supported (FreeBSD! sysvinit/Devuan/whatever for systemd haters!)
I would like you to stop labeling people who have reasons for not using a specific software as haters.
It is not productive to call Ceph developers "GlusterFS haters", nor to call Redhat users Debian haters.
It is simple not an accurate representation.
You're right, and I apologize. My intention was to point out that we tried to keep the door open to everyone, even those who might be called "haters", but I clearly missed the mark. sage
-----Original Message----- Sent: Monday, 21 June 2021 16:44 Subject: Re: [ceph-users] Re: Why you might want packages not containers for Ceph deployments
I think the primary goal of a container environments are resource isolation. At least when I read about the history I never read anything about a tool for people to skip to learn something. Containers allow to use mixed version of the same dependency, despite being a shared dependency, doing this in a non container envrionment basically means rebuilding containers But you clearly portray the situation here. Nobody here is against container environment, because every tool has its purpose. But the container solution proposed here is not for the purpose of utilizing container benefits, but for creating a tool so 'I do not know what to do' people can use ceph. And because this is the development perspective and ceph is not really adapted for being used in containers, you get the current friction with accepting cephadm.
Is this like an insult? The point is not about not learning something, but I would choose docker/container over ansible every time for this
So why did you make this a point previously stating something that you were able to use ceph with kubernetes like commands. How can you compare docker/containers to ansible? It is like comparing a drill to fruit mixer or so.
(ansible has its place, but it has no concept of current state, as a ceph orchestrator heavily benefits from).
I kinda feel that you have not worked a lot with containers, created a strawman what you think benefits are, and then continue to burn down that strawman while not even recognizing a lot of benefits.
Your feelings mislead you, you would understand this if you knew what I am writing about.
Eg. this command is wrong in my opinion
ceph orch device ls
What has a device list to do with having an OC or not? Well you need to start a small tool to actually find devices on every system, for this you need to first find all nodes that exist in the cluster, if not the orchestrator, who should have this information, especially since it can change at runtime due to hosts being added. I do not even know why ceph is working on something that uses Kubernetes, let the Kubernetes platform implement a ceph solution. If ceph wants be used in a container environment, start making the daemons ready to run in container environments.
So my question still stands: What problem is it actually that the cephadm dev team is trying to solve? That is not clear to me.
Again, clean orchestration, being able to upgrade each deamon without influencing running ones, this is just not possible with the native packages.
1. Is this even being done then cephadm? I still see commands that upgrade a whole node. 2. Has this ever been a problem in the past? I had no problems going from Kraken -> Luminous -> nautilus. There is no significant problem updating and/or upgrading.
Being able to run on a lot more platforms, as native packages are just not maintainable for all platforms that could easily run a docker daemon
Yes and let me guess the solution is sending a whole centos/rhel container image? Not!
On Sun, Jun 20, 2021 at 9:51 AM Marc <Marc@f1-outsourcing.eu> wrote:
Remarks about your cephadm approach/design:
1. I am not interested in learning podman, rook or kubernetes. I am using mesos which is also on my osd nodes to use the extra available memory and cores. Furthermore your cephadm OC is limited to only ceph nodes. While my mesos OC is spread across a larger cluster and has rules when, and when not to run tasks on the osd nodes. You incorrectly assume that rgw, grafana, prometheus, haproxy are going to be ran on your ceph OC.
rgw, grafana, prom, haproxy, etc are all optional components. The monitoring stack is deployed by default but is trivially disabled via a flag to the bootstrap command. We are well aware that not everyone wants these, but we cannot ignore the vast majority of users that wants things to Just Work without figuring out how to properly deploy and manage all of these extraneous integrated components.
2. Nico pointed out that you do not have alpine linux container images. I did not even know you were using container images. So how big are these? Where are these stored. And why are these not as small as they can be? Such an osd container image should be 20MB or so at most. I would even expect statically build binary container image, why even a tiny os? 4. Ok found the container images[2] (I think). Sorry but this has ‘nothing’ to do with container thinking. I expected to find container images for osd, msd, rgw separately and smaller. This looks more like an OS deployment. Early on the team building the container images opted for a single image that includes all of the daemons for simplicity. We could build stripped down images for each daemon type, but that's an investment in developer time and complexity and we haven't heard any complaints about the container size. (Usually a few hundred MB on a large scale storage server isn't a problem.)
3. Why is in this cephadm still being talked about systemd? Your orchestrator should handle restarts,namespaces and failed tasks not? There should be no need to have a systemd dependency, at least I have not seen any container images relying on this.
Something needs to start the ceph daemon containers when the system reboots. We integrated with systemd since all major distros adopted it. Cephadm could be extended to support other init systems with pretty minimal effort... we aren't doing anything fancy with systemd.
5. I have been writing this previously on the mailing list here. Is each rgw still requiring its own dedicated client id? Is it still true, that if you want to spawn 3 rgw instances, they need to authorize like client.rgw1, client.rgw2 and client.rgw3? This does not allow for auto scaling. The idea of using an OC is that you launch a task, and that you can scale this task automatically when necessary. So you would get multiple instances of rgw1. If this is still and issue with rgw, mds and mgr etc. Why even bother doing something with an OC and containers?
The orchestrator automates the creation and cleanup of credentials for each rgw instance. (It also trivially scales them up/down, ala k8s.) If you have an autoscaler, you just need to tell cephadm how many you want and it will add/remove daemons. If you are using cephadm's ingress (haproxy) capability, the LB configuration will be adjusted for you. If you are using an external LB, you can query cephadm for a description of the current daemons and their endpoints and feed that info into your own ingress solution.
6. As I wrote before I do not want my rgw or haproxy running in a OC that has the ability to give tasks capability SYSADMIN. So that would mean I have to run my osd daemons/containers separately.
Only the OSD containers get extra caps to deal with the storage hardware.
7. If you are not setting cpu and memory limits on your cephadm containers, then again there is an argument why even use containers.
Memory limits are partially implemented; we haven't gotten to CPU limits yet. It's on the list!
8. I still see lots of comments on the mailing list about accessing logs. I have all my containers log to a remote syslog server, if you still have your ceph daemons that can not do this (correctly). What point is it even going to containers.
By default, we log to stderr and your logs are in journald or whatever alternative your container runtime has set up. You can trivially flip a switch and you get traditional file-based logs with a logrotated.d config, primary to satisfy users (like me!) who aren't comfortable with the newfangled log management style.
9. I am updating my small cluster something like this:
ssh root@c01 "ceph osd set noout ; ceph osd set noscrub ; ceph osd set nodeep-scrub" ssh root@c01 "ceph tell osd.* injectargs '--osd_max_scrubs=0'"
ssh root@c01 "yum update 'ceph-*' -y" ...
ssh root@c01 "service ceph-mon@a restart" ...
ssh root@c01 "service ceph-mgr@a restart" ...
# wait for up and recovery to finish ssh root@c01 "systemctl restart 'ceph-osd@*'" …
I am never going to run a ‘ceph orch upgrade start –ceph-version 16.2.0’. I want to see if everything is ok after each command I issue. I want to see if scrubbing stopped, I want to see if osd have correctly accepted the new config. I have a small cluster so I do not see this procedure as a waste of time. If I look at your telemetry data[3]. I see 600 clusters with 35k osd’s, that is an average of 60 osd per cluster. So these are quite small clusters, I would think these admins have a similar point of view as I have.
That leaves these big clusters of >3k osd’s. I wonder what these admins require, are they at CERN really waiting for something like cephadm?
I humbly contend that most users, especially those with small clusters, would rather issue a single command and have the cluster upgrade itself--with all of the latest and often version-specific safety checks and any special per-release steps implemented for them--than to do it themselves. sage
I've actually had rook-ceph not proceed with something that I would have continued on with. Turns out I was wrong and it was right. Its checking was more through then mine. Thought that was pretty cool. It eventually cleared itself and finished up. For a large ceph cluster, the orchestration is very nice. Thanks, Kevin ________________________________________ From: Sage Weil <sage@newdream.net> Sent: Thursday, June 24, 2021 1:46 PM To: Marc Cc: Anthony D'Atri; Nico Schottelius; Matthew Vernon; ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. On Sun, Jun 20, 2021 at 9:51 AM Marc <Marc@f1-outsourcing.eu> wrote:
Remarks about your cephadm approach/design:
1. I am not interested in learning podman, rook or kubernetes. I am using mesos which is also on my osd nodes to use the extra available memory and cores. Furthermore your cephadm OC is limited to only ceph nodes. While my mesos OC is spread across a larger cluster and has rules when, and when not to run tasks on the osd nodes. You incorrectly assume that rgw, grafana, prometheus, haproxy are going to be ran on your ceph OC.
rgw, grafana, prom, haproxy, etc are all optional components. The monitoring stack is deployed by default but is trivially disabled via a flag to the bootstrap command. We are well aware that not everyone wants these, but we cannot ignore the vast majority of users that wants things to Just Work without figuring out how to properly deploy and manage all of these extraneous integrated components.
2. Nico pointed out that you do not have alpine linux container images. I did not even know you were using container images. So how big are these? Where are these stored. And why are these not as small as they can be? Such an osd container image should be 20MB or so at most. I would even expect statically build binary container image, why even a tiny os? 4. Ok found the container images[2] (I think). Sorry but this has ‘nothing’ to do with container thinking. I expected to find container images for osd, msd, rgw separately and smaller. This looks more like an OS deployment. Early on the team building the container images opted for a single image that includes all of the daemons for simplicity. We could build stripped down images for each daemon type, but that's an investment in developer time and complexity and we haven't heard any complaints about the container size. (Usually a few hundred MB on a large scale storage server isn't a problem.)
3. Why is in this cephadm still being talked about systemd? Your orchestrator should handle restarts,namespaces and failed tasks not? There should be no need to have a systemd dependency, at least I have not seen any container images relying on this.
Something needs to start the ceph daemon containers when the system reboots. We integrated with systemd since all major distros adopted it. Cephadm could be extended to support other init systems with pretty minimal effort... we aren't doing anything fancy with systemd.
5. I have been writing this previously on the mailing list here. Is each rgw still requiring its own dedicated client id? Is it still true, that if you want to spawn 3 rgw instances, they need to authorize like client.rgw1, client.rgw2 and client.rgw3? This does not allow for auto scaling. The idea of using an OC is that you launch a task, and that you can scale this task automatically when necessary. So you would get multiple instances of rgw1. If this is still and issue with rgw, mds and mgr etc. Why even bother doing something with an OC and containers?
The orchestrator automates the creation and cleanup of credentials for each rgw instance. (It also trivially scales them up/down, ala k8s.) If you have an autoscaler, you just need to tell cephadm how many you want and it will add/remove daemons. If you are using cephadm's ingress (haproxy) capability, the LB configuration will be adjusted for you. If you are using an external LB, you can query cephadm for a description of the current daemons and their endpoints and feed that info into your own ingress solution.
6. As I wrote before I do not want my rgw or haproxy running in a OC that has the ability to give tasks capability SYSADMIN. So that would mean I have to run my osd daemons/containers separately.
Only the OSD containers get extra caps to deal with the storage hardware.
7. If you are not setting cpu and memory limits on your cephadm containers, then again there is an argument why even use containers.
Memory limits are partially implemented; we haven't gotten to CPU limits yet. It's on the list!
8. I still see lots of comments on the mailing list about accessing logs. I have all my containers log to a remote syslog server, if you still have your ceph daemons that can not do this (correctly). What point is it even going to containers.
By default, we log to stderr and your logs are in journald or whatever alternative your container runtime has set up. You can trivially flip a switch and you get traditional file-based logs with a logrotated.d config, primary to satisfy users (like me!) who aren't comfortable with the newfangled log management style.
9. I am updating my small cluster something like this:
ssh root@c01 "ceph osd set noout ; ceph osd set noscrub ; ceph osd set nodeep-scrub" ssh root@c01 "ceph tell osd.* injectargs '--osd_max_scrubs=0'"
ssh root@c01 "yum update 'ceph-*' -y" ...
ssh root@c01 "service ceph-mon@a restart" ...
ssh root@c01 "service ceph-mgr@a restart" ...
# wait for up and recovery to finish ssh root@c01 "systemctl restart 'ceph-osd@*'" …
I am never going to run a ‘ceph orch upgrade start –ceph-version 16.2.0’. I want to see if everything is ok after each command I issue. I want to see if scrubbing stopped, I want to see if osd have correctly accepted the new config. I have a small cluster so I do not see this procedure as a waste of time. If I look at your telemetry data[3]. I see 600 clusters with 35k osd’s, that is an average of 60 osd per cluster. So these are quite small clusters, I would think these admins have a similar point of view as I have.
That leaves these big clusters of >3k osd’s. I wonder what these admins require, are they at CERN really waiting for something like cephadm?
I humbly contend that most users, especially those with small clusters, would rather issue a single command and have the cluster upgrade itself--with all of the latest and often version-specific safety checks and any special per-release steps implemented for them--than to do it themselves. sage _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
rgw, grafana, prom, haproxy, etc are all optional components. The
Is this Prometheus stateful? Where is this data stored?
Early on the team building the container images opted for a single image that includes all of the daemons for simplicity. We could build stripped down images for each daemon type, but that's an investment in developer time and complexity and we haven't heard any complaints about the container size. (Usually a few hundred MB on a large scale storage server isn't a problem.)
To me it looks like you do not take the containerization seriously, a container development team that does not want to spend time on container images. You create something >10x slower to start, >10x more disk space used (times 2 when upgrading). Haproxy is 9MB. Your osd is 350MB.
5. I have been writing this previously on the mailing list here. Is each rgw still requiring its own dedicated client id? Is it still true, that if you want to spawn 3 rgw instances, they need to authorize like client.rgw1, client.rgw2 and client.rgw3? This does not allow for auto scaling. The idea of using an OC is that you launch a task, and that you can scale this task automatically when necessary. So you would get multiple instances of rgw1. If this is still and issue with rgw, mds and mgr etc. Why even bother doing something with an OC and containers?
The orchestrator automates the creation and cleanup of credentials for each rgw instance. (It also trivially scales them up/down, ala k8s.)
I do not understand this. This sounds more to me like creating a new task, instead of scaling a second instance of an existing task. Are you currently able to automatically scale up/down instances of a rgw or is your statement hypothetical? I can remember on the mesos mailing list/issue tracker talk about the difficulty of determining a tasks 'number' . Because tasks are being killed/started at random, based on resource offers. Thus supplying them with the correct different credentials is not as trivial as it would seem. So I wonder how you are scaling this? If there are already so many differences between OC's, I would even recon they differ in this area quite a lot. So the most plausible solution would be fixing this in at the rgw daemon.
If you have an autoscaler, you just need to tell cephadm how many you want and it will add/remove daemons. If you are using cephadm's ingress (haproxy) capability, the LB configuration will be adjusted for you. If you are using an external LB, you can query cephadm for a description of the current daemons and their endpoints and feed that info into your own ingress solution.
Forgive me for not looking at all the video links before writing this. But from the video's I saw about cephadm it was more always like a command reference. Would be nice to maybe show the above in ceph tech talk or so. I think a lot of people would be interested seeing this.
6. As I wrote before I do not want my rgw or haproxy running in a OC that has the ability to give tasks capability SYSADMIN. So that would mean I have to run my osd daemons/containers separately.
Only the OSD containers get extra caps to deal with the storage hardware.
I know, that is why I choose to run drivers that require such SYSADMIN rights, to run outside of my OC environment. My OC environment does not allow any tasks to use the SYSADMIN.
Memory limits are partially implemented; we haven't gotten to CPU limits yet. It's on the list!
To me it is sort of clear what the focus of the cephadm team is.
I humbly contend that most users,
Hmmmm, most, most, most is not most mostly the average? Most people drive a Toyota, less people drive Porsche and even less drive a Ferrari. It is your choice who your target audience is and what you are 'selling' them.
especially those with small clusters, would rather issue a single command and have the cluster upgrade itself--with all of the latest and often version-specific safety checks and any special per-release steps implemented for them--than to do it themselves.
The flip side to this approach is. That if you guys make a mistake in some script, lots of ceph clusters could go down. Is this not a bit of a paradox, a team that has problems with their software dependencies (ceph-ansible/ceph-deploy?), I should blindly trust to script the update of my cluster? I know I have been very critical/sceptical about this cephadm. Please do also note I just love this ceph storage, and I am advertising whenever possible. So a big thanks to the whole team still!!!
Orchestration is hard, especially with every permutation. The devs have implemented what they feel is the right solution for their own needs from the sound of it. The orchestration was made modular to support non containerized deployment. It just takes someone to step up and implement the permutations desired. And ultimately that's what opensource is geared towards. With opensource and some desired feature, you can: 1. Implement it 2. Pay someone else to implement it 3. Convince someone else to implement it in their spare time. The thread seems to be currently focused around #3 but no developer seems to be interested in implementing it. So that leaves options 1 and 2? To move this forward, is anyone interested in developing package support in the orchestration system or paying to have it implemented? ________________________________________ From: Oliver Freyermuth <freyermuth@physik.uni-bonn.de> Sent: Wednesday, June 2, 2021 2:26 PM To: Matthew Vernon; ceph-users@ceph.io Subject: [ceph-users] Re: Why you might want packages not containers for Ceph deployments Check twice before you click! This email originated from outside PNNL. Hi, that's also a +1 from me — we also use containers heavily for scientific workflows, and know their benefits well. But they are not the "best", or rather, the most fitting tool in every situation. You have provided a great summary and I agree with all points, and thank you a lot for this very competent and concise write-up. Since in this lengthy thread, static linking and solving the issue of many inter-dependencies for production services with containers have been mentioned as solutions, I'd like to add another point to your list of complexities: * Keeping production systems secure may be a lot more of a hassle. Even though the following article is long and many may regard it as controversial, I'd like to link to a concise write-up from a packager discussing this topic in a quite generic way: https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblogs.gent... While the article discusses the issues of static linking and package management performed in language-specific domains, it applies all the same to containers. If I operate services in containers built by developers, of course this ensures the setup works, and dependencies are well tested, and even upgrades work well — but it also means that, at the end of the day, if I run 50 services in 50 different containers from 50 different upstreams, I'll have up to 50 different versions of OpenSSL floating around my production servers. If a security issue is found in any of the packages used in all the container images, I now need to trust the security teams of all the 50 developer groups building these containers (and most FOSS projects won't have the ressources, understandably...), instead of the one security team of the disto I use. And then, I also have to re-pull all these containers, after finding out that a security fix has become available. Or I need to build all these containers myself, and effectively take over the complete job, and have my own security team. This may scale somewhat well, if you have a team of 50 people, and every person takes care of one service. Containers are often your friend in this case[1], since it allows to isolate the different responsibilities along with the service. But this is rarely the case outside of industry, and especially not in academics. So the approach we chose for us is to have one common OS everywhere, and automate all of our deployment and configuration management with Puppet. Of course, that puts is in one of the many corners out there, but it scales extremely well to all services we operate, and I can still trust the distro maintainers to keep the base OS safe on all our servers, automate reboots etc. For Ceph, we've actually seen questions about security issues already on the list[0] (never answered AFAICT). To conclude, I strongly believe there's no one size fits all here. That was why I was hopeful when I first heard about the Ceph orchestrator idea, when it looked to be planned out to be modular, with the different tasks being implementable in several backends, so one could imagine them being implemented with containers, with classic SSH on bare-metal (i.e. ceph-deploy-like), ansible, rook or maybe others. Sadly, it seems it ended up being "container-only". Containers certainly have many uses, and we run thousands of them daily, but neither do they fit each and every existing requirement, nor are they a magic bullet to solve all issues. Cheers, Oliver [0] https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ceph... [1] But you may also just have a very well structured configuration management system fitting your organizational structure. Am 02.06.21 um 11:36 schrieb Matthew Vernon:
Hi,
In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0].
I'd like to suggest, briefly, that:
* containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future
We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes.
For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice.
Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages):
* you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice)
We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part.
Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems!
I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder".
I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :)
Regards, and apologies for the wall of text,
Matthew
[0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming
-- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.047 Nußallee 12 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --
Orchestration is hard, especially with every permutation. The devs have implemented what they feel is the right solution for their own needs from the sound of it. The orchestration was made modular to support non containerized deployment. It just takes someone to step up and implement the permutations desired. And ultimately that's what opensource is geared towards. With opensource and some desired feature, you can: 1. Implement it 2. Pay someone else to implement it 3. Convince someone else to implement it in their spare time.
The thread seems to be currently focused around #3 but no developer seems to be interested in implementing it. So that leaves options 1 and 2?
Imho a bit simplistic view to opensource and the current thread.
participants (26)
-
Anthony D'Atri
-
Daniel Baumann
-
Dave Hall
-
DHilsbos＠performair.com
-
Eneko Lacunza
-
Erik Lindahl
-
Fox, Kevin M
-
Harry G. Coin
-
Kai Börnert
-
Magnus HAGDORN
-
Marc
-
Martin Verges
-
Matthew Vernon
-
maxadamo＠gmail.com
-
Nico Schottelius
-
Oliver Freyermuth
-
Phil Regnauld
-
Reed Dier
-
Robert Sander
-
Robert W. Eckert
-
Rok Jaklič
-
Sage Weil
-
Sasha Litvak
-
Stefan Kooman
-
Szabo, Istvan (Agoda)
-
胡 玮文