4 Jun
2021
4 Jun
'21
7:09 p.m.
Do you use rbd images in containers that are residing on osd nodes? Does this give any
problems? I used to have kernel mounted cephfs on a osd node, after a specific luminous
release this was giving me problems.
-----Original Message-----
From: Eneko Lacunza <elacunza(a)binovo.es>
Sent: Friday, 4 June 2021 15:49
To: ceph-users(a)ceph.io
Subject: *****SPAM***** [ceph-users] Re: Why you might want packages
not containers for Ceph deployments
Hi,
We operate a few Ceph hyperconverged clusters with Proxmox, that
provides a custom ceph package repository. They do a great work; and
deployment is a brezee.
So, even as currently we would rely on Proxmox packages/distribution and
not upstream, we have a number of other projects deployed with
containers and we even distribute some of our own development in deb and
container packages, so I will comment on our view:
El 2/6/21 a las 23:26, Oliver Freyermuth escribió:
[...]
If I operate services in containers built by developers, of course
this ensures the setup works, and dependencies are well tested, and
even upgrades work well — but it also means that,
at the end of the day, if I run 50 services in 50 different containers
from 50 different upstreams, I'll have up to 50 different versions of
OpenSSL floating around my production servers.
If a security issue is found in any of the packages used in all the
container images, I now need to trust the security teams of all the 50
developer groups building these containers
(and most FOSS projects won't have the ressources, understandably...),
instead of the one security team of the disto I use. And then, I also
have to re-pull all these containers, after finding out that a
security fix has become available.
Or I need to build all these containers myself, and effectively take
over the complete job, and have my own security team.
This may scale somewhat well, if you have a team of 50 people, and
every person takes care of one service. Containers are often your
friend in this case[1],
since it allows to isolate the different responsibilities along with
the service.
But this is rarely the case outside of industry, and especially not in
academics.
So the approach we chose for us is to have one common OS everywhere,
and automate all of our deployment and configuration management with
Puppet.
Of course, that puts is in one of the many corners out there, but it
scales extremely well to all services we operate,
and I can still trust the distro maintainers to keep the base OS safe
on all our servers, automate reboots etc.
For Ceph, we've actually seen questions about security issues already
on the list[0] (never answered AFAICT).
These are the two main issues I find with containers really:
- Keeping hosts uptodate is more complex (apt-get update+apt-get
dist-upgrade and also some kind of docker pull+docker
restart/docker-compose up ...). Much of the time the second part is not
standard (just deployed a Harbor service, upgrade is quite simple but I
have to know how to do it as it's speciffic, maintenance would be much
easier if it was packaged in Debian). I won't say it's more difficult,
but it will be more diverse and complex.
- Container image quality and security support quality, that will vary
from upstream to upstream. You have to research each of them to know
were they stand. A distro (specially a good one like Debian, Ubuntu,
RHEL or SUSE) has known, quality security support for the repositories.
They will even fix issues not fixed by upstream (o backport them to
distro's version...). This is more an upstream vs distro issue, really.
About debugging issues reported with Ceph containers, I think those are
things waiting for a fix: why are logs writen in container image (or an
ephemeral volume, I don't know really how is that done right now)
instead of an external name volume o a local mapped dir in /var/log/ceph ?
All that said, I think that it makes sense for an upstream project like
Ceph, to distribute container images, as it is the most generic way to
distribute (you can deploy on any system/distro supporting container
images) and eases development. But only distributing container images
could make more users depend on third party distribution (global or
specific distros), which would delay feeback/bugreport to upstream.
Cheers and thanks for the great work!
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7 Jun
7 Jun
12:24 p.m.
New subject: Why you might want packages not containers for Ceph deployments
Hi Marc,
El 4/6/21 a las 16:39, Marc escribió:
Do you use rbd images in containers that are residing
on osd nodes? Does this give any problems? I used to have kernel mounted cephfs on a osd
node, after a specific luminous release this was giving me problems.
No, we use Ceph
for VM storage. Some of the VMs host containers.
Cheers
> -----Original Message-----
> From: Eneko Lacunza <elacunza(a)binovo.es>
> Sent: Friday, 4 June 2021 15:49
> To: ceph-users(a)ceph.io
> Subject: *****SPAM***** [ceph-users] Re: Why you might want packages
> not containers for Ceph deployments
>
> Hi,
>
> We operate a few Ceph hyperconverged clusters with Proxmox, that
> provides a custom ceph package repository. They do a great work; and
> deployment is a brezee.
>
> So, even as currently we would rely on Proxmox packages/distribution and
> not upstream, we have a number of other projects deployed with
> containers and we even distribute some of our own development in deb and
> container packages, so I will comment on our view:
>
> El 2/6/21 a las 23:26, Oliver Freyermuth escribió:
> [...]
>> If I operate services in containers built by developers, of course
>> this ensures the setup works, and dependencies are well tested, and
>> even upgrades work well — but it also means that,
>> at the end of the day, if I run 50 services in 50 different containers
>> from 50 different upstreams, I'll have up to 50 different versions of
>> OpenSSL floating around my production servers.
>> If a security issue is found in any of the packages used in all the
>> container images, I now need to trust the security teams of all the 50
>> developer groups building these containers
>> (and most FOSS projects won't have the ressources, understandably...),
>> instead of the one security team of the disto I use. And then, I also
>> have to re-pull all these containers, after finding out that a
>> security fix has become available.
>> Or I need to build all these containers myself, and effectively take
>> over the complete job, and have my own security team.
>>
>> This may scale somewhat well, if you have a team of 50 people, and
>> every person takes care of one service. Containers are often your
>> friend in this case[1],
>> since it allows to isolate the different responsibilities along with
>> the service.
>>
>> But this is rarely the case outside of industry, and especially not in
>> academics.
>> So the approach we chose for us is to have one common OS everywhere,
>> and automate all of our deployment and configuration management with
>> Puppet.
>> Of course, that puts is in one of the many corners out there, but it
>> scales extremely well to all services we operate,
>> and I can still trust the distro maintainers to keep the base OS safe
>> on all our servers, automate reboots etc.
>>
>> For Ceph, we've actually seen questions about security issues already
>> on the list[0] (never answered AFAICT).
> These are the two main issues I find with containers really:
>
> - Keeping hosts uptodate is more complex (apt-get update+apt-get
> dist-upgrade and also some kind of docker pull+docker
> restart/docker-compose up ...). Much of the time the second part is not
> standard (just deployed a Harbor service, upgrade is quite simple but I
> have to know how to do it as it's speciffic, maintenance would be much
> easier if it was packaged in Debian). I won't say it's more difficult,
> but it will be more diverse and complex.
>
> - Container image quality and security support quality, that will vary
> from upstream to upstream. You have to research each of them to know
> were they stand. A distro (specially a good one like Debian, Ubuntu,
> RHEL or SUSE) has known, quality security support for the repositories.
> They will even fix issues not fixed by upstream (o backport them to
> distro's version...). This is more an upstream vs distro issue, really.
>
> About debugging issues reported with Ceph containers, I think those are
> things waiting for a fix: why are logs writen in container image (or an
> ephemeral volume, I don't know really how is that done right now)
> instead of an external name volume o a local mapped dir in /var/log/ceph ?
>
> All that said, I think that it makes sense for an upstream project like
> Ceph, to distribute container images, as it is the most generic way to
> distribute (you can deploy on any system/distro supporting container
> images) and eases development. But only distributing container images
> could make more users depend on third party distribution (global or
> specific distros), which would delay feeback/bugreport to upstream.
>
> Cheers and thanks for the great work!
>
> Eneko Lacunza
> Zuzendari teknikoa | Director técnico
> Binovo IT Human Project
>
> Tel. +34 943 569 206 | https://www.binovo.es
> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
>
> https://www.youtube.com/user/CANALBINOVO
> https://www.linkedin.com/company/37269706/
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
8:37 p.m.
New subject: Why you might want packages not containers for Ceph deployments
We have no one currently using containers for anything. *Therefore, we
run old CEPH code to avoid them.
If there was an option to not do containers on modern CEPH, that would
be better for alot of people who don't want them.
-Ed
On 6/7/2021 2:54 AM, Eneko Lacunza wrote:
Hi Marc,
El 4/6/21 a las 16:39, Marc escribió:
--
Thank you for your time,
Edward H. Kalk IV
Information Technology Dept.
Server Specialist
Datacenter Virtualization and Storage Systems
Socket Telecom, LLC.
2703 Clark Lane
Columbia, MO 65202
573-817-0000 or 800-socket3 X218
Do you use rbd images in containers that are
residing on osd nodes?
Does this give any problems? I used to have kernel mounted cephfs on
a osd node, after a specific luminous release this was giving me
problems.
No, we use Ceph for VM storage. Some of the VMs host containers.
Cheers
> -----Original Message-----
> From: Eneko Lacunza <elacunza(a)binovo.es>
> Sent: Friday, 4 June 2021 15:49
> To: ceph-users(a)ceph.io
> Subject: *****SPAM***** [ceph-users] Re: Why you might want packages
> not containers for Ceph deployments
>
> Hi,
>
> We operate a few Ceph hyperconverged clusters with Proxmox, that
> provides a custom ceph package repository. They do a great work; and
> deployment is a brezee.
>
> So, even as currently we would rely on Proxmox packages/distribution
> and
> not upstream, we have a number of other projects deployed with
> containers and we even distribute some of our own development in deb
> and
> container packages, so I will comment on our view:
>
> El 2/6/21 a las 23:26, Oliver Freyermuth escribió:
> [...]
>> If I operate services in containers built by developers, of course
>> this ensures the setup works, and dependencies are well tested, and
>> even upgrades work well — but it also means that,
>> at the end of the day, if I run 50 services in 50 different containers
>> from 50 different upstreams, I'll have up to 50 different versions of
>> OpenSSL floating around my production servers.
>> If a security issue is found in any of the packages used in all the
>> container images, I now need to trust the security teams of all the 50
>> developer groups building these containers
>> (and most FOSS projects won't have the ressources, understandably...),
>> instead of the one security team of the disto I use. And then, I also
>> have to re-pull all these containers, after finding out that a
>> security fix has become available.
>> Or I need to build all these containers myself, and effectively take
>> over the complete job, and have my own security team.
>>
>> This may scale somewhat well, if you have a team of 50 people, and
>> every person takes care of one service. Containers are often your
>> friend in this case[1],
>> since it allows to isolate the different responsibilities along with
>> the service.
>>
>> But this is rarely the case outside of industry, and especially not in
>> academics.
>> So the approach we chose for us is to have one common OS everywhere,
>> and automate all of our deployment and configuration management with
>> Puppet.
>> Of course, that puts is in one of the many corners out there, but it
>> scales extremely well to all services we operate,
>> and I can still trust the distro maintainers to keep the base OS safe
>> on all our servers, automate reboots etc.
>>
>> For Ceph, we've actually seen questions about security issues already
>> on the list[0] (never answered AFAICT).
> These are the two main issues I find with containers really:
>
> - Keeping hosts uptodate is more complex (apt-get update+apt-get
> dist-upgrade and also some kind of docker pull+docker
> restart/docker-compose up ...). Much of the time the second part is not
> standard (just deployed a Harbor service, upgrade is quite simple but I
> have to know how to do it as it's speciffic, maintenance would be much
> easier if it was packaged in Debian). I won't say it's more difficult,
> but it will be more diverse and complex.
>
> - Container image quality and security support quality, that will vary
> from upstream to upstream. You have to research each of them to know
> were they stand. A distro (specially a good one like Debian, Ubuntu,
> RHEL or SUSE) has known, quality security support for the repositories.
> They will even fix issues not fixed by upstream (o backport them to
> distro's version...). This is more an upstream vs distro issue, really.
>
> About debugging issues reported with Ceph containers, I think those are
> things waiting for a fix: why are logs writen in container image (or an
> ephemeral volume, I don't know really how is that done right now)
> instead of an external name volume o a local mapped dir in
> /var/log/ceph ?
>
> All that said, I think that it makes sense for an upstream project like
> Ceph, to distribute container images, as it is the most generic way to
> distribute (you can deploy on any system/distro supporting container
> images) and eases development. But only distributing container images
> could make more users depend on third party distribution (global or
> specific distros), which would delay feeback/bugreport to upstream.
>
> Cheers and thanks for the great work!
>
> Eneko Lacunza
> Zuzendari teknikoa | Director técnico
> Binovo IT Human Project
>
> Tel. +34 943 569 206 | https://www.binovo.es
> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
>
> https://www.youtube.com/user/CANALBINOVO
> https://www.linkedin.com/company/37269706/
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1073
days inactive
1076
days old
2 comments
3 participants
participants (3)
-
Ed Kalk -
Eneko Lacunza -
Marc