12 Nov
2019
12 Nov
'19
5:52 p.m.
Right now the way ceph-daemon is used by the ssh orchestrator is designed
to minimize the dependencies/setup complexity. The only requirements for
a host to be added to the cluster are
- python (2 or 3)
- systemd
- either podman or docker installed
- the ceph cluster's pub key in /root/.ssh/authorized_keys
No other software (including Ceph) needs to be installed. The mgr/ssh
module invokes ceph-daemon on the remote host by running /usr/bin/python
over ssh and piping the cluster's version of ceph-daemon to stdin.
The downside to this approach is that some users might not like the idea
of ceph having an ssh key with root access. For large clusters I'm not
sure how much this really matters--if you pwn ceph you can delete TB to PB
of data so do you really care if someone has root?--but for hyperconverged
cases this might be a problem.
One alternative might be to
- create a ceph user on the node, and put the cluster's key in
that user's authorized_keys
- install a package that includes ceph-daemon (/usr/bin/ceph-damaen)
- install an /etc/sudoers.d/ceph file that lets the ceph user
'sudo ceph-daemon ...'
Cons:
- This makes the bootstrap process slightly more complicated: (1) install
package, (2) create user, (3) install ssh key (vs just #3).
- The remote version of ceph-daemon can get out of sync with the
cluster.. either stale and missing some feature, or even too new and
not behaving the way the cluster expects.
Pros:
- This limits the attack surface area (if someone manages to get the
cluster's ssh key) to the functions that ceph-daemon implements, vs
full root.
We could mitigate the 'keep ceph-daemon up to date' problem somewhat by
implementing a 'ceph-daemon update' function that will apt/dnf/yum install
ceph-daemon on the local host, so that the cluster could self-update the
remote host.
Or... we could skip the package entirely and install the ceph-daemon
script in /home/ceph/ceph-daemon, and include an update function that
updates the script in place. That is less complicated than knowing how to
apt/dnf/yum install a package for all the random distros and repo location
combinations (one of the biggest benefits of containers IMO). But it
still requires some sort of process to keep ceph-daemon up to date.
Maybe if we always pass an md5sum to ceph-daemon whenever we invoke it to
assert that we are running the version we want, and if there is a
mismatch, ceph-daemon bails out with a special exit code that triggers and
update and retry?
Anyway, what are people's thoughts here? How much more complicated are we
interested or willing to make this to make people more comfortable with
the idea that ceph owns an ssh key?
sage
12 Nov
12 Nov
6:08 p.m.
On Tue, Nov 12, 2019 at 9:52 AM Sage Weil <sweil(a)redhat.com> wrote:
Right now the way ceph-daemon is used by the ssh orchestrator is designed
to minimize the dependencies/setup complexity. The only requirements for
a host to be added to the cluster are
- python (2 or 3)
- systemd
- either podman or docker installed
- the ceph cluster's pub key in /root/.ssh/authorized_keys
No other software (including Ceph) needs to be installed. The mgr/ssh
module invokes ceph-daemon on the remote host by running /usr/bin/python
over ssh and piping the cluster's version of ceph-daemon to stdin.
The downside to this approach is that some users might not like the idea
of ceph having an ssh key with root access. For large clusters I'm not
sure how much this really matters--if you pwn ceph you can delete TB to PB
of data so do you really care if someone has root?--but for hyperconverged
cases this might be a problem.
One alternative might be to
- create a ceph user on the node, and put the cluster's key in
that user's authorized_keys
- install a package that includes ceph-daemon (/usr/bin/ceph-damaen)
- install an /etc/sudoers.d/ceph file that lets the ceph user
'sudo ceph-daemon ...'
Cons:
- This makes the bootstrap process slightly more complicated: (1) install
package, (2) create user, (3) install ssh key (vs just #3).
- The remote version of ceph-daemon can get out of sync with the
cluster.. either stale and missing some feature, or even too new and
not behaving the way the cluster expects.
Pros:
- This limits the attack surface area (if someone manages to get the
cluster's ssh key) to the functions that ceph-daemon implements, vs
full root.
We could mitigate the 'keep ceph-daemon up to date' problem somewhat by
implementing a 'ceph-daemon update' function that will apt/dnf/yum install
ceph-daemon on the local host, so that the cluster could self-update the
remote host.
Or... we could skip the package entirely and install the ceph-daemon
script in /home/ceph/ceph-daemon, and include an update function that
updates the script in place. That is less complicated than knowing how to
apt/dnf/yum install a package for all the random distros and repo location
combinations (one of the biggest benefits of containers IMO). But it
still requires some sort of process to keep ceph-daemon up to date.
Maybe if we always pass an md5sum to ceph-daemon whenever we invoke it to
assert that we are running the version we want, and if there is a
mismatch, ceph-daemon bails out with a special exit code that triggers and
update and retry?
Updating a system executable via its own - circumventing distro's
packaging policies isn't something that is going to be easy to sell.
Anyway, what are people's thoughts here? How much more complicated are we
interested or willing to make this to make people more comfortable with
the idea that ceph owns an ssh key?
Why would this need to be solved by ceph-daemon at all? In other
software (Vagrant for example) that has similar needs, an "insecure"
SSH key is provided so that
initial setup (or bootstrap) is done with ease. It is up to the
administrator to ensure that key is updated after bootstrapping or
create a separate SSH user with restrictions.
In lots of cases this step of securing the SSH key is not needed, like
testing or playing around with a throwaway cluster.
sage
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
6:29 p.m.
On Tue, 12 Nov 2019, Alfredo Deza wrote:
This would be "installed" at /home/$cephuser/ceph-daemon, so it isn't a
'system executable' (or even 'installed') in the conventional sense. I
don't think the system packaging policies are relevant.
Currently (and in this proposal) everything ceph-daemon does to the host
is done in the way local system configuration is normally layered on top
of the system by a human user (so as not to ruffle packager feathers).
Currently, the cluster bootstrap process generates its own ssh key, and
that's the one you use to authorize the cluster to manage each host.
It's clear from the authorized_keys file that the key is for a ceph
cluster with a particular fsid.
The goal of all of this is to minimize or eliminate administrator
decisions like which key to use, where to put it, and so on. We want
step-by-step instructions with a very small number of steps (i.e. <5) and
(ideally) no decisions about where and how to do things, both to make
things easy, and to minimize variation between systems.
In this case, the goal is a short bootstrap process at the command line
(currently ~3 steps), after which point users can switch to the dashboard
(or ceph CLI if they prefer) to do the rest. And, ideally, a user would
never have to go back to the command line if they don't want to. Getting
new hosts added to the cluster makes that hard, so we want to make the
process as simple as possible.
Right now the 'prep' sequence for a new host can be almost as simple as
running a command like
echo ...pub.key.here... | sudo tee -a /root/.ssh/authorized_keys
on the new node, and then telling the cluster the new host's hostname.
Once we start installing packages or relying on external orchestration
tools ("go log into your ansible master node, then update your ansible
inventory like so, then run this playbook like so, wait 5 minutes,
then..." etc) the process is no longer simple.
Right now we have an extremely minimal footprint on the managed host,
but the cost is a root-authorized key. I'm trying to figure out how to
reduce that exposure without (significantly) increasing the footprint
or complexity for the user...
sage
Or... we could
skip the package entirely and install the ceph-daemon
script in /home/ceph/ceph-daemon, and include an update function that
updates the script in place. That is less complicated than knowing how to
apt/dnf/yum install a package for all the random distros and repo location
combinations (one of the biggest benefits of containers IMO). But it
still requires some sort of process to keep ceph-daemon up to date.
Maybe if we always pass an md5sum to ceph-daemon whenever we invoke it to
assert that we are running the version we want, and if there is a
mismatch, ceph-daemon bails out with a special exit code that triggers and
update and retry?
Updating a system executable via its own - circumventing distro's
packaging policies isn't something that is going to be easy to sell. Anyway, what
are people's thoughts here? How much more complicated are we
interested or willing to make this to make people more comfortable with
the idea that ceph owns an ssh key?
Why would this need to be solved by ceph-daemon at all? In other
software (Vagrant for example) that has similar needs, an "insecure"
SSH key is provided so that
initial setup (or bootstrap) is done with ease. It is up to the
administrator to ensure that key is updated after bootstrapping or
create a separate SSH user with restrictions.
In lots of cases this step of securing the SSH key is not needed, like
testing or playing around with a throwaway cluster. 
6:50 p.m.
On 12.11.19 16:29, Sage Weil wrote:
Right now we have an extremely minimal footprint on
the managed host,
but the cost is a root-authorized key. I'm trying to figure out how to
reduce that exposure without (significantly) increasing the footprint
or complexity for the user...
I may be totally wrong here but in larger or even
hyper-converged
environments, isn't there always a "central" admin server? You have to
keep your servers up2date anyway and therefor you need any automated way
to update the servers. So I'm unsure if this is really a problem at that
point.
--
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg
GF:Geschäftsführer: Felix Imendörffer, (HRB 36809, AG München)
7:53 p.m.
Am 12.11.19 um 15:52 schrieb Sage Weil:
Right now the way ceph-daemon is used by the ssh
orchestrator is designed
to minimize the dependencies/setup complexity. The only requirements for
a host to be added to the cluster are
- python (2 or 3)
- systemd
- either podman or docker installed
- the ceph cluster's pub key in /root/.ssh/authorized_keys
No other software (including Ceph) needs to be installed. The mgr/ssh
module invokes ceph-daemon on the remote host by running /usr/bin/python
over ssh and piping the cluster's version of ceph-daemon to stdin.
The downside to this approach is that some users might not like the idea
of ceph having an ssh key with root access. For large clusters I'm not
sure how much this really matters--if you pwn ceph you can delete TB to PB
of data so do you really care if someone has root?--but for hyperconverged
cases this might be a problem.
having the possibility to call sudo ceph-daemon sounds like a good thing
to me!
One alternative might be to
- create a ceph user on the node, and put the cluster's key in
that user's authorized_keys
- install a package that includes ceph-daemon (/usr/bin/ceph-damaen)
- install an /etc/sudoers.d/ceph file that lets the ceph user
'sudo ceph-daemon ...'
Cons:
- This makes the bootstrap process slightly more complicated: (1) install
package, (2) create user, (3) install ssh key (vs just #3).
- The remote version of ceph-daemon can get out of sync with the
cluster.. either stale and missing some feature, or even too new and
not behaving the way the cluster expects.>
Pros:
- This limits the attack surface area (if someone manages to get the
cluster's ssh key) to the functions that ceph-daemon implements, vs
full root.
We could mitigate the 'keep ceph-daemon up to date' problem somewhat by
implementing a 'ceph-daemon update' function that will apt/dnf/yum install
ceph-daemon on the local host, so that the cluster could self-update the
remote host.
I guess there are better tools for that job for installing and updateing
software (Ansible, Salt, pip).
Or... we could skip the package entirely and install the ceph-daemon
script in /home/ceph/ceph-daemon, and include an update function that
updates the script in place. That is less complicated than knowing how to
apt/dnf/yum install a package for all the random distros and repo location
combinations (one of the biggest benefits of containers IMO). But it
still requires some sort of process to keep ceph-daemon up to date.
Maybe if we always pass an md5sum to ceph-daemon whenever we invoke it to
assert that we are running the version we want, and if there is a
mismatch, ceph-daemon bails out with a special exit code that triggers and
update and retry?
Sounds like re-implementing pip from scratch or so.
I'm +1 for adding some validation to make sure we're running with a
compatible version of ceph-daemon. Maybe defining an API version of
ceph-daemon.
But I guess we need to properly install ceph-daemon via RPM for
downstream (We need to install ceph-daemon for the bootstrap anyway).
For the community, I'd also be ok with something like pip install
--update or password-less ssh rook access.
Anyway, what are people's thoughts here? How much more complicated are we
interested or willing to make this to make people more comfortable with
the idea that ceph owns an ssh key?
Make it optional, by validating that we're running with a compatible
ceph-daemon version, if ceph-daemon exists on the host?
Sebastian
sage
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
--
SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
8:12 p.m.
On Tue, 12 Nov 2019, Sebastian Wagner wrote:
It sounds a bit like pip, but I don't think pip would actually work here.
The idea is to keep ceph-daemon in sync with the remote mgr/ssh module,
not to have the latest/greatest, or whatever broken thing pypi might have.
Nor do we want to add a new external web service dependency to the
process--right now we only depend on a container registry, and that may be
dockerhub by default, but it could be anything you specify in your --image
argument.
Am 12.11.19 um 15:52 schrieb Sage Weil:
Yeah... I don't like the idea of having to think about packaging tools,
repo locations, and the like.
Right now the way ceph-daemon is used by the ssh
orchestrator is designed
to minimize the dependencies/setup complexity. The only requirements for
a host to be added to the cluster are
- python (2 or 3)
- systemd
- either podman or docker installed
- the ceph cluster's pub key in /root/.ssh/authorized_keys
No other software (including Ceph) needs to be installed. The mgr/ssh
module invokes ceph-daemon on the remote host by running /usr/bin/python
over ssh and piping the cluster's version of ceph-daemon to stdin.
The downside to this approach is that some users might not like the idea
of ceph having an ssh key with root access. For large clusters I'm not
sure how much this really matters--if you pwn ceph you can delete TB to PB
of data so do you really care if someone has root?--but for hyperconverged
cases this might be a problem.
having the possibility to call sudo ceph-daemon sounds like a good thing
to me!
One alternative might be to
- create a ceph user on the node, and put the cluster's key in
that user's authorized_keys
- install a package that includes ceph-daemon (/usr/bin/ceph-damaen)
- install an /etc/sudoers.d/ceph file that lets the ceph user
'sudo ceph-daemon ...'
Cons:
- This makes the bootstrap process slightly more complicated: (1) install
package, (2) create user, (3) install ssh key (vs just #3).
- The remote version of ceph-daemon can get out of sync with the
cluster.. either stale and missing some feature, or even too new and
not behaving the way the cluster expects.>
Pros:
- This limits the attack surface area (if someone manages to get the
cluster's ssh key) to the functions that ceph-daemon implements, vs
full root.
We could mitigate the 'keep ceph-daemon up to date' problem somewhat by
implementing a 'ceph-daemon update' function that will apt/dnf/yum install
ceph-daemon on the local host, so that the cluster could self-update the
remote host.
I guess there are better tools for that job for installing and updateing
software (Ansible, Salt, pip). Or... we could
skip the package entirely and install the ceph-daemon
script in /home/ceph/ceph-daemon, and include an update function that
updates the script in place. That is less complicated than knowing how to
apt/dnf/yum install a package for all the random distros and repo location
combinations (one of the biggest benefits of containers IMO). But it
still requires some sort of process to keep ceph-daemon up to date.
Maybe if we always pass an md5sum to ceph-daemon whenever we invoke it to
assert that we are running the version we want, and if there is a
mismatch, ceph-daemon bails out with a special exit code that triggers and
update and retry?
Sounds like re-implementing pip from scratch or so. I'm +1 for adding some validation to make sure
we're running with a
compatible version of ceph-daemon. Maybe defining an API version of
ceph-daemon.
But I guess we need to properly install ceph-daemon via RPM for
downstream (We need to install ceph-daemon for the bootstrap anyway).
For the community, I'd also be ok with something like pip install
--update or password-less ssh rook access.
I have a feeling distros/products will have a packaged ceph-daemon
available for bootstrapping, and perhaps 'ceph-daemon shell'.
But IMO using a packaged version for mgr/ssh is a fragile/broken approach
because the ceph cluster will be upgrading itself, and part of that
upgrade will invitably involve updated versions of ceph-daemon. And we
don't want ceph to have to learn about package managers.
Again, the only reason we'd write ceph-daemon to disk on the remote node
*at all* is so that we can sudo run it.
...
The problem with this, now that I think about it, is that $cephuser can
sudo /home/$cephuser/ceph-daemon and also rewrite it, which makes it
useless as a security barrier. I think ceph-daemon has to be root-owned
and in a root-owned directory (/root/ceph-daemon?) in order for sudo to be
potentially useful here.
But then in order for the script to be updateable, mgr/ssh sshing in as
$cephuser needs to be able to update it. Which means that if someone has
that ssh key they can always update ceph-daemon to run /bin/bash (or
whatever) and circumvent any protection. So I think ability to have
mgr/ssh keep ceph-daemon in sync with itself is fundamentally incompatible
with providing any meaningful protection.
Sigh...
sage
8:46 p.m.
On Tue, Nov 12, 2019 at 9:12 AM Sage Weil <sweil(a)redhat.com> wrote:
I have a feeling distros/products will have a packaged
ceph-daemon
available for bootstrapping, and perhaps 'ceph-daemon shell'.
But IMO using a packaged version for mgr/ssh is a fragile/broken approach
because the ceph cluster will be upgrading itself, and part of that
upgrade will invitably involve updated versions of ceph-daemon. And we
don't want ceph to have to learn about package managers.
Again, the only reason we'd write ceph-daemon to disk on the remote node
*at all* is so that we can sudo run it.
...
The problem with this, now that I think about it, is that $cephuser can
sudo /home/$cephuser/ceph-daemon and also rewrite it, which makes it
useless as a security barrier. I think ceph-daemon has to be root-owned
and in a root-owned directory (/root/ceph-daemon?) in order for sudo to be
potentially useful here.
But then in order for the script to be updateable, mgr/ssh sshing in as
$cephuser needs to be able to update it. Which means that if someone has
that ssh key they can always update ceph-daemon to run /bin/bash (or
whatever) and circumvent any protection. So I think ability to have
mgr/ssh keep ceph-daemon in sync with itself is fundamentally incompatible
with providing any meaningful protection.
Sigh...
This is just a specific instance of a generalized problem, right? If
the Ceph cluster has the ability to update itself, then anybody who
pwns it can update to arbitrary code and do whatever they want. The
best we can do is narrow attack surfaces so that an attacker needs to
get more than a trivial level of access.
So it seems to me like the right way to handle this is by putting in
break points where the cluster admin can trade off convenience for
security as they see fit, but given your statements I guess we default
to convenience. One way might be to
1) always create both a ceph and ceph-admin user account on the remote
nodes when provisioning them, using an ssh key with root access. But
let the cluster admin do this step for us if they don't want to hand
out that root-accessible key.
2) grant all the managers easy access to the ceph account, but don't
give it write access to ceph-daemon.sh
3) use the ceph-admin user account solely to update ceph-daemon.sh,
and grant access in circumstances as limited as possible. We can
probably make it possible to require cluster admin intervention before
the ceph-admin key is unlocked, eg by requiring a passphrase that Ceph
doesn't store?
9:55 p.m.
On Tue, 12 Nov 2019, Gregory Farnum wrote:
This is just a specific instance of a generalized
problem, right? If
the Ceph cluster has the ability to update itself, then anybody who
pwns it can update to arbitrary code and do whatever they want. The
Yeah, exactly.
I think we just have two modes, then: (1) the root key one that's
implemented now for maximum ease of use, seamless upgrades, etc., and then
(2) a more paranoid mode where
1- admin is responsible for ceph-daemon being installed and/or upgraded
when necessary.
2- ceph-daemon package creates a cephdaemon user and sudoers.d file
3- mgr/ssh has a mode=... setting and/or user=... setting
4- node addition instructions have the paranoid edition where the ssh key
is put in cephdaemon user's (instead of root's) authorized_keys file
sage
8:50 p.m.
Am 12.11.19 um 18:12 schrieb Sage Weil:
On Tue, 12 Nov 2019, Sebastian Wagner wrote:
... snip ...
Again, the only reason we'd write ceph-daemon to
disk on the remote node
*at all* is so that we can sudo run it.
$ python -c 'import os;print(os.getuid())'
1000
$ sudo python -c 'import os;print(os.getuid())'
0
Technically, we don't need that just for calling sudo...
Sebastian
Sigh...
sage
--
SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
11:10 p.m.
For fine-grained authorization, Open Policy Agent (a CNCF incubating
project) easily integrates with PAM
(https://www.openpolicyagent.org/docs/latest/ssh-and-sudo-authorization/).
There's a dockerized demo ready to use at:
https://github.com/open-policy-agent/contrib/tree/master/pam_authz.
That one uses keyboard-interactive log-on, but it can be easily
switched to publickey.
For the authentication, if you don't want to mess with authorized_keys
and root, you can make sshd trust a CA instead
(https://smallstep.com/blog/use-ssh-certificates/). BTW, by using a CA
you can also avoid ssh's TOFU issue (host keys can be signed by the CA
too).
Kind regards,
Ernesto
On Tue, Nov 12, 2019 at 3:52 PM Sage Weil <sweil(a)redhat.com> wrote:
Right now the way ceph-daemon is used by the ssh orchestrator is designed
to minimize the dependencies/setup complexity. The only requirements for
a host to be added to the cluster are
- python (2 or 3)
- systemd
- either podman or docker installed
- the ceph cluster's pub key in /root/.ssh/authorized_keys
No other software (including Ceph) needs to be installed. The mgr/ssh
module invokes ceph-daemon on the remote host by running /usr/bin/python
over ssh and piping the cluster's version of ceph-daemon to stdin.
The downside to this approach is that some users might not like the idea
of ceph having an ssh key with root access. For large clusters I'm not
sure how much this really matters--if you pwn ceph you can delete TB to PB
of data so do you really care if someone has root?--but for hyperconverged
cases this might be a problem.
One alternative might be to
- create a ceph user on the node, and put the cluster's key in
that user's authorized_keys
- install a package that includes ceph-daemon (/usr/bin/ceph-damaen)
- install an /etc/sudoers.d/ceph file that lets the ceph user
'sudo ceph-daemon ...'
Cons:
- This makes the bootstrap process slightly more complicated: (1) install
package, (2) create user, (3) install ssh key (vs just #3).
- The remote version of ceph-daemon can get out of sync with the
cluster.. either stale and missing some feature, or even too new and
not behaving the way the cluster expects.
Pros:
- This limits the attack surface area (if someone manages to get the
cluster's ssh key) to the functions that ceph-daemon implements, vs
full root.
We could mitigate the 'keep ceph-daemon up to date' problem somewhat by
implementing a 'ceph-daemon update' function that will apt/dnf/yum install
ceph-daemon on the local host, so that the cluster could self-update the
remote host.
Or... we could skip the package entirely and install the ceph-daemon
script in /home/ceph/ceph-daemon, and include an update function that
updates the script in place. That is less complicated than knowing how to
apt/dnf/yum install a package for all the random distros and repo location
combinations (one of the biggest benefits of containers IMO). But it
still requires some sort of process to keep ceph-daemon up to date.
Maybe if we always pass an md5sum to ceph-daemon whenever we invoke it to
assert that we are running the version we want, and if there is a
mismatch, ceph-daemon bails out with a special exit code that triggers and
update and retry?
Anyway, what are people's thoughts here? How much more complicated are we
interested or willing to make this to make people more comfortable with
the idea that ceph owns an ssh key?
sage
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
1636
days inactive
1636
days old
9 comments
6 participants
participants (6)
-
Alfredo Deza -
Ernesto Puerta -
Gregory Farnum -
Kai Wagner -
Sage Weil -
Sebastian Wagner