The key in /etc/ceph also gets lost if you repave the OS. I discovered the hard way when
I had to repave a node and the OSDs wouldn’t start. They had to be redeployed, which was
a whole lot of data movement that shouldn’t have needed to happen. I subsequently found
another 150 or so that I redeployed to avoid another rude awakening.
The current (well as of Luminous) scheme is to store them on the mons, is that somehow no
longer feasible?
On Fri, 6 Dec 2019, Alfredo Deza wrote:
On Fri, Dec 6, 2019 at 8:31 AM Sage Weil
<sage(a)newdream.net> wrote:
My thoughts here are still pretty inconclusive...
I agree that we should invest a non-LVM mode, but there isn't a way to do
that currently that supports dm-crypt that isn't complicated and
convoluted, so it cannot be a full replacement for the LVM mode.
The `ceph-volume simple` sub-command does allow dmcrypt. The key is
stored in the JSON file in /etc/ceph/osd.
Is there a scenario you've seen where this is not possible? The
`simple` sub-command would even allow partitions (regardless of
ceph-disk).
For the dm-crypt case, I'm assuming we need the key to be attached to the
device in some way. LVM does this with another LV (IIRC); ceph-disk did
it with a tiny partition. Putting it in /etc/ceph means you can't move a
disk to another server without manually copying files arounds.
sage