On Fri, 6 Dec 2019, Anthony D'Atri wrote:
The key in /etc/ceph also gets lost if you repave the OS. I discovered the hard way when I had to repave a node and the OSDs wouldn’t start. They had to be redeployed, which was a whole lot of data movement that shouldn’t have needed to happen. I subsequently found another 150 or so that I redeployed to avoid another rude awakening.
The current (well as of Luminous) scheme is to store them on the mons, is that somehow no longer feasible?
Yeah, that's the scheme I'm talking about. There is metadata and a key-fetching-key on the device itself, though, that is used to get the actual dm-crypt decryption key from the mon. That needs to go somewhere (another partition, a small LV, LVM metadata, etc.). The c-v simple mode is more like the pre-luminous scheme, so not really useful for dm-crypt. sage
On Fri, 6 Dec 2019, Alfredo Deza wrote:
On Fri, Dec 6, 2019 at 8:31 AM Sage Weil <sage@newdream.net> wrote:
My thoughts here are still pretty inconclusive...
I agree that we should invest a non-LVM mode, but there isn't a way to do that currently that supports dm-crypt that isn't complicated and convoluted, so it cannot be a full replacement for the LVM mode.
The `ceph-volume simple` sub-command does allow dmcrypt. The key is stored in the JSON file in /etc/ceph/osd.
Is there a scenario you've seen where this is not possible? The `simple` sub-command would even allow partitions (regardless of ceph-disk).
For the dm-crypt case, I'm assuming we need the key to be attached to the device in some way. LVM does this with another LV (IIRC); ceph-disk did it with a tiny partition. Putting it in /etc/ceph means you can't move a disk to another server without manually copying files arounds.
sage
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io