The key in /etc/ceph also gets lost if you repave the OS. I discovered the hard way when I had to repave a node and the OSDs wouldn’t start. They had to be redeployed, which was a whole lot of data movement that shouldn’t have needed to happen. I subsequently found another 150 or so that I redeployed to avoid another rude awakening. The current (well as of Luminous) scheme is to store them on the mons, is that somehow no longer feasible?
On Fri, 6 Dec 2019, Alfredo Deza wrote:
On Fri, Dec 6, 2019 at 8:31 AM Sage Weil <sage@newdream.net> wrote:
My thoughts here are still pretty inconclusive...
I agree that we should invest a non-LVM mode, but there isn't a way to do that currently that supports dm-crypt that isn't complicated and convoluted, so it cannot be a full replacement for the LVM mode.
The `ceph-volume simple` sub-command does allow dmcrypt. The key is stored in the JSON file in /etc/ceph/osd.
Is there a scenario you've seen where this is not possible? The `simple` sub-command would even allow partitions (regardless of ceph-disk).
For the dm-crypt case, I'm assuming we need the key to be attached to the device in some way. LVM does this with another LV (IIRC); ceph-disk did it with a tiny partition. Putting it in /etc/ceph means you can't move a disk to another server without manually copying files arounds.
sage