Hi, I've started working on a saner way to deploy OSD with Rook so that they don't use the rook binary image. Why were/are we using the rook binary to activate the OSD? A bit of background on containers first, when executing a container, we need to provide a command entrypoint that will act as PID 1. So if you want to do pre/post action before running the process you need to use a wrapper. In Rook, that's the rook binary, which has a CLI and can then "activate" an OSD. Currently, this "rook osd activate" call does the following: * sed the lvm.conf * run c-v lvm activate * run the osd process On shutdown, we intercept the signal, "kill -9" the osd and de-activate the LV. I have a patch here: https://github.com/rook/rook/pull/4386, that solves the initial bullet points but one thing we cannot do is the signal catching and the lv de-activation. Before you ask, Kubernetes has pre/post-hook but they are not reliable, it's known and documented that there is no guarantee they would actually run before or after the container starts/stops. We tried and we had issues. Why do we want to stop using the rook binary for activation? Because each time we get a new binary version (new operator version), this will restart all the OSDs, even if the deployment spec didn't change, at least if nothing else than the rook image version changed. Also with containers, we have seen so many issues working with LVM, just to name a few: * adapt lvm filters * interactions with udev - need to tune the lvm config, even c-v itself has lvm flag to not sync with udev built-in * several bindmounts * lvm package must be present on the host even if running in containers * SELinux, yes lvm calls SELinux commands under the hood and pollute the logs in some scenarios

Currently, one of the ways I can see this working is by not using LVM when bootstrapping OSDs. Unfortunately, some of the logic cannot go in the OSD code since the lv de-activation happens after the OSD stops. We need to de-activate the LV so when running in the Cloud the block can safely be re-attached to a new machine without LVM issues. I know this will be a bit challenging and might ultimately look like ceph-disk but it'd be nice to consider it. What about a small prototype for Bluestore with block/db/wal on the same disk? If this gets rejected, I might try a prototype for not using c-v in Rook or something else that might come up with this discussion.

Thanks! ––––––––– Sébastien Han Senior Principal Software Engineer, Storage Architect "Always give 100%. Unless you're giving blood." _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

Hi, I've started working on a saner way to deploy OSD with Rook so that they don't use the rook binary image. Why were/are we using the rook binary to activate the OSD? A bit of background on containers first, when executing a container, we need to provide a command entrypoint that will act as PID 1. So if you want to do pre/post action before running the process you need to use a wrapper. In Rook, that's the rook binary, which has a CLI and can then "activate" an OSD. Currently, this "rook osd activate" call does the following: * sed the lvm.conf * run c-v lvm activate * run the osd process On shutdown, we intercept the signal, "kill -9" the osd and de-activate the LV. I have a patch here: https://github.com/rook/rook/pull/4386, that solves the initial bullet points but one thing we cannot do is the signal catching and the lv de-activation. Before you ask, Kubernetes has pre/post-hook but they are not reliable, it's known and documented that there is no guarantee they would actually run before or after the container starts/stops. We tried and we had issues. Why do we want to stop using the rook binary for activation? Because each time we get a new binary version (new operator version), this will restart all the OSDs, even if the deployment spec didn't change, at least if nothing else than the rook image version changed. Also with containers, we have seen so many issues working with LVM, just to name a few: * adapt lvm filters * interactions with udev - need to tune the lvm config, even c-v itself has lvm flag to not sync with udev built-in * several bindmounts * lvm package must be present on the host even if running in containers * SELinux, yes lvm calls SELinux commands under the hood and pollute the logs in some scenarios Currently, one of the ways I can see this working is by not using LVM when bootstrapping OSDs. Unfortunately, some of the logic cannot go in the OSD code since the lv de-activation happens after the OSD stops. We need to de-activate the LV so when running in the Cloud the block can safely be re-attached to a new machine without LVM issues. I know this will be a bit challenging and might ultimately look like ceph-disk but it'd be nice to consider it. What about a small prototype for Bluestore with block/db/wal on the same disk?

If this gets rejected, I might try a prototype for not using c-v in Rook or something else that might come up with this discussion. Thanks! ––––––––– Sébastien Han Senior Principal Software Engineer, Storage Architect "Always give 100%. Unless you're giving blood." _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

Hi, I've started working on a saner way to deploy OSD with Rook so that they don't use the rook binary image. Why were/are we using the rook binary to activate the OSD? A bit of background on containers first, when executing a container, we need to provide a command entrypoint that will act as PID 1. So if you want to do pre/post action before running the process you need to use a wrapper. In Rook, that's the rook binary, which has a CLI and can then "activate" an OSD. Currently, this "rook osd activate" call does the following: * sed the lvm.conf * run c-v lvm activate * run the osd process On shutdown, we intercept the signal, "kill -9" the osd and de-activate the LV. I have a patch here: https://github.com/rook/rook/pull/4386, that solves the initial bullet points but one thing we cannot do is the signal catching and the lv de-activation.

Before you ask, Kubernetes has pre/post-hook but they are not reliable, it's known and documented that there is no guarantee they would actually run before or after the container starts/stops. We tried and we had issues. Why do we want to stop using the rook binary for activation? Because each time we get a new binary version (new operator version), this will restart all the OSDs, even if the deployment spec didn't change, at least if nothing else than the rook image version changed. Also with containers, we have seen so many issues working with LVM, just to name a few: * adapt lvm filters * interactions with udev - need to tune the lvm config, even c-v itself has lvm flag to not sync with udev built-in * several bindmounts * lvm package must be present on the host even if running in containers * SELinux, yes lvm calls SELinux commands under the hood and pollute the logs in some scenarios Currently, one of the ways I can see this working is by not using LVM when bootstrapping OSDs. Unfortunately, some of the logic cannot go in the OSD code since the lv de-activation happens after the OSD stops. We need to de-activate the LV so when running in the Cloud the block can safely be re-attached to a new machine without LVM issues. I know this will be a bit challenging and might ultimately look like ceph-disk but it'd be nice to consider it. What about a small prototype for Bluestore with block/db/wal on the same disk? If this gets rejected, I might try a prototype for not using c-v in Rook or something else that might come up with this discussion.

Just to reiterate, I'm currently only looking at the simplest scenario which is the most common one: a Bluestore non-encrypted OSD with block/db/wal on the same disk.

On 2019-12-03T19:58:55, Sage Weil <sage(a)newdream.net> wrote: I'm not sure. LVM provides many management functions - such as being able to transparently re-map/move data from one device to another, or increasing LV sizes, say - that otherwise would need to be reimplemented by the OSD processes. Yes, but given the current deployment ratios, I'd bet this would also not be applicable to the majority of deployments. Yes, it'd get the very very simple ones of the ground, but we'd still have to solve the actual problems. (Plus then maintain the "simple" way on top, and how to go from there to the more complex one, feature-disparities, DriveGroups for each, etc etc) -- SUSE Software Solutions Germany GmbH, MD: Felix Imendörffer, HRB 36809 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden) _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

On Thu, Dec 5, 2019 at 4:34 AM Sebastien Han &lt;shan(a)redhat.com&gt; wrote: I would still be interested to see how (why) partitions created by rook and then populated in /etc/ceph/osd/<id>.json does not work for the purposes you've stated. It doesn't tie you to LVM which seems problematic for you, and still allows you to do all the OSD variations including dmcrypt.

On 05.12.19 10:33, Sebastien Han wrote: Yes because that's basically what you're doing here.

Personally I don't see the value in just switching back again as I didn't found real good and absolutely necessary reasons to do so and why rook and/or ceph-volume couldn't be fixed in that regard. I also didn't find out why the new solution should be any better in the case that we don't talk about switching back in a few month again because all of those issues are fixed now - You said it yourself "ultimately fixed" which at the same time means that such tools need some time to reach the point were all those issues are fixed. We need to start to talk across boundaries and to reach out and involve other project maintainer to work on real solutions instead of moving out of their way.

Just imagine we would switch back to partitions, that would mean that we're doing again a breaking change between releases. This change would result in the same thing such like filestore->bluestore or ceph-disk->ceph-volume - people have to rewrite the whole data of their cluster at one point or another (or do we want to support both tools forever?).

I have the feeling that we sometimes treat this whole project still just like a kick-starter project were everything can be switched and changed between releases in any direction and whenever we think it would be nice to do so. Could we please start to think about that there's a big and luckily growing user base behind that are actively relaying on this solution and they're not keen on moving their data around just because we had a "feeling"?

No one should feel personally offended by the last remark but introducing such changes isn't something we could/should just do. We really should have 100% valid arguments why replacing one tool by another tool, which we had already in the past, now magically solves everything and we're sure that we can't fix such issues in the current solution.

Kai -- SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF:Geschäftsführer: Felix Imendörffer, (HRB 36809, AG Nürnberg)

ceph-volume is a sunk cost! And your argument basically falls into that paradigm, "oh we have invested so much already, that we cannot stop and we should continue even though this will only bring more trouble". Incapable of accepting this sunk cost. All the issues that have been fixed with a lot of pain. All that pain could have been avoided if LVM wasn't there and pursuing in that direction will only lead us to more pain again.

Also, I'm not saying we should replace the tool but allow not using LVM for a simple scenario to start with

Hi Sebastien and thanks for your feedback. On 06.12.19 10:00, Sebastien Han wrote: The reason I disagree here is the scenario were the WAL/DB is on a separate device and a single OSD crashes. In that case you would like to recreate just that single OSD instead of the whole group. Also if we deprecate a tool such like we did with ceph-disk, users have to migrate sooner or later if they don't want to do everything manually on the CLI (by that I mean via fdisk/pure lvm commands and so on). We could argue now that this can still be done on the command line manually but all our efforts are towards simplicity/automation and having everything in the Dashboard. If the underlying tool/functionality isn't there anymore, that isn't possible.

Which then leads me to, why couldn't such functionality be implemented into a single tool instead of having two at the end? So don't get me wrong, I'm not saying that I'm against everything I'm just saying that I think this is a topic that should be discussed in more depth.

As said, just my two cents here. Kai -- SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF:Geschäftsführer: Felix Imendörffer, (HRB 36809, AG Nürnberg)

My thoughts here are still pretty inconclusive... I agree that we should invest a non-LVM mode, but there isn't a way to do that currently that supports dm-crypt that isn't complicated and convoluted, so it cannot be a full replacement for the LVM mode. At the same time, Real Soon Now we're going to be building crimson OSDs backed by ZNS SSDs (and eventually persistent memory), which will also very clearly not be LVM-based. I'm a bit hesitant to introduce a bare-bones bluestore mode right now just because we'll be adding yet another variation soon, and it may be that we construct a general approach to both... but probably not. And the whole point of c-v's architecture was to be pluggable. So maybe a bare-bones bluestore mode makes sense. In the simple case, it really should be *very* simple. But its scope pretty quickly expodes: what about wal and db devices? We have labels for those, so we could support those, also easily... if the user has to partition the devices beforehand manually. They'll immediately want to use the new auto/batch thing, but that's tied to the LVM implementation. And what if one of the db/wal/main devices is an LV and another is not? We'd need to make sure the lvm mode machinery doesn't trigger unless all of its labels are there, but it might be confusing. All of which means that this is probably *only* useful for single-device OSDs. On the one hand, those are increasingly common (hello, all-SSD clusters), but on the other hand, for fast SSDs we may want to deploy N of them per device. Since we can't cover all of that, and at a minimum, we can't cover dm-crypt, Rook will need to behave with the lvm mode one way or another. So we need to have a wrapper (or something similar) no matter what. So I suggest we start there. sage On Fri, 6 Dec 2019, Sebastien Han wrote: > Hi Kai, > > Thanks! > ––––––––– > Sébastien Han > Senior Principal Software Engineer, Storage Architect > > "Always give 100%. Unless you're giving blood." > > On Fri, Dec 6, 2019 at 10:44 AM Kai Wagner &lt;kwagner(a)suse.com&gt; wrote: > > > > Hi Sebastien and thanks for your feedback. > > > > On 06.12.19 10:00, Sebastien Han wrote: > > > ceph-volume is a sunk cost! > > > And your argument basically falls into that paradigm, "oh we have > > > invested so much already, that we cannot stop and we should continue > > > even though this will only bring more trouble". Incapable of accepting > > > this sunk cost. > > > All the issues that have been fixed with a lot of pain. > > > All that pain could have been avoided if LVM wasn't there and pursuing > > > in that direction will only lead us to more pain again. > > > > The reason I disagree here is the scenario were the WAL/DB is on a > > separate device and a single OSD crashes. In that case you would like to > > recreate just that single OSD instead of the whole group. Also if we > > deprecate a tool such like we did with ceph-disk, users have to migrate > > sooner or later if they don't want to do everything manually on the CLI > > (by that I mean via fdisk/pure lvm commands and so on). > > > > We could argue now that this can still be done on the command line > > manually but all our efforts are towards simplicity/automation and > > having everything in the Dashboard. If the underlying tool/functionality > > isn't there anymore, that isn't possible. > > > > I understand your position, yes when we start separating block/db/wal > things get really complex that's why I'm sticking with block/db/wal in > the same block. > Also, we haven't seen any request for separating those when running > OSDs on PVC in the Cloud. So we would likely continue to do so for a > while. > > > > Also, I'm not saying we should replace the tool but allow not using > > > LVM for a simple scenario to start with > > > > Which then leads me to, why couldn't such functionality be implemented > > into a single tool instead of having two at the end? > > > > So don't get me wrong, I'm not saying that I'm against everything I'm > > just saying that I think this is a topic that should be discussed in > > more depth. > > Yes, that's for sure. > > > > > As said, just my two cents here. > > > > Kai > > > > -- > > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg > > GF:Geschäftsführer: Felix Imendörffer, (HRB 36809, AG Nürnberg) > > > > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

On Fri, Dec 6, 2019 at 8:31 AM Sage Weil &lt;sage(a)newdream.net&gt; wrote: The `ceph-volume simple` sub-command does allow dmcrypt. The key is stored in the JSON file in /etc/ceph/osd. Is there a scenario you've seen where this is not possible? The `simple` sub-command would even allow partitions (regardless of ceph-disk).

_______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

On Fri, 6 Dec 2019, Alfredo Deza wrote: For the dm-crypt case, I'm assuming we need the key to be attached to the device in some way.

LVM does this with another LV (IIRC); ceph-disk did it with a tiny partition. Putting it in /etc/ceph means you can't move a disk to another server without manually copying files arounds.

sage > > > > > At the same time, Real Soon Now we're going to be building crimson OSDs > > backed by ZNS SSDs (and eventually persistent memory), which will also > > very clearly not be LVM-based. I'm a bit hesitant to introduce a > > bare-bones bluestore mode right now just because we'll be adding yet > > another variation soon, and it may be that we construct a general approach > > to both... but probably not. And the whole point of c-v's architecture > > was to be pluggable. > > > > So maybe a bare-bones bluestore mode makes sense. In the simple case, it > > really should be *very* simple. But its scope pretty quickly expodes: > > what about wal and db devices? We have labels for those, so we could > > support those, also easily... if the user has to partition the devices > > beforehand manually. They'll immediately want to use the new > > auto/batch thing, but that's tied to the LVM implementation. And what > > if one of the db/wal/main devices is an LV and another is not? We'd > > need to make sure the lvm mode machinery doesn't trigger unless all of > > its labels are there, but it might be confusing. All of which means that > > this is probably *only* useful for single-device OSDs. On the one hand, > > those are increasingly common (hello, all-SSD clusters), but on the other > > hand, for fast SSDs we may want to deploy N of them per device. > > > > Since we can't cover all of that, and at a minimum, we can't cover > > dm-crypt, Rook will need to behave with the lvm mode one way or another. > > So we need to have a wrapper (or something similar) no matter what. So I > > suggest we start there. > > > > sage > > > > > > On Fri, 6 Dec 2019, Sebastien Han wrote: > > > > > Hi Kai, > > > > > > Thanks! > > > ––––––––– > > > Sébastien Han > > > Senior Principal Software Engineer, Storage Architect > > > > > > "Always give 100%. Unless you're giving blood." > > > > > > On Fri, Dec 6, 2019 at 10:44 AM Kai Wagner &lt;kwagner(a)suse.com&gt; wrote: > > > > > > > > Hi Sebastien and thanks for your feedback. > > > > > > > > On 06.12.19 10:00, Sebastien Han wrote: > > > > > ceph-volume is a sunk cost! > > > > > And your argument basically falls into that paradigm, "oh we have > > > > > invested so much already, that we cannot stop and we should continue > > > > > even though this will only bring more trouble". Incapable of accepting > > > > > this sunk cost. > > > > > All the issues that have been fixed with a lot of pain. > > > > > All that pain could have been avoided if LVM wasn't there and pursuing > > > > > in that direction will only lead us to more pain again. > > > > > > > > The reason I disagree here is the scenario were the WAL/DB is on a > > > > separate device and a single OSD crashes. In that case you would like to > > > > recreate just that single OSD instead of the whole group. Also if we > > > > deprecate a tool such like we did with ceph-disk, users have to migrate > > > > sooner or later if they don't want to do everything manually on the CLI > > > > (by that I mean via fdisk/pure lvm commands and so on). > > > > > > > > We could argue now that this can still be done on the command line > > > > manually but all our efforts are towards simplicity/automation and > > > > having everything in the Dashboard. If the underlying tool/functionality > > > > isn't there anymore, that isn't possible. > > > > > > > > > > I understand your position, yes when we start separating block/db/wal > > > things get really complex that's why I'm sticking with block/db/wal in > > > the same block. > > > Also, we haven't seen any request for separating those when running > > > OSDs on PVC in the Cloud. So we would likely continue to do so for a > > > while. > > > > > > > > Also, I'm not saying we should replace the tool but allow not using > > > > > LVM for a simple scenario to start with > > > > > > > > Which then leads me to, why couldn't such functionality be implemented > > > > into a single tool instead of having two at the end? > > > > > > > > So don't get me wrong, I'm not saying that I'm against everything I'm > > > > just saying that I think this is a topic that should be discussed in > > > > more depth. > > > > > > Yes, that's for sure. > > > > > > > > > > > As said, just my two cents here. > > > > > > > > Kai > > > > > > > > -- > > > > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg > > > > GF:Geschäftsführer: Felix Imendörffer, (HRB 36809, AG Nürnberg) > > > > > > > > > > > _______________________________________________ > > > Dev mailing list -- dev(a)ceph.io > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > _______________________________________________ > > Dev mailing list -- dev(a)ceph.io > > To unsubscribe send an email to dev-leave(a)ceph.io > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

My thoughts here are still pretty inconclusive... I agree that we should invest a non-LVM mode, but there isn't a way to do that currently that supports dm-crypt that isn't complicated and convoluted, so it cannot be a full replacement for the LVM mode. At the same time, Real Soon Now we're going to be building crimson OSDs backed by ZNS SSDs (and eventually persistent memory), which will also very clearly not be LVM-based. I'm a bit hesitant to introduce a bare-bones bluestore mode right now just because we'll be adding yet another variation soon, and it may be that we construct a general approach to both... but probably not. And the whole point of c-v's architecture was to be pluggable. So maybe a bare-bones bluestore mode makes sense. In the simple case, it really should be *very* simple. But its scope pretty quickly expodes: what about wal and db devices? We have labels for those, so we could support those, also easily... if the user has to partition the devices beforehand manually. They'll immediately want to use the new auto/batch thing, but that's tied to the LVM implementation. And what if one of the db/wal/main devices is an LV and another is not? We'd need to make sure the lvm mode machinery doesn't trigger unless all of its labels are there, but it might be confusing. All of which means that this is probably *only* useful for single-device OSDs. On the one hand, those are increasingly common (hello, all-SSD clusters), but on the other hand, for fast SSDs we may want to deploy N of them per device.

Since we can't cover all of that, and at a minimum, we can't cover dm-crypt, Rook will need to behave with the lvm mode one way or another. So we need to have a wrapper (or something similar) no matter what. So I suggest we start there.

sage On Fri, 6 Dec 2019, Sebastien Han wrote: > Hi Kai, > > Thanks! > ––––––––– > Sébastien Han > Senior Principal Software Engineer, Storage Architect > > "Always give 100%. Unless you're giving blood." > > On Fri, Dec 6, 2019 at 10:44 AM Kai Wagner &lt;kwagner(a)suse.com&gt; wrote: > > > > Hi Sebastien and thanks for your feedback. > > > > On 06.12.19 10:00, Sebastien Han wrote: > > > ceph-volume is a sunk cost! > > > And your argument basically falls into that paradigm, "oh we have > > > invested so much already, that we cannot stop and we should continue > > > even though this will only bring more trouble". Incapable of accepting > > > this sunk cost. > > > All the issues that have been fixed with a lot of pain. > > > All that pain could have been avoided if LVM wasn't there and pursuing > > > in that direction will only lead us to more pain again. > > > > The reason I disagree here is the scenario were the WAL/DB is on a > > separate device and a single OSD crashes. In that case you would like to > > recreate just that single OSD instead of the whole group. Also if we > > deprecate a tool such like we did with ceph-disk, users have to migrate > > sooner or later if they don't want to do everything manually on the CLI > > (by that I mean via fdisk/pure lvm commands and so on). > > > > We could argue now that this can still be done on the command line > > manually but all our efforts are towards simplicity/automation and > > having everything in the Dashboard. If the underlying tool/functionality > > isn't there anymore, that isn't possible. > > > > I understand your position, yes when we start separating block/db/wal > things get really complex that's why I'm sticking with block/db/wal in > the same block. > Also, we haven't seen any request for separating those when running > OSDs on PVC in the Cloud. So we would likely continue to do so for a > while. > > > > Also, I'm not saying we should replace the tool but allow not using > > > LVM for a simple scenario to start with > > > > Which then leads me to, why couldn't such functionality be implemented > > into a single tool instead of having two at the end? > > > > So don't get me wrong, I'm not saying that I'm against everything I'm > > just saying that I think this is a topic that should be discussed in > > more depth. > > Yes, that's for sure. > > > > > As said, just my two cents here. > > > > Kai > > > > -- > > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg > > GF:Geschäftsführer: Felix Imendörffer, (HRB 36809, AG Nürnberg) > > > > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

_______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

My two cents, likely controvesial but it has to be said: [...]