Can ceph-volume manage the LVs optionally used for DB / WAL at all?
Hey ceph-users, I was wondering if ceph-volume did anything in regards to the management (creation, setting metadata, ....) of LVs which are used for DB / WAL of an OSD? Reading the documentation at https://docs.ceph.com/en/latest/man/8/ceph-volume/#new-db is seems to indicate that the LV to be used as e.g. DB needs to be created manually (without ceph-volume) and exist prior to using ceph-volume to move the DB to that LV? I suppose the same is true for "ceph-volume lvm create" or "ceph-volume lvm prepare" and "--block.db" It's not that creating a few LVs is hard... it's just that ceph volume does apply some structure to the naming of LVM VGs and LVs on the OSD device and also adds metadata. That would then be up to the user, right? Regards Christian
Hi, if you deploy OSDs from scratch you don't have to create LVs manually, that is handled entirely by ceph-volume (for example on cephadm based clusters you only provide a drivegroup definition). I'm not sure if automating db/wal migration has been considered, it might be (too) difficult. But moving the db/wal devices to new/different devices doesn't seem to be a reoccuring issue (corner case?), so maybe having control over that process for each OSD individually is the safe(r) option in case something goes wrong. Regards, Eugen Zitat von Christian Rohmann <christian.rohmann@inovex.de>:
Hey ceph-users,
I was wondering if ceph-volume did anything in regards to the management (creation, setting metadata, ....) of LVs which are used for DB / WAL of an OSD?
Reading the documentation at https://docs.ceph.com/en/latest/man/8/ceph-volume/#new-db is seems to indicate that the LV to be used as e.g. DB needs to be created manually (without ceph-volume) and exist prior to using ceph-volume to move the DB to that LV? I suppose the same is true for "ceph-volume lvm create" or "ceph-volume lvm prepare" and "--block.db"
It's not that creating a few LVs is hard... it's just that ceph volume does apply some structure to the naming of LVM VGs and LVs on the OSD device and also adds metadata. That would then be up to the user, right?
Regards
Christian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 11.08.23 16:06, Eugen Block wrote:
if you deploy OSDs from scratch you don't have to create LVs manually, that is handled entirely by ceph-volume (for example on cephadm based clusters you only provide a drivegroup definition).
By looking at https://docs.ceph.com/en/latest/man/8/ceph-volume/#cmdoption-ceph-volume-lvm... it seems that ceph-volume wants an LV or partition. So it's apparently not just taking a VG itself? Also if there were multiple VGs / devices , I likely would need to at least pick those. But I suppose this orchestration would then require cephadm (https://docs.ceph.com/en/latest/cephadm/services/osd/#drivegroups) and cannot be done via ceph-volume which merely takes care of ONE OSD at a time.
I'm not sure if automating db/wal migration has been considered, it might be (too) difficult. But moving the db/wal devices to new/different devices doesn't seem to be a reoccuring issue (corner case?), so maybe having control over that process for each OSD individually is the safe(r) option in case something goes wrong.
Sorry for the confusion. I was not talking about any migrations, just the initial creation of spinning rust OSDs with DB or WAL on fast storage. Regards Christian
Hi, I'm still not sure if we're on the same page.
By looking at https://docs.ceph.com/en/latest/man/8/ceph-volume/#cmdoption-ceph-volume-lvm... it seems that ceph-volume wants an LV or partition. So it's apparently not just taking a VG itself? Also if there were multiple VGs / devices , I likely would need to at least pick those.
ceph-volume creates all required VGs/LVs automatically, and the OSD creation happens in batch mode, for example when run by cephadm: ceph-volume lvm batch --yes /dev/sdb /dev/sdc /dev/sdd In a non-cephadm deployment you can fiddle with ceph-volume manually, where you also can deploy single OSDs, with or without providing your own pre-built VGs/LVs. In a cephadm deployment manually creating OSDs will result in "stray daemons not managed by cephadm" warnings. Before we upgraded to Pacific we did manage our block.db devices manually with pre-built LVs, e.g.: $ lvcreate -L 30G -n bluefsdb-30 ceph-journals $ ceph-volume lvm create --data /dev/sdh --block.db ceph-journals/bluefsdb-30
Sorry for the confusion. I was not talking about any migrations, just the initial creation of spinning rust OSDs with DB or WAL on fast storage.
So the question is, is your cluster (or multiple clusters) managed cephadm? If so, you don't need to worry about ceph-volume, it will be handled for you in batch mode (you can inspect the ceph-volume.log afterwards). You just need to provide a yaml file that fits your needs with regards to block.db and data devices. Zitat von Christian Rohmann <christian.rohmann@inovex.de>:
On 11.08.23 16:06, Eugen Block wrote:
if you deploy OSDs from scratch you don't have to create LVs manually, that is handled entirely by ceph-volume (for example on cephadm based clusters you only provide a drivegroup definition).
By looking at https://docs.ceph.com/en/latest/man/8/ceph-volume/#cmdoption-ceph-volume-lvm... it seems that ceph-volume wants an LV or partition. So it's apparently not just taking a VG itself? Also if there were multiple VGs / devices , I likely would need to at least pick those.
But I suppose this orchestration would then require cephadm (https://docs.ceph.com/en/latest/cephadm/services/osd/#drivegroups) and cannot be done via ceph-volume which merely takes care of ONE OSD at a time.
I'm not sure if automating db/wal migration has been considered, it might be (too) difficult. But moving the db/wal devices to new/different devices doesn't seem to be a reoccuring issue (corner case?), so maybe having control over that process for each OSD individually is the safe(r) option in case something goes wrong.
Sorry for the confusion. I was not talking about any migrations, just the initial creation of spinning rust OSDs with DB or WAL on fast storage.
Regards
Christian
On 25.08.23 09:09, Eugen Block wrote:
I'm still not sure if we're on the same page.
Maybe not, I'll respond inline to clarify.
By looking at https://docs.ceph.com/en/latest/man/8/ceph-volume/#cmdoption-ceph-volume-lvm... it seems that ceph-volume wants an LV or partition. So it's apparently not just taking a VG itself? Also if there were multiple VGs / devices , I likely would need to at least pick those.
ceph-volume creates all required VGs/LVs automatically, and the OSD creation happens in batch mode, for example when run by cephadm: ceph-volume lvm batch --yes /dev/sdb /dev/sdc /dev/sdd
In a non-cephadm deployment you can fiddle with ceph-volume manually, where you also can deploy single OSDs, with or without providing your own pre-built VGs/LVs. In a cephadm deployment manually creating OSDs will result in "stray daemons not managed by cephadm" warnings.
1) I am mostly asking about an non-cephadm environment and would just like to know if ceph-volume can also manage the VG of a DB/WAL device that is used for multiple OSD and create the individual LVs which are used for DB or WAL devices when creating a single OSD. Below you give an example "before we upgraded to Pacific" in which you run lvcreate manually. Is that not required anymore with >= Quincy? 2) Even with cephadm there is the "db_devices" as part of the drivegroups. But the question remains if cephadm can use a single db_device for multiple OSDs.
Before we upgraded to Pacific we did manage our block.db devices manually with pre-built LVs, e.g.:
$ lvcreate -L 30G -n bluefsdb-30 ceph-journals $ ceph-volume lvm create --data /dev/sdh --block.db ceph-journals/bluefsdb-30
As asked and explained in the paragraph above, this is what I am currently doing (lvcreate + ceph-volume lvm create). My question therefore is, if ceph-volume (!) could somehow create this LV for the DB automagically if I'd just give it a device (or existing VG)? Thank you very much for your patience in clarifying and responding to my questions. Regards Christian
Hi,
1) I am mostly asking about an non-cephadm environment and would just like to know if ceph-volume can also manage the VG of a DB/WAL device that is used for multiple OSD and create the individual LVs which are used for DB or WAL devices when creating a single OSD. Below you give an example "before we upgraded to Pacific" in which you run lvcreate manually. Is that not required anymore with >= Quincy?
yes, ceph-volume handles the creation for you in both cases, multiple OSDs as well as single OSDs, there are plenty of options available, here's another example (Nautilus): ---snip--- test-node:~ # ceph-volume lvm batch /dev/sdb /dev/sdc --db-devices /dev/sdd --> passed data devices: 2 physical, 0 LVM --> relative data size: 1.0 --> passed block_db devices: 1 physical, 0 LVM Total OSDs: 2 Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- data /dev/sdb 5.00 GB 100.00% block_db /dev/sdd 2.50 GB 50.00% ---------------------------------------------------------------------------------------------------- data /dev/sdc 5.00 GB 100.00% block_db /dev/sdd 2.50 GB 50.00% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) ---snip--- Note that it calculated the db size automatically (I didn't specify a db size, which I could have), on this node I have three 10GB disks. You can create your individual layout and ceph-volume asks before deploying (also when using the --report argument). And this has nothing to do with Quincy, this worked that way since Luminous when ceph-volume was introduced, IIRC, and the old ceph-disk utility was deprecated with Mimic (https://docs.ceph.com/en/mimic/ceph-volume/). You *can* but you *don't have to* create VGs/LVs before deploying. Manually creating them was often used when operators didn't want to consume an entire SSD just for ceph-volume, or for testing purposes, whatever.
2) Even with cephadm there is the "db_devices" as part of the drivegroups. But the question remains if cephadm can use a single db_device for multiple OSDs.
Yes, it can. If your server only has one SSD and your drivegroup.yaml reflects that, cephadm will use only one SSD for multiple HDDs. You can --dry-run a drivegroup.yaml (ceph orch apply -i drivegroup.yaml --dry-run) to see what cephadm would do with your specification.
As asked and explained in the paragraph above, this is what I am currently doing (lvcreate + ceph-volume lvm create). My question therefore is, if ceph-volume (!) could somehow create this LV for the DB automagically if I'd just give it a device (or existing VG)?
Yes, as explained before, that's what it's developed for. ;-) If for some reason you are required to use only parts of an SSD it makes sense to create LVs manually, but it can be handled for you. Does that clear things up a bit? Are we on the same page now? :-) Regards, Eugen Zitat von Christian Rohmann <christian.rohmann@inovex.de>:
On 25.08.23 09:09, Eugen Block wrote:
I'm still not sure if we're on the same page.
Maybe not, I'll respond inline to clarify.
By looking at https://docs.ceph.com/en/latest/man/8/ceph-volume/#cmdoption-ceph-volume-lvm... it seems that ceph-volume wants an LV or partition. So it's apparently not just taking a VG itself? Also if there were multiple VGs / devices , I likely would need to at least pick those.
ceph-volume creates all required VGs/LVs automatically, and the OSD creation happens in batch mode, for example when run by cephadm: ceph-volume lvm batch --yes /dev/sdb /dev/sdc /dev/sdd
In a non-cephadm deployment you can fiddle with ceph-volume manually, where you also can deploy single OSDs, with or without providing your own pre-built VGs/LVs. In a cephadm deployment manually creating OSDs will result in "stray daemons not managed by cephadm" warnings.
1) I am mostly asking about an non-cephadm environment and would just like to know if ceph-volume can also manage the VG of a DB/WAL device that is used for multiple OSD and create the individual LVs which are used for DB or WAL devices when creating a single OSD. Below you give an example "before we upgraded to Pacific" in which you run lvcreate manually. Is that not required anymore with >= Quincy? 2) Even with cephadm there is the "db_devices" as part of the drivegroups. But the question remains if cephadm can use a single db_device for multiple OSDs.
Before we upgraded to Pacific we did manage our block.db devices manually with pre-built LVs, e.g.:
$ lvcreate -L 30G -n bluefsdb-30 ceph-journals $ ceph-volume lvm create --data /dev/sdh --block.db ceph-journals/bluefsdb-30
As asked and explained in the paragraph above, this is what I am currently doing (lvcreate + ceph-volume lvm create). My question therefore is, if ceph-volume (!) could somehow create this LV for the DB automagically if I'd just give it a device (or existing VG)?
Thank you very much for your patience in clarifying and responding to my questions. Regards
Christian
participants (2)
-
Christian Rohmann
-
Eugen Block