cephadm and disk partitions
Greetings, I have a question regarding the use of cephadm and disk partitions. I have noticed that the cephadm documentation mentions that a device cannot have partitions to be considered "available" for use. In my situation I don't want to use a device with partitions, but rather a partition itself as an osd. I've noticed that partitions do not show up when using `ceph orch device ls`. I've also noticed that partitions can still be used as osds by running something like `ceph orch daemon add osd node1:/dev/sda4`. My question is should I? Am I going to run into trouble by using a partition for an osd instead of a full device? Thanks, Jason
Jason, Using partitions won't get you into trouble, but depending on the version of Ceph you are using, you may want to leverage LVM instead of partitions. For our Filestore cluster, we had two partitions on NVMe to get more performance and it worked fine. I'm using LVM to carve out NVMe drives for RocksDB and then an NVMe pool for CephFS metadata. The problem with some of these configurations is that the tools don't set them up well, so I did the partitioning/LVM setup with a script I wrote, then just had Ceph use the final partition/LV. Hope that helps, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Sat, Jul 25, 2020 at 8:59 AM Jason Borden <jason@acedatacenter.com> wrote:
Greetings,
I have a question regarding the use of cephadm and disk partitions. I have noticed that the cephadm documentation mentions that a device cannot have partitions to be considered "available" for use. In my situation I don't want to use a device with partitions, but rather a partition itself as an osd. I've noticed that partitions do not show up when using `ceph orch device ls`. I've also noticed that partitions can still be used as osds by running something like `ceph orch daemon add osd node1:/dev/sda4`. My question is should I? Am I going to run into trouble by using a partition for an osd instead of a full device?
Thanks, Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Robert! Thanks for answering my question. I take it you're working a lot with Ceph these days! On my pre-octopus clusters I did use LVM backed by partitions, but I always kind of wondered if it was a good practice or not as it added an additional layer and obscures the underlying disk topology. Then on this new octopus cluster I wanted to use the new cephadm approach for management and it seems to steer you away from using partitions or LVM directly, thus my question. I don't really have the option to not use partitions in this particular instance. I was merely curious if there was a particular reason that cephadm doesn't consider partitions (or LVM) as being "available" devices. All the storage in this cluster is the same so no need to split metadata on to faster storage in my instance. Anyway, it's good to hear from you. Hope you and your family are doing well. Thanks, Jason
Jason, The family and I are doing well, thanks for asking. I haven't worked with Octopus yet, so I can't really talk towards that. Ceph historically hasn't cared about physical disk layout, and personally I think the Ceph code path is too heavy to really worry about optimizations there. The LVM layer generally is pretty light and you usually get more benefits than performance hit such relocating and extending devices online. Sorry, I can't talk to how cephadm operates. Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Jul 28, 2020 at 9:25 PM Jason Borden <jason@acedatacenter.com> wrote:
Hi Robert!
Thanks for answering my question. I take it you're working a lot with Ceph these days! On my pre-octopus clusters I did use LVM backed by partitions, but I always kind of wondered if it was a good practice or not as it added an additional layer and obscures the underlying disk topology. Then on this new octopus cluster I wanted to use the new cephadm approach for management and it seems to steer you away from using partitions or LVM directly, thus my question. I don't really have the option to not use partitions in this particular instance. I was merely curious if there was a particular reason that cephadm doesn't consider partitions (or LVM) as being "available" devices. All the storage in this cluster is the same so no need to split metadata on to faster storage in my instance. Anyway, it's good to hear from you. Hope you and your family are doing well.
Thanks, Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
cephadm will handle the LVM for you when you deploy using an OSD specification. For example, we have NVME and rotational drives, and cephadm will automatically deploy servers with the DB/WAL on NVME and the data on the rotational drives, with a limit of 12 rotational per NVME - it handles all the LVM magic as long as we feed it bare drives. If you're happy with how it works, it makes management/expansion fairly easy with a well-written OSD specification vs. doing all of it manually (I had scripts I had written prior to bootstrap ceph node storage prior to deployment). On Tue, Jul 28, 2020 at 11:25 PM Jason Borden <jason@acedatacenter.com> wrote:
Hi Robert!
Thanks for answering my question. I take it you're working a lot with Ceph these days! On my pre-octopus clusters I did use LVM backed by partitions, but I always kind of wondered if it was a good practice or not as it added an additional layer and obscures the underlying disk topology. Then on this new octopus cluster I wanted to use the new cephadm approach for management and it seems to steer you away from using partitions or LVM directly, thus my question. I don't really have the option to not use partitions in this particular instance. I was merely curious if there was a particular reason that cephadm doesn't consider partitions (or LVM) as being "available" devices. All the storage in this cluster is the same so no need to split metadata on to faster storage in my instance. Anyway, it's good to hear from you. Hope you and your family are doing well.
Thanks, Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
David Orman
-
Jason Borden
-
Robert LeBlanc