25 Jul
2020
25 Jul
'20
12:58 p.m.
Greetings,
I have a question regarding the use of cephadm and disk partitions. I have noticed that
the cephadm documentation mentions that a device cannot have partitions to be considered
"available" for use. In my situation I don't want to use a device with
partitions, but rather a partition itself as an osd. I've noticed that partitions do
not show up when using `ceph orch device ls`. I've also noticed that partitions can
still be used as osds by running something like `ceph orch daemon add osd
node1:/dev/sda4`. My question is should I? Am I going to run into trouble by using a
partition for an osd instead of a full device?
Thanks,
Jason
28 Jul
28 Jul
9:03 p.m.
Jason,
Using partitions won't get you into trouble, but depending on the
version of Ceph you are using, you may want to leverage LVM instead of
partitions. For our Filestore cluster, we had two partitions on NVMe
to get more performance and it worked fine. I'm using LVM to carve out
NVMe drives for RocksDB and then an NVMe pool for CephFS metadata. The
problem with some of these configurations is that the tools don't set
them up well, so I did the partitioning/LVM setup with a script I
wrote, then just had Ceph use the final partition/LV.
Hope that helps,
Robert LeBlanc
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Sat, Jul 25, 2020 at 8:59 AM Jason Borden <jason(a)acedatacenter.com> wrote:
>
> Greetings,
>
> I have a question regarding the use of cephadm and disk partitions. I have noticed
that the cephadm documentation mentions that a device cannot have partitions to be
considered "available" for use. In my situation I don't want to use a device
with partitions, but rather a partition itself as an osd. I've noticed that partitions
do not show up when using `ceph orch device ls`. I've also noticed that partitions can
still be used as osds by running something like `ceph orch daemon add osd
node1:/dev/sda4`. My question is should I? Am I going to run into trouble by using a
partition for an osd instead of a full device?
>
> Thanks,
> Jason
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
29 Jul
29 Jul
1:25 a.m.
Hi Robert!
Thanks for answering my question. I take it you're working a lot with Ceph these days!
On my pre-octopus clusters I did use LVM backed by partitions, but I always kind of
wondered if it was a good practice or not as it added an additional layer and obscures the
underlying disk topology. Then on this new octopus cluster I wanted to use the new cephadm
approach for management and it seems to steer you away from using partitions or LVM
directly, thus my question. I don't really have the option to not use partitions in
this particular instance. I was merely curious if there was a particular reason that
cephadm doesn't consider partitions (or LVM) as being "available" devices.
All the storage in this cluster is the same so no need to split metadata on to faster
storage in my instance. Anyway, it's good to hear from you. Hope you and your family
are doing well.
Thanks,
Jason
2:40 p.m.
Jason,
The family and I are doing well, thanks for asking. I haven't worked
with Octopus yet, so I can't really talk towards that. Ceph
historically hasn't cared about physical disk layout, and personally I
think the Ceph code path is too heavy to really worry about
optimizations there. The LVM layer generally is pretty light and you
usually get more benefits than performance hit such relocating and
extending devices online. Sorry, I can't talk to how cephadm operates.
Robert LeBlanc
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Tue, Jul 28, 2020 at 9:25 PM Jason Borden <jason(a)acedatacenter.com> wrote:
>
> Hi Robert!
>
> Thanks for answering my question. I take it you're working a lot with Ceph these
days! On my pre-octopus clusters I did use LVM backed by partitions, but I always kind of
wondered if it was a good practice or not as it added an additional layer and obscures the
underlying disk topology. Then on this new octopus cluster I wanted to use the new cephadm
approach for management and it seems to steer you away from using partitions or LVM
directly, thus my question. I don't really have the option to not use partitions in
this particular instance. I was merely curious if there was a particular reason that
cephadm doesn't consider partitions (or LVM) as being "available" devices.
All the storage in this cluster is the same so no need to split metadata on to faster
storage in my instance. Anyway, it's good to hear from you. Hope you and your family
are doing well.
>
> Thanks,
> Jason
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:28 p.m.
cephadm will handle the LVM for you when you deploy using an OSD
specification. For example, we have NVME and rotational drives, and cephadm
will automatically deploy servers with the DB/WAL on NVME and the data on
the rotational drives, with a limit of 12 rotational per NVME - it handles
all the LVM magic as long as we feed it bare drives.
If you're happy with how it works, it makes management/expansion fairly
easy with a well-written OSD specification vs. doing all of it manually (I
had scripts I had written prior to bootstrap ceph node storage prior to
deployment).
On Tue, Jul 28, 2020 at 11:25 PM Jason Borden <jason(a)acedatacenter.com>
wrote:
Hi Robert!
Thanks for answering my question. I take it you're working a lot with Ceph
these days! On my pre-octopus clusters I did use LVM backed by partitions,
but I always kind of wondered if it was a good practice or not as it added
an additional layer and obscures the underlying disk topology. Then on this
new octopus cluster I wanted to use the new cephadm approach for management
and it seems to steer you away from using partitions or LVM directly, thus
my question. I don't really have the option to not use partitions in this
particular instance. I was merely curious if there was a particular reason
that cephadm doesn't consider partitions (or LVM) as being "available"
devices. All the storage in this cluster is the same so no need to split
metadata on to faster storage in my instance. Anyway, it's good to hear
from you. Hope you and your family are doing well.
Thanks,
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1372
days inactive
1376
days old
4 comments
3 participants
participants (3)
-
David Orman -
Jason Borden -
Robert LeBlanc