30 Mar
2021
30 Mar
'21
4:02 p.m.
Hello,
I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and
have a few questions mostly regarding planning of the infra.
1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to
use Ubuntu 20.04 instead or should I stick with 18.04?
2) The documentation recommends using Cephadm for new deployments, so I will use that but
I read that with Cephadm everything is running in containers, so is this the new way to
go? Or is Ceph in containers kind of still experimental?
3) As I will be needing cephfs I will also need MDS servers so with a total of 6 nodes I
am planning the following layout:
Node 1: MGR+MON+MDS
Node 2: MGR+MON+MDS
Node 3: MGR+MON+MDS
Node 4: OSD
Node 5: OSD
Node 6: OSD
Does this make sense? I am mostly interested in stability and HA with this setup.
4) Is there any special kind of demand in terms of disks on the MGR+MON+MDS nodes? Or can
I use have my OS disks on these nodes? As far as I understand the MDS will create a
metadata pool on the OSDs.
Thanks for the hints.
Best,
Mabi
30 Mar
30 Mar
7:19 p.m.
Mabi;
We're running Nautilus, and I am not wholly convinced of the "everything in
containers" view of the world, so take this with a small grain of salt...
1) We don't run Ubuntu, sorry. I suspect the documentation highlights 18.04 because
it's the current LTS release. Personally, if I had a preference of 20.04 over 18.04,
I would attempt to build a cluster on 20.04, and see how it goes. You might also look at
this: https://www.server-world.info/en/note?os=Ubuntu_20.04&p=ceph15&f=1
2) Containers are the preferred way of doing things in Octopus, so yes it's considered
stable.
3) Our first evaluation cluster was 3 Intel Atom C3000 nodes, with each node running all
the daemons (MON, MGR, MDS, 2 x OSD). Worked fine, and allowed me to demonstrate the
concepts in a size I could carry around.
4) Yes, and No... When the Cluster is happy, everything is generally happy. In certain
Warning and Error situations, MONs can chew through the HD space fairly quickly. I'm
not familiar with the HD usage of the daemons.
Thank you,
Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: mabi [mailto:mabi@protonmail.ch]
Sent: Tuesday, March 30, 2021 12:03 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] First 6 nodes cluster with Octopus
Hello,
I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and
have a few questions mostly regarding planning of the infra.
1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to
use Ubuntu 20.04 instead or should I stick with 18.04?
2) The documentation recommends using Cephadm for new deployments, so I will use that but
I read that with Cephadm everything is running in containers, so is this the new way to
go? Or is Ceph in containers kind of still experimental?
3) As I will be needing cephfs I will also need MDS servers so with a total of 6 nodes I
am planning the following layout:
Node 1: MGR+MON+MDS
Node 2: MGR+MON+MDS
Node 3: MGR+MON+MDS
Node 4: OSD
Node 5: OSD
Node 6: OSD
Does this make sense? I am mostly interested in stability and HA with this setup.
4) Is there any special kind of demand in terms of disks on the MGR+MON+MDS nodes? Or can
I use have my OS disks on these nodes? As far as I understand the MDS will create a
metadata pool on the OSDs.
Thanks for the hints.
Best,
Mabi
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
31 Mar
31 Mar
4:01 a.m.
On 3/30/21 9:02 PM, mabi wrote:
Hello,
I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and
have a few questions mostly regarding planning of the infra.
1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to
use Ubuntu 20.04 instead or should I stick with 18.04?
20.04 is also an LTS, and perfectly fine to use. On a test cluster we
use it with ceph-ansible with a docker deployment.
2) The documentation recommends using Cephadm for new
deployments, so I will use that but I read that with Cephadm everything is running in
containers, so is this the new way to go? Or is Ceph in containers kind of still
experimental?
It should just work. There is an ongoing effort to improve the
documentation for it. It will get even better in future releases
(Pacific and beyond).
3) As I will be needing cephfs I will also need MDS servers so with a total of 6 nodes I
am planning the following layout:
Node 1: MGR+MON+MDS
Node 2: MGR+MON+MDS
Node 3: MGR+MON+MDS
Node 4: OSD
Node 5: OSD
Node 6: OSD
Does this make sense? I am mostly interested in stability and HA with this setup.
It depends on the specifications of the systems. If you want the least
amount of surprises you isolate each daemon on separate hardware. But
that's not a requirement, just from a ops point of view (my opinion).
For a test setup its probably fine. Do note that MDS daemons can use a
lot of memory (if you allow them). So it depends on the workloads you
want to test. The MDS / MGR don't use any local disks to store state or
whatsoever. But if you want to enable debug logging in some point in
time you needs loads of disk space as it can easily grow with GB/s per
minute.
4) Is there any special kind of demand in terms of disks on the MGR+MON+MDS nodes? Or can
I use have my OS disks on these nodes? As far as I understand the MDS will create a
metadata pool on the OSDs.
For best performance you want to give the MONs their own disk,
preferably flash. Ceph MONs start to use disk space when the cluster is
in an unhealthy state (as to keep track of all PG changes). So it
depends as well. If you know you can fix any kind of disk / hardware
problem within a certain time frame, you don't need *that* big drives.
But if the MONs run out of disk space it's a show stopper.
You *might* run into deadlocks when trying to use CephFS on the MDS
nodes themselves, so try to avoid that.
Gr. Stefan
9:52 a.m.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, March 31, 2021 9:01 AM, Stefan Kooman <stefan(a)bit.nl> wrote:
For best performance you want to give the MONs their
own disk,
preferably flash. Ceph MONs start to use disk space when the cluster is
in an unhealthy state (as to keep track of all PG changes). So it
depends as well. If you know you can fix any kind of disk / hardware
problem within a certain time frame, you don't need that big drives.
But if the MONs run out of disk space it's a show stopper.
You might run into deadlocks when trying to use CephFS on the MDS
nodes themselves, so try to avoid that.
Thanks Stefan for your answer.
That totally makes sense, so on my nodes 1, 2 and 3 which will have MON+MGR+MDS I will add
an additional SSD disk just for MON.
Now because I am planning to use cephadm which deploys everything in containers what would
be the best point point in order to mount that dedidcated SSD disk? Would you suggest to
simply mount that disk under /var/lib/docker or even /var/lib/docker/volumes and like that
have all the containers (MGR+MDS+MON) use the dedicated disk? or is there a ceph.conf
config parameter where I can say which disk I want to use for MON? or any other best
practice suggestions in this regard?
10:16 a.m.
On 3/31/21 2:52 PM, mabi wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, March 31, 2021 9:01 AM, Stefan Kooman <stefan(a)bit.nl> wrote:
The ceph.conf file, nowadays, is hardly used anymore. It's only there to
tell the daemons where the monitors can be found. And if you used DNS
based configuration, you don't need them at all. You can still use them
howerver, but you do not use them to configure disk layout and such.
cephadm uses podman. And from the documentation [1] you should be able
to specify a directory for the monitors to use:
"Daemon containers deployed with cephadm, however, do not need /etc/ceph
at all. Use the --output-dir *<directory>* option to put them in a
different directory (for example, .). This may help avoid conflicts with
an existing Ceph configuration (cephadm or otherwise) on the same host."
So yeah, it looks like you can just mount the SSD in a suitable place
and tell cephadm what it is. For the OSDs you can use a "spec" file
(YAML) to specify how and what drives you want to use: ceph orch daemon
add osd *<host>*:*<device-path>*
Maybe that also works for the monitor disk? Not sure, as I have not
(yet) deployed with cephadm. In our docker setup I mount the SSD bedore
the ceph services (docker containers) are started (systemd mnt dependency).
Gr. Stefan
[1]:
https://docs.ceph.com/en/latest/cephadm/install/#further-information-about-…
For best performance you want to give the MONs
their own disk,
preferably flash. Ceph MONs start to use disk space when the cluster is
in an unhealthy state (as to keep track of all PG changes). So it
depends as well. If you know you can fix any kind of disk / hardware
problem within a certain time frame, you don't need that big drives.
But if the MONs run out of disk space it's a show stopper.
You might run into deadlocks when trying to use CephFS on the MDS
nodes themselves, so try to avoid that.
Thanks Stefan for your answer.
That totally makes sense, so on my nodes 1, 2 and 3 which will have MON+MGR+MDS I will
add an additional SSD disk just for MON.
Now because I am planning to use cephadm which deploys everything in containers what
would be the best point point in order to mount that dedidcated SSD disk? Would you
suggest to simply mount that disk under /var/lib/docker or even /var/lib/docker/volumes
and like that have all the containers (MGR+MDS+MON) use the dedicated disk? or is there a
ceph.conf config parameter where I can say which disk I want to use for MON? or any other
best practice suggestions in this regard?
5:32 p.m.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, March 31, 2021 3:16 PM, Stefan Kooman <stefan(a)bit.nl> wrote:
"Daemon containers deployed with cephadm,
however, do not need /etc/ceph
at all. Use the --output-dir <directory> option to put them in a
different directory (for example, .). This may help avoid conflicts with
an existing Ceph configuration (cephadm or otherwise) on the same host."
So yeah, it looks like you can just mount the SSD in a suitable place
and tell cephadm what it is. For the OSDs you can use a "spec" file
(YAML) to specify how and what drives you want to use: ceph orch daemon
add osd <host>:<device-path>
If I understand the meaning of the "--output-dir" parameter of cephadm correctly
this only says where to store the ceph.conf file and does not say for example where the
data of the MON is stored.
Based on the ceph-mon man page it looks like the correct parameter is called "mon
data", as I deduct from the following man page phrase:
"The mon datapath refers to a directory on a local file system storing monitor data.
It is normally specified via the mon data option in the configuration file." (Source:
https://docs.ceph.com/en/latest/man/8/ceph-mon/)
So how would I pass that parameter to cephadm? and would that data path be a path inside
the container or outside on the host?
3 Apr
3 Apr
6:22 p.m.
On Tue, Mar 30, 2021 at 2:03 PM mabi <mabi(a)protonmail.ch> wrote:
Hello,
I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and
have a few questions mostly regarding planning of the infra.
1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to
use Ubuntu 20.04 instead or should I stick with 18.04?
All of our clusters are 20.04.2 + HWE kernel, they work wonderfully.
2) The documentation recommends using Cephadm for new
deployments, so I will use that but I read that with Cephadm everything is running in
containers, so is this the new way to go? Or is Ceph in containers kind of still
experimental?
We use cephadm + podman for our production clusters, and have had a
great experience. You just need to know how to operate with
containers, so make sure to do some reading about how containers work.
We're using Octopus 15.2.10 (started with earlier 15.2.x and have
upgraded). We will be upgrading to Pacific in the future.
3) As I will be needing cephfs I will also need MDS
servers so with a total of 6 nodes I am planning the following layout:
Node 1: MGR+MON+MDS
Node 2: MGR+MON+MDS
Node 3: MGR+MON+MDS
Node 4: OSD
Node 5: OSD
Node 6: OSD
Does this make sense? I am mostly interested in stability and HA with this setup.
We don't use CephFS, so I can't help here.
4) Is there any special kind of demand in terms of
disks on the MGR+MON+MDS nodes? Or can I use have my OS disks on these nodes? As far as I
understand the MDS will create a metadata pool on the OSDs.
Same, no MDS experience.
> Thanks for the hints.
>
> Best,
> Mabi
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5 Apr
5 Apr
6:20 a.m.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, April 3, 2021 11:22 PM, David Orman <ormandj(a)corenode.com> wrote:
We use cephadm + podman for our production clusters,
and have had a
great experience. You just need to know how to operate with
containers, so make sure to do some reading about how containers work.
We're using Octopus 15.2.10 (started with earlier 15.2.x and have
upgraded). We will be upgrading to Pacific in the future.
Hi David and thanks for your experience feedback.
Regarding doing some container reading, I already know Docker quite well but never used
podman. What kind of reading do you recommend to do beforehand? Anything specific on
docs.ceph.com? or just more general reading on the podman website?
1123
days inactive
1129
days old
7 comments
4 participants
participants (4)
-
David Orman -
DHilsbos＠performair.com -
mabi -
Stefan Kooman