First 6 nodes cluster with Octopus
Hello, I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and have a few questions mostly regarding planning of the infra. 1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to use Ubuntu 20.04 instead or should I stick with 18.04? 2) The documentation recommends using Cephadm for new deployments, so I will use that but I read that with Cephadm everything is running in containers, so is this the new way to go? Or is Ceph in containers kind of still experimental? 3) As I will be needing cephfs I will also need MDS servers so with a total of 6 nodes I am planning the following layout: Node 1: MGR+MON+MDS Node 2: MGR+MON+MDS Node 3: MGR+MON+MDS Node 4: OSD Node 5: OSD Node 6: OSD Does this make sense? I am mostly interested in stability and HA with this setup. 4) Is there any special kind of demand in terms of disks on the MGR+MON+MDS nodes? Or can I use have my OS disks on these nodes? As far as I understand the MDS will create a metadata pool on the OSDs. Thanks for the hints. Best, Mabi
Mabi; We're running Nautilus, and I am not wholly convinced of the "everything in containers" view of the world, so take this with a small grain of salt... 1) We don't run Ubuntu, sorry. I suspect the documentation highlights 18.04 because it's the current LTS release. Personally, if I had a preference of 20.04 over 18.04, I would attempt to build a cluster on 20.04, and see how it goes. You might also look at this: https://www.server-world.info/en/note?os=Ubuntu_20.04&p=ceph15&f=1 2) Containers are the preferred way of doing things in Octopus, so yes it's considered stable. 3) Our first evaluation cluster was 3 Intel Atom C3000 nodes, with each node running all the daemons (MON, MGR, MDS, 2 x OSD). Worked fine, and allowed me to demonstrate the concepts in a size I could carry around. 4) Yes, and No... When the Cluster is happy, everything is generally happy. In certain Warning and Error situations, MONs can chew through the HD space fairly quickly. I'm not familiar with the HD usage of the daemons. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: mabi [mailto:mabi@protonmail.ch] Sent: Tuesday, March 30, 2021 12:03 PM To: ceph-users@ceph.io Subject: [ceph-users] First 6 nodes cluster with Octopus Hello, I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and have a few questions mostly regarding planning of the infra. 1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to use Ubuntu 20.04 instead or should I stick with 18.04? 2) The documentation recommends using Cephadm for new deployments, so I will use that but I read that with Cephadm everything is running in containers, so is this the new way to go? Or is Ceph in containers kind of still experimental? 3) As I will be needing cephfs I will also need MDS servers so with a total of 6 nodes I am planning the following layout: Node 1: MGR+MON+MDS Node 2: MGR+MON+MDS Node 3: MGR+MON+MDS Node 4: OSD Node 5: OSD Node 6: OSD Does this make sense? I am mostly interested in stability and HA with this setup. 4) Is there any special kind of demand in terms of disks on the MGR+MON+MDS nodes? Or can I use have my OS disks on these nodes? As far as I understand the MDS will create a metadata pool on the OSDs. Thanks for the hints. Best, Mabi _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 3/30/21 9:02 PM, mabi wrote:
Hello,
I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and have a few questions mostly regarding planning of the infra.
1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to use Ubuntu 20.04 instead or should I stick with 18.04?
20.04 is also an LTS, and perfectly fine to use. On a test cluster we use it with ceph-ansible with a docker deployment.
2) The documentation recommends using Cephadm for new deployments, so I will use that but I read that with Cephadm everything is running in containers, so is this the new way to go? Or is Ceph in containers kind of still experimental?
It should just work. There is an ongoing effort to improve the documentation for it. It will get even better in future releases (Pacific and beyond).
3) As I will be needing cephfs I will also need MDS servers so with a total of 6 nodes I am planning the following layout:
Node 1: MGR+MON+MDS Node 2: MGR+MON+MDS Node 3: MGR+MON+MDS Node 4: OSD Node 5: OSD Node 6: OSD
Does this make sense? I am mostly interested in stability and HA with this setup.
It depends on the specifications of the systems. If you want the least amount of surprises you isolate each daemon on separate hardware. But that's not a requirement, just from a ops point of view (my opinion). For a test setup its probably fine. Do note that MDS daemons can use a lot of memory (if you allow them). So it depends on the workloads you want to test. The MDS / MGR don't use any local disks to store state or whatsoever. But if you want to enable debug logging in some point in time you needs loads of disk space as it can easily grow with GB/s per minute.
4) Is there any special kind of demand in terms of disks on the MGR+MON+MDS nodes? Or can I use have my OS disks on these nodes? As far as I understand the MDS will create a metadata pool on the OSDs.
For best performance you want to give the MONs their own disk, preferably flash. Ceph MONs start to use disk space when the cluster is in an unhealthy state (as to keep track of all PG changes). So it depends as well. If you know you can fix any kind of disk / hardware problem within a certain time frame, you don't need *that* big drives. But if the MONs run out of disk space it's a show stopper. You *might* run into deadlocks when trying to use CephFS on the MDS nodes themselves, so try to avoid that. Gr. Stefan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, March 31, 2021 9:01 AM, Stefan Kooman <stefan@bit.nl> wrote:
For best performance you want to give the MONs their own disk, preferably flash. Ceph MONs start to use disk space when the cluster is in an unhealthy state (as to keep track of all PG changes). So it depends as well. If you know you can fix any kind of disk / hardware problem within a certain time frame, you don't need that big drives. But if the MONs run out of disk space it's a show stopper.
You might run into deadlocks when trying to use CephFS on the MDS nodes themselves, so try to avoid that.
Thanks Stefan for your answer. That totally makes sense, so on my nodes 1, 2 and 3 which will have MON+MGR+MDS I will add an additional SSD disk just for MON. Now because I am planning to use cephadm which deploys everything in containers what would be the best point point in order to mount that dedidcated SSD disk? Would you suggest to simply mount that disk under /var/lib/docker or even /var/lib/docker/volumes and like that have all the containers (MGR+MDS+MON) use the dedicated disk? or is there a ceph.conf config parameter where I can say which disk I want to use for MON? or any other best practice suggestions in this regard?
On 3/31/21 2:52 PM, mabi wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, March 31, 2021 9:01 AM, Stefan Kooman <stefan@bit.nl> wrote:
For best performance you want to give the MONs their own disk, preferably flash. Ceph MONs start to use disk space when the cluster is in an unhealthy state (as to keep track of all PG changes). So it depends as well. If you know you can fix any kind of disk / hardware problem within a certain time frame, you don't need that big drives. But if the MONs run out of disk space it's a show stopper.
You might run into deadlocks when trying to use CephFS on the MDS nodes themselves, so try to avoid that.
Thanks Stefan for your answer.
That totally makes sense, so on my nodes 1, 2 and 3 which will have MON+MGR+MDS I will add an additional SSD disk just for MON.
Now because I am planning to use cephadm which deploys everything in containers what would be the best point point in order to mount that dedidcated SSD disk? Would you suggest to simply mount that disk under /var/lib/docker or even /var/lib/docker/volumes and like that have all the containers (MGR+MDS+MON) use the dedicated disk? or is there a ceph.conf config parameter where I can say which disk I want to use for MON? or any other best practice suggestions in this regard?
The ceph.conf file, nowadays, is hardly used anymore. It's only there to tell the daemons where the monitors can be found. And if you used DNS based configuration, you don't need them at all. You can still use them howerver, but you do not use them to configure disk layout and such. cephadm uses podman. And from the documentation [1] you should be able to specify a directory for the monitors to use: "Daemon containers deployed with cephadm, however, do not need /etc/ceph at all. Use the --output-dir *<directory>* option to put them in a different directory (for example, .). This may help avoid conflicts with an existing Ceph configuration (cephadm or otherwise) on the same host." So yeah, it looks like you can just mount the SSD in a suitable place and tell cephadm what it is. For the OSDs you can use a "spec" file (YAML) to specify how and what drives you want to use: ceph orch daemon add osd *<host>*:*<device-path>* Maybe that also works for the monitor disk? Not sure, as I have not (yet) deployed with cephadm. In our docker setup I mount the SSD bedore the ceph services (docker containers) are started (systemd mnt dependency). Gr. Stefan [1]: https://docs.ceph.com/en/latest/cephadm/install/#further-information-about-c...
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, March 31, 2021 3:16 PM, Stefan Kooman <stefan@bit.nl> wrote:
"Daemon containers deployed with cephadm, however, do not need /etc/ceph at all. Use the --output-dir <directory> option to put them in a different directory (for example, .). This may help avoid conflicts with an existing Ceph configuration (cephadm or otherwise) on the same host."
So yeah, it looks like you can just mount the SSD in a suitable place and tell cephadm what it is. For the OSDs you can use a "spec" file (YAML) to specify how and what drives you want to use: ceph orch daemon add osd <host>:<device-path>
If I understand the meaning of the "--output-dir" parameter of cephadm correctly this only says where to store the ceph.conf file and does not say for example where the data of the MON is stored. Based on the ceph-mon man page it looks like the correct parameter is called "mon data", as I deduct from the following man page phrase: "The mon datapath refers to a directory on a local file system storing monitor data. It is normally specified via the mon data option in the configuration file." (Source: https://docs.ceph.com/en/latest/man/8/ceph-mon/) So how would I pass that parameter to cephadm? and would that data path be a path inside the container or outside on the host?
On Tue, Mar 30, 2021 at 2:03 PM mabi <mabi@protonmail.ch> wrote:
Hello,
I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and have a few questions mostly regarding planning of the infra.
1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to use Ubuntu 20.04 instead or should I stick with 18.04?
All of our clusters are 20.04.2 + HWE kernel, they work wonderfully.
2) The documentation recommends using Cephadm for new deployments, so I will use that but I read that with Cephadm everything is running in containers, so is this the new way to go? Or is Ceph in containers kind of still experimental?
We use cephadm + podman for our production clusters, and have had a great experience. You just need to know how to operate with containers, so make sure to do some reading about how containers work. We're using Octopus 15.2.10 (started with earlier 15.2.x and have upgraded). We will be upgrading to Pacific in the future.
3) As I will be needing cephfs I will also need MDS servers so with a total of 6 nodes I am planning the following layout:
Node 1: MGR+MON+MDS Node 2: MGR+MON+MDS Node 3: MGR+MON+MDS Node 4: OSD Node 5: OSD Node 6: OSD
Does this make sense? I am mostly interested in stability and HA with this setup.
We don't use CephFS, so I can't help here.
4) Is there any special kind of demand in terms of disks on the MGR+MON+MDS nodes? Or can I use have my OS disks on these nodes? As far as I understand the MDS will create a metadata pool on the OSDs.
Same, no MDS experience.
Thanks for the hints.
Best, Mabi
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Saturday, April 3, 2021 11:22 PM, David Orman <ormandj@corenode.com> wrote:
We use cephadm + podman for our production clusters, and have had a great experience. You just need to know how to operate with containers, so make sure to do some reading about how containers work. We're using Octopus 15.2.10 (started with earlier 15.2.x and have upgraded). We will be upgrading to Pacific in the future.
Hi David and thanks for your experience feedback. Regarding doing some container reading, I already know Docker quite well but never used podman. What kind of reading do you recommend to do beforehand? Anything specific on docs.ceph.com? or just more general reading on the podman website?
participants (4)
-
David Orman
-
DHilsbos＠performair.com
-
mabi
-
Stefan Kooman