Do you need to use a dedicated server for the MON service?
Hi Ceph users I am designing a CEPH system with 6 servers running full NVMe. Do I need to use 3 separate servers to run the MON services that communicate with OpenStack, or should I integrate the MON services into the OSD servers? What is the recommendation? Thank you. -- Email: tranphong079@gmail.com Skype: tranphong079
Hi, MON servers typically don't consume a lot of resources, so it's fairly common to run MGR/MON daemons on the same machines as the OSDs in a cluster. Do you have any insights on the type of workload or data you plan to use with your Ceph cluster? Thank you, Bogdan Velica croit.io On Fri, Aug 23, 2024 at 12:05 PM Phong Tran Thanh <tranphong079@gmail.com> wrote:
Hi Ceph users
I am designing a CEPH system with 6 servers running full NVMe. Do I need to use 3 separate servers to run the MON services that communicate with OpenStack, or should I integrate the MON services into the OSD servers? What is the recommendation? Thank you.
-- Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ah, for OpenStack.. Yeah I've seen setups like that so it should be fine... On Fri, Aug 23, 2024 at 1:27 PM Bogdan Adrian Velica <vbogdan@gmail.com> wrote:
Hi,
MON servers typically don't consume a lot of resources, so it's fairly common to run MGR/MON daemons on the same machines as the OSDs in a cluster. Do you have any insights on the type of workload or data you plan to use with your Ceph cluster?
Thank you, Bogdan Velica croit.io
On Fri, Aug 23, 2024 at 12:05 PM Phong Tran Thanh <tranphong079@gmail.com> wrote:
Hi Ceph users
I am designing a CEPH system with 6 servers running full NVMe. Do I need to use 3 separate servers to run the MON services that communicate with OpenStack, or should I integrate the MON services into the OSD servers? What is the recommendation? Thank you.
-- Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This is my first time setting up a Ceph cluster for OpenStack. Will running both the Mon service and the OSD service on the same node affect maintenance or upgrades in the future? While running Mon, MDS, and OSD services on the same node does offer some benefits, could someone provide me with additional advice? Vào Th 6, 23 thg 8, 2024 vào lúc 17:31 Bogdan Adrian Velica < vbogdan@gmail.com> đã viết:
Ah, for OpenStack.. Yeah I've seen setups like that so it should be fine...
On Fri, Aug 23, 2024 at 1:27 PM Bogdan Adrian Velica <vbogdan@gmail.com> wrote:
Hi,
MON servers typically don't consume a lot of resources, so it's fairly common to run MGR/MON daemons on the same machines as the OSDs in a cluster. Do you have any insights on the type of workload or data you plan to use with your Ceph cluster?
Thank you, Bogdan Velica croit.io
On Fri, Aug 23, 2024 at 12:05 PM Phong Tran Thanh <tranphong079@gmail.com> wrote:
Hi Ceph users
I am designing a CEPH system with 6 servers running full NVMe. Do I need to use 3 separate servers to run the MON services that communicate with OpenStack, or should I integrate the MON services into the OSD servers? What is the recommendation? Thank you.
-- Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079
Den fre 23 aug. 2024 kl 18:30 skrev Phong Tran Thanh <tranphong079@gmail.com>:
This is my first time setting up a Ceph cluster for OpenStack. Will running both the Mon service and the OSD service on the same node affect maintenance or upgrades in the future? While running Mon, MDS, and OSD services on the same node does offer some benefits, could someone provide me with additional advice?
OSDs can use HUGE amounts of ram when doing internal consistency checks and things like that, so co locating the mon(s) with the OSDs would mean the host can end up swapping or even dying from lack of RAM, and while the machine is heading this way the mon part will probably not run well either, so the OSDs could starve the mon and you would be having two problems at the same time if this ever occurs. -- May the most significant bit of your life be positive.
It is important to understand the behaviour of the ceph services and to make the right hardware dimensioning. You can also limit the resource consumption of each service to prevent the memory consumption of the ceph osd from affecting the ceph mon service on the same node. The resource required for the ceph mds depends heavily on the expected workload. We use the setup you describe very often. Hope this helps you one step further, Joachim Joachim Kraftmayer CEO +49 89 2152527-21 <https://call.ctrlq.org/+49%2089%202152527-21> joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Am Mo., 26. Aug. 2024 um 09:48 Uhr schrieb Janne Johansson < icepic.dz@gmail.com>:
Den fre 23 aug. 2024 kl 18:30 skrev Phong Tran Thanh < tranphong079@gmail.com>:
This is my first time setting up a Ceph cluster for OpenStack. Will running both the Mon service and the OSD service on the same node affect maintenance or upgrades in the future? While running Mon, MDS, and OSD services on the same node does offer some benefits, could someone provide me with additional advice?
OSDs can use HUGE amounts of ram when doing internal consistency checks and things like that, so co locating the mon(s) with the OSDs would mean the host can end up swapping or even dying from lack of RAM, and while the machine is heading this way the mon part will probably not run well either, so the OSDs could starve the mon and you would be having two problems at the same time if this ever occurs.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Ceph users, Thank you for everyone's helpful advice Have a good day. Phong Tran Vào Th 2, 26 thg 8, 2024 vào lúc 20:48 Joachim Kraftmayer < joachim.kraftmayer@clyso.com> đã viết:
It is important to understand the behaviour of the ceph services and to make the right hardware dimensioning. You can also limit the resource consumption of each service to prevent the memory consumption of the ceph osd from affecting the ceph mon service on the same node. The resource required for the ceph mds depends heavily on the expected workload.
We use the setup you describe very often.
Hope this helps you one step further, Joachim
Joachim Kraftmayer
CEO
+49 89 2152527-21 <https://call.ctrlq.org/+49%2089%202152527-21>
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Am Mo., 26. Aug. 2024 um 09:48 Uhr schrieb Janne Johansson < icepic.dz@gmail.com>:
Den fre 23 aug. 2024 kl 18:30 skrev Phong Tran Thanh < tranphong079@gmail.com>:
This is my first time setting up a Ceph cluster for OpenStack. Will running both the Mon service and the OSD service on the same node affect maintenance or upgrades in the future? While running Mon, MDS, and OSD services on the same node does offer some benefits, could someone
provide
me with additional advice?
OSDs can use HUGE amounts of ram when doing internal consistency checks and things like that, so co locating the mon(s) with the OSDs would mean the host can end up swapping or even dying from lack of RAM, and while the machine is heading this way the mon part will probably not run well either, so the OSDs could starve the mon and you would be having two problems at the same time if this ever occurs.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079
If you're using cephadm or other container deployment scheme, upgrades shouldn't be affected. When using a traditional package-based install, there can be snags in the unlikely event of a system rebooting at exactly the wrong time. I personally prefer to keep mon/mgr services on dedicated, modest nodes, having experienced unexpected effects from mon restarts in the past. But with recent Ceph releases, the daemons are better-behaved and lots of people converge in just this way. If your nodes do not have extra RAM available for mon/MDS/etc, you might starve the OSDs a bit. MDS is not multithreaded so it benefits from high freq CPU and doesn't care about cores. Some people thus tie MDS to specifically-chosen systems.
On Aug 23, 2024, at 11:10 AM, Phong Tran Thanh <tranphong079@gmail.com> wrote:
This is my first time setting up a Ceph cluster for OpenStack. Will running both the Mon service and the OSD service on the same node affect maintenance or upgrades in the future? While running Mon, MDS, and OSD services on the same node does offer some benefits, could someone provide me with additional advice?
Vào Th 6, 23 thg 8, 2024 vào lúc 17:31 Bogdan Adrian Velica < vbogdan@gmail.com> đã viết:
Ah, for OpenStack.. Yeah I've seen setups like that so it should be fine...
On Fri, Aug 23, 2024 at 1:27 PM Bogdan Adrian Velica <vbogdan@gmail.com> wrote:
Hi,
MON servers typically don't consume a lot of resources, so it's fairly common to run MGR/MON daemons on the same machines as the OSDs in a cluster. Do you have any insights on the type of workload or data you plan to use with your Ceph cluster?
Thank you, Bogdan Velica croit.io
On Fri, Aug 23, 2024 at 12:05 PM Phong Tran Thanh <tranphong079@gmail.com> wrote:
Hi Ceph users
I am designing a CEPH system with 6 servers running full NVMe. Do I need to use 3 separate servers to run the MON services that communicate with OpenStack, or should I integrate the MON services into the OSD servers? What is the recommendation? Thank you.
-- Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Trân trọng, ----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 8/23/24 10:58, Phong Tran Thanh wrote:
Hi Ceph users
I am designing a CEPH system with 6 servers running full NVMe. Do I need to use 3 separate servers to run the MON services that communicate with OpenStack, or should I integrate the MON services into the OSD servers? What is the recommendation? Thank you.
We often co-locate MONs, MGRs and OSDs (using LXD containers) but you need to make sure you're meeting resource requirements for all. See https://docs.ceph.com/en/latest/start/hardware-recommendations/ for some guidelines. This is obviously important for disk space, but also for RAM. Important because while memory consumption of those services might be deceptively low while the cluster is healthy, the memory usage during recovery and rebalancing could be much higher, so be sure to give it some room. Also, especially NVMe OSDs benefit from additional RAM in terms of performance
-- Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Anthony D'Atri
-
Bogdan Adrian Velica
-
Janne Johansson
-
Joachim Kraftmayer
-
Peter Sabaini
-
Phong Tran Thanh