Ceph MON onde RAM recommandations
Hi community, we’re designing our first Ceph cluster here and we’re struggeling with the MON (and also MDS) nodes RAM recommendations. It sais „like 1GB per daemon instance“, but what daemons are meant here? The range is from per MON (MDS) daemon, which would be fairly low, to per OSD daemon, which would be fairly high. Any help would be appreciated. Kind regards Sebastian -- Sebastian Wiesner IT-Systemadministrator / Application Developer Department of Operative Processes and Systems (OPS) Dresden University of Technology Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden, Germany phone: +49 351 463-37828 fax: +49 351 463-32164 e-mail: sebastian.wiesner@tu-dresden.de WWW: http://www.tu-dresden.de/zih
Hi, From what I can remember there is a bit of a difference if you have OSD daemons on SSD or HDD and their capacity... If you have the OSD daemons on HDD the recommendation is to have a bit more than 1G of RAM. If you have OSD on SSD disk then 1G of ram is the minim recommendation ... Best regards, Bogdan. On Tue, Jul 9, 2019 at 7:55 PM Sebastian Wiesner < sebastian.wiesner@tu-dresden.de> wrote:
Hi community,
we’re designing our first Ceph cluster here and we’re struggeling with the MON (and also MDS) nodes RAM recommendations. It sais „like 1GB per daemon instance“, but what daemons are meant here? The range is from per MON (MDS) daemon, which would be fairly low, to per OSD daemon, which would be fairly high. Any help would be appreciated.
Kind regards
Sebastian
--
Sebastian Wiesner
IT-Systemadministrator / Application Developer
Department of Operative Processes and Systems (OPS)
Dresden University of Technology
Center for Information Services and High Performance Computing (ZIH)
D-01062 Dresden, Germany
phone: +49 351 463-37828
fax: +49 351 463-32164
e-mail: sebastian.wiesner@tu-dresden.de
WWW: http://www.tu-dresden.de/zih
_______________________________________________ Ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, it’s not about the OSD RAM recommendations here. This ist the only point that is absolutely clear to us. It’s about the MON and MDS recommendations. The official Ceph documentation also says „1 GB per daemon instance“ for monitors and metadata hosts. But for what daemon instances? There will be 3 monitor and 1 metadata servers for now, each of them running one of their daemon instances. Or am I completely wrong here? So this would lead to 1 GB of RAM per node (disregarding OS needs), which I can’t believe is true. Otherwise there will be 180 OSD instances, which would lead to 180 GB of RAM per monitor and metadata node, which I don’t want to believe is true 😉 Kind regards Sebastian -- Sebastian Wiesner IT-Systemadministrator / Application Developer Department of Operative Processes and Systems (OPS) Dresden University of Technology Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden, Germany phone: +49 351 463-37828 fax: +49 351 463-32164 e-mail: sebastian.wiesner@tu-dresden.de WWW: http://www.tu-dresden.de/zih Von: Bogdan Adrian Velica <vbogdan@gmail.com> Datum: Mittwoch, 10. Juli 2019 um 09:13 An: "Wiesner, Sebastian" <sebastian.wiesner@tu-dresden.de> Cc: "ceph-users@ceph.io" <ceph-users@ceph.io> Betreff: Re: [Ceph-users] Ceph MON onde RAM recommandations Hi, From what I can remember there is a bit of a difference if you have OSD daemons on SSD or HDD and their capacity... If you have the OSD daemons on HDD the recommendation is to have a bit more than 1G of RAM. If you have OSD on SSD disk then 1G of ram is the minim recommendation ... Best regards, Bogdan. On Tue, Jul 9, 2019 at 7:55 PM Sebastian Wiesner <sebastian.wiesner@tu-dresden.de> wrote: Hi community, we’re designing our first Ceph cluster here and we’re struggeling with the MON (and also MDS) nodes RAM recommendations. It sais „like 1GB per daemon instance“, but what daemons are meant here? The range is from per MON (MDS) daemon, which would be fairly low, to per OSD daemon, which would be fairly high. Any help would be appreciated. Kind regards Sebastian -- Sebastian Wiesner IT-Systemadministrator / Application Developer Department of Operative Processes and Systems (OPS) Dresden University of Technology Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden, Germany phone: +49 351 463-37828 fax: +49 351 463-32164 e-mail: sebastian.wiesner@tu-dresden.de WWW: http://www.tu-dresden.de/zih _______________________________________________ Ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
RAM recommendations are tricky. We feel compelled to come up with a formula, but there are multiple factors. I will share my experience : We started with min 32 GB per mon, which seemed fine for 12x24 OSDs. As one specific cluster grew to 41x24 I found that 32 was enough for steady-state runtime, but not enough to boot. I did emergency upgrades to avert disaster. I’ve seen occasions where one mon in any cluster going down caused another, perhaps the new leader but not necessarily, to spike to >100 GB used. Sometimes a heap release works, sometimes restarts are needed. The number of PGs is likely a factor as is the OS - this is on Ubuntu Trusty. Some time ago there was discussion where Sage expressed a belief that newer releases of TCmalloc behave better. Also remember the Ceph-mgr daemon running in parallel. Usage can grow during distress events too. In my situation 128 GB is the min I’ll accept. YMMV.
participants (3)
-
Anthony D'Atri
-
Bogdan Adrian Velica
-
Sebastian Wiesner