mons with "low on available space" message
Hi! its about my Lab setup. As I didn't have a thought about the space requirements for the root file system I want to know if there is a rule by thumb how much space a cpeh mon might need? rightnow I only have the "whoami" container running in rook-ceph so it does consume nearly no space in the OSD. But how far can I go with only 1.7 GB left on my Raspberry Pi's sdcard? Greetings filip
Hi! its about my Lab setup. As I didn't have a thought about the space requirements for the root file system I want to know if there is a rule by thumb how much space a cpeh mon might need? rightnow I only have the "whoami" container running in rook-ceph so it does consume nearly no space in the OSD. But how far can I go with only 1.7 GB left on my Raspberry Pi's sdcard?
Mons can consume A LOT of space when the cluster is sad, so all tests done while the cluster is ok and all OSDs are working fine will not tell you the requirements later. -- May the most significant bit of your life be positive.
That's a function of how big the cluster is, and how much churn there is. For production size clusters I would allot at least 50GB, but it sounds like you don't have one. I suspect that OSD and mon RocksDBs will burn an SD card in short order. ymmv. 1.7 GB seems iffy, with all the other stuff that a system might write, including temporary usage during DB compactions.
On Nov 15, 2025, at 4:10 PM, filip Mutterer <filip@3dds.io> wrote:
Hi! its about my Lab setup. As I didn't have a thought about the space requirements for the root file system I want to know if there is a rule by thumb how much space a cpeh mon might need? rightnow I only have the "whoami" container running in rook-ceph so it does consume nearly no space in the OSD. But how far can I go with only 1.7 GB left on my Raspberry Pi's sdcard?
Greetings filip
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi everyone, I recalled the figure of 120GB and sought to verify its source:: - It appears to originate from the RHCS 7 documentation [1]: "Create a dedicated Storage Partition Size for /var/lib/ceph with a minimum size of 120 GB; 240 GB is recommended. If a dedicated partition is not feasible, ensure that /var is on a dedicated partition with at least the above-mentioned free space." - The RHCS 8 + IBM Storage Ceph documentation states "10 GB per mon-container, 50 GB Recommended" [2] which appears quite low and risky to me. - The Upstream doc says "100 GB per daemon, SSD is recommended" [3]. This highlights an inconsistency. I've seen DBs growing up to 60GB during long step recovery on 1000+ OSDs clusters (see [4] for details). So I think 120GB+ is still a good number. Best regards, Frédéric. [1] https://docs.redhat.com/en/documentation/red_hat_ceph_storage/7/html-single/... [2] https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=hardware-minimum-recomm... [3] https://docs.ceph.com/en/latest/start/hardware-recommendations/ [4] https://docs.ceph.com/en/latest/rados/operations/health-checks/#mon-disk-big -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Check our online Config Diff Tool, it's great! https://analyzer.clyso.com/#/analyzer/config-diff https://clyso.com | frederic.nass@clyso.com Le sam. 15 nov. 2025 à 22:54, Anthony D'Atri <aad@dreamsnake.net> a écrit :
That's a function of how big the cluster is, and how much churn there is. For production size clusters I would allot at least 50GB, but it sounds like you don't have one. I suspect that OSD and mon RocksDBs will burn an SD card in short order. ymmv. 1.7 GB seems iffy, with all the other stuff that a system might write, including temporary usage during DB compactions.
On Nov 15, 2025, at 4:10 PM, filip Mutterer <filip@3dds.io> wrote:
Hi! its about my Lab setup. As I didn't have a thought about the space requirements for the root file system I want to know if there is a rule by thumb how much space a cpeh mon might need? rightnow I only have the "whoami" container running in rook-ceph so it does consume nearly no space in the OSD. But how far can I go with only 1.7 GB left on my Raspberry Pi's sdcard?
Greetings filip
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 11/15/25 22:10, filip Mutterer wrote:
its about my Lab setup. As I didn't have a thought about the space requirements for the root file system I want to know if there is a rule by thumb how much space a cpeh mon might need? rightnow I only have the "whoami" container running in rook-ceph so it does consume nearly no space in the OSD. But how far can I go with only 1.7 GB left on my Raspberry Pi's sdcard?
Do not run something that does a lot of random writes on an SD card. I have seen Ceph clusters dying because the MON db was not fast enough because of a detoriating SD card. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
This is true even with mons on HDD, though presumably this is a very small cluster with relatively light mon workload
On Nov 16, 2025, at 12:15 PM, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
On 11/15/25 22:10, filip Mutterer wrote:
its about my Lab setup. As I didn't have a thought about the space requirements for the root file system I want to know if there is a rule by thumb how much space a cpeh mon might need? rightnow I only have the "whoami" container running in rook-ceph so it does consume nearly no space in the OSD. But how far can I go with only 1.7 GB left on my Raspberry Pi's sdcard?
Do not run something that does a lot of random writes on an SD card.
I have seen Ceph clusters dying because the MON db was not fast enough because of a detoriating SD card.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, as it is a test cluster I think I'll run it until its full and then set it up with the minimum specs as mentioned by Frédéric. While the cluster was doing "nothing" it used 200MB so that thing will never fly for long. Greetings filip
participants (5)
-
Anthony D'Atri
-
filip Mutterer
-
Frédéric Nass
-
Janne Johansson
-
Robert Sander