I recently set up a 5-node Proxmox/Ceph cluster with 10 OSDs each node for a total for 50 nodes (approximately 135TB). All the nodes have 768GB RAM and a single 72-Core (144 thread) Epyx CPU. I set the ceph pool size to: 3 and the minimum size: 2. The performance seemed good and everything was happy in cephland! The other day I spoke to a ceph consultant and he recommended I change the pool size from 3/2 to 2/1. He cited several valid points: more usable storage space, faster rebuilds, better performance... I followed his advice and changed it. The benchmark performance was about the same, but the recovery time when I took a node down was improved. I really do like the idea of the extra storage space! So now I am confused whether I should leave it or go back to 3 replicas. Anyone have any thoughts or compelling reasons I should leave it or change it back? Shawn Shawn Heil Phone: (608) 836-7041 | Direct: 6084102333 Email: shawn.heil@brucecompany.com<mailto:shawn.heil@brucecompany.com> | Website: www.brucecompany.com<http://www.brucecompany.com> [cid:SocialLink_Facebook_32x32_24bd5ab0-6667-4cda-9838-e5caaf24a024.png]Find us on Facebook<https://www.facebook.com/pages/The-Bruce-Company/113279807065?ref=hl> [cid:BruceCompanyHomepageLogo(2025)_429ac410-efd6-49ca-8739-0317666483cd.jpg]
I recently set up a 5-node Proxmox/Ceph cluster with 10 OSDs each node for a total for 50 nodes (approximately 135TB). All the nodes have 768GB RAM and a single 72-Core (144 thread) Epyx CPU. I set the ceph pool size to: 3 and the minimum size: 2.
The performance seemed good and everything was happy in cephland! The other day I spoke to a ceph consultant and he recommended I change the pool size from 3/2 to 2/1.
Danger, Will Robinson!
He cited several valid points: more usable storage space, faster rebuilds, better performance… I followed his advice and changed it. The benchmark performance was about the same, but the recovery time when I took a node down was improved. I really do like the idea of the extra storage space! So now I am confused whether I should leave it or go back to 3 replicas.
While all of the above are true, you run the risk of data loss or corruption. Sometimes data is a scratchpad or easily recreated, but if your VMs are using RBD volumes for boot drives, that likely isn't the case. With 2/1 there are certain sequences of drive / node / network / daemon failures that can result in the loss of data, or not knowing which if either copy is actually up to date. I have seen this happen with my own eyes, after I advised $company of the risk and they decided it wasn't a priority. The result was that a customer lost data. Say you take a node down for maintenance, and while it's down, an OSD drive on another node fails. Most likely it's taken writes after the first node was taken down, so now the only current copy of data is lost. Off this mortal coil. Pushing up the daisies. You get the idea. See Dan's presentation from Ceph Day Seattle for context.
Anyone have any thoughts or compelling reasons I should leave it or change it back?
Unless you have very specific needs, change it back. You can have a separate size=2 pool for non-critical data if you like, with all manner of warnings to users.
Shawn Shawn Heil Phone: (608) 836-7041 | Direct: 6084102333 Email: shawn.heil@brucecompany.com <mailto:shawn.heil@brucecompany.com> | Website: www.brucecompany.com <http://www.brucecompany.com/> <SocialLink_Facebook_32x32_24bd5ab0-6667-4cda-9838-e5caaf24a024.png>Find us on Facebook <https://www.facebook.com/pages/The-Bruce-Company/113279807065?ref=hl>
<BruceCompanyHomepageLogo(2025)_429ac410-efd6-49ca-8739-0317666483cd.jpg>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hi Shawn, setting replikation size to 2 is dangerous. It's not a question of if you lose data, but when! With regard to data integrity, this is an extremely bad decision. If you want, I can find the mathematical proof for you. Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Anthony D'Atri <aad@dreamsnake.net> schrieb am Sa., 16. Aug. 2025, 03:20:
I recently set up a 5-node Proxmox/Ceph cluster with 10 OSDs each node for a total for 50 nodes (approximately 135TB). All the nodes have 768GB RAM and a single 72-Core (144 thread) Epyx CPU. I set the ceph pool size to: 3 and the minimum size: 2.
The performance seemed good and everything was happy in cephland! The other day I spoke to a ceph consultant and he recommended I change the pool size from 3/2 to 2/1.
Danger, Will Robinson!
He cited several valid points: more usable storage space, faster rebuilds, better performance… I followed his advice and changed it. The benchmark performance was about the same, but the recovery time when I took a node down was improved. I really do like the idea of the extra storage space! So now I am confused whether I should leave it or go back to 3 replicas.
While all of the above are true, you run the risk of data loss or corruption. Sometimes data is a scratchpad or easily recreated, but if your VMs are using RBD volumes for boot drives, that likely isn't the case.
With 2/1 there are certain sequences of drive / node / network / daemon failures that can result in the loss of data, or not knowing which if either copy is actually up to date.
I have seen this happen with my own eyes, after I advised $company of the risk and they decided it wasn't a priority. The result was that a customer lost data.
Say you take a node down for maintenance, and while it's down, an OSD drive on another node fails. Most likely it's taken writes after the first node was taken down, so now the only current copy of data is lost. Off this mortal coil. Pushing up the daisies. You get the idea. See Dan's presentation from Ceph Day Seattle for context.
Anyone have any thoughts or compelling reasons I should leave it or change it back?
Unless you have very specific needs, change it back. You can have a separate size=2 pool for non-critical data if you like, with all manner of warnings to users.
Shawn Shawn Heil Phone: (608) 836-7041 | Direct: 6084102333 Email: shawn.heil@brucecompany.com <mailto:shawn.heil@brucecompany.com>
| Website: www.brucecompany.com <http://www.brucecompany.com/>
<SocialLink_Facebook_32x32_24bd5ab0-6667-4cda-9838-e5caaf24a024.png>Find us on Facebook < https://www.facebook.com/pages/The-Bruce-Company/113279807065?ref=hl>
<BruceCompanyHomepageLogo(2025)_429ac410-efd6-49ca-8739-0317666483cd.jpg>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io <mailto: ceph-users-leave@ceph.io>
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Anthony D'Atri
-
Joachim Kraftmayer
-
Shawn Heil