concept of ceph and 2 datacenters
hi there, i have a design/concept question, to see, whats outside and which kind of redundancy you use. actually, we use 2 ceph cluster with rbd-mirror to have an cold-standby clone. but, rbd mirror is not application consistend. so we cannot be sure, that all vms (kvm/proxy) are running. we also waste a lot of hardware. so now, we think about one big cluster over the two datacenters (two buildings). my queston is, do you care about ceph redundancy or is one ceph with backups enough for you? i know, with ceph, we are aware of hdd or server failure. but, are software failures a real scenario? would be great, to get some ideas from you. also about the bandwidth between the 2 datacenters. we are using 2x 6 proxmox server with 2x6x9 osd (sas ssd). thanks for help, my minds are rotating. kind regards, ronny -- Ronny Lippold System Administrator -- Spark 5 GmbH Rheinstr. 97 64295 Darmstadt Germany -- Fon: +49-6151-8508-050 Fax: +49-6151-8508-111 Mail: ronny.lippold@spark5.de Web: https://www.spark5.de -- Geschäftsführer: Henning Munte, Michael Mylius Amtsgericht Darmstadt, HRB 7809 --
Hi Ronny, This is a good starting point for your design. https://docs.ceph.com/en/latest/rados/operations/stretch-mode/ My personal experience says that 2 DC Ceph deployment could suffer from a 'split brain' situation. If you have any chance to create a 3 DC configuration, I would suggest to consider it. It could be more expensive, but it definitely will be more reliable and fault tolerant. Sincerely, Vladimir Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: ronny.lippold@spark5.de <ronny.lippold@spark5.de> Sent: Tuesday, February 13, 2024 6:50:50 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] concept of ceph and 2 datacenters hi there, i have a design/concept question, to see, whats outside and which kind of redundancy you use. actually, we use 2 ceph cluster with rbd-mirror to have an cold-standby clone. but, rbd mirror is not application consistend. so we cannot be sure, that all vms (kvm/proxy) are running. we also waste a lot of hardware. so now, we think about one big cluster over the two datacenters (two buildings). my queston is, do you care about ceph redundancy or is one ceph with backups enough for you? i know, with ceph, we are aware of hdd or server failure. but, are software failures a real scenario? would be great, to get some ideas from you. also about the bandwidth between the 2 datacenters. we are using 2x 6 proxmox server with 2x6x9 osd (sas ssd). thanks for help, my minds are rotating. kind regards, ronny -- Ronny Lippold System Administrator -- Spark 5 GmbH Rheinstr. 97 64295 Darmstadt Germany -- Fon: +49-6151-8508-050 Fax: +49-6151-8508-111 Mail: ronny.lippold@spark5.de Web: https://www.spark5.de -- Geschäftsführer: Henning Munte, Michael Mylius Amtsgericht Darmstadt, HRB 7809 -- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 14.02.24 06:59, Vladimir Sigunov wrote:
Hi Ronny, This is a good starting point for your design. https://docs.ceph.com/en/latest/rados/operations/stretch-mode/
My personal experience says that 2 DC Ceph deployment could suffer from a 'split brain' situation. If you have any chance to create a 3 DC configuration, I would suggest to consider it. It could be more expensive, but it definitely will be more reliable and fault tolerant.
The docs you linked mention[0] a tiebreaker monitor (which could be a VM / in the cloud) -- have you used something like this? [0] https://docs.ceph.com/en/latest/rados/operations/stretch-mode/#limitations-o... cheers, peter.
Sincerely, Vladimir
Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: ronny.lippold@spark5.de <ronny.lippold@spark5.de> Sent: Tuesday, February 13, 2024 6:50:50 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] concept of ceph and 2 datacenters
hi there, i have a design/concept question, to see, whats outside and which kind of redundancy you use.
actually, we use 2 ceph cluster with rbd-mirror to have an cold-standby clone. but, rbd mirror is not application consistend. so we cannot be sure, that all vms (kvm/proxy) are running. we also waste a lot of hardware.
so now, we think about one big cluster over the two datacenters (two buildings).
my queston is, do you care about ceph redundancy or is one ceph with backups enough for you? i know, with ceph, we are aware of hdd or server failure. but, are software failures a real scenario?
would be great, to get some ideas from you. also about the bandwidth between the 2 datacenters. we are using 2x 6 proxmox server with 2x6x9 osd (sas ssd).
thanks for help, my minds are rotating.
kind regards, ronny
-- Ronny Lippold System Administrator
-- Spark 5 GmbH Rheinstr. 97 64295 Darmstadt Germany -- Fon: +49-6151-8508-050 Fax: +49-6151-8508-111 Mail: ronny.lippold@spark5.de Web: https://www.spark5.de -- Geschäftsführer: Henning Munte, Michael Mylius Amtsgericht Darmstadt, HRB 7809 -- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Notably, the tiebreaker should be in a third location.
On Feb 14, 2024, at 05:16, Peter Sabaini <peter@sabaini.at> wrote:
On 14.02.24 06:59, Vladimir Sigunov wrote:
Hi Ronny, This is a good starting point for your design. https://docs.ceph.com/en/latest/rados/operations/stretch-mode/
My personal experience says that 2 DC Ceph deployment could suffer from a 'split brain' situation. If you have any chance to create a 3 DC configuration, I would suggest to consider it. It could be more expensive, but it definitely will be more reliable and fault tolerant.
The docs you linked mention[0] a tiebreaker monitor (which could be a VM / in the cloud) -- have you used something like this?
[0] https://docs.ceph.com/en/latest/rados/operations/stretch-mode/#limitations-o...
cheers, peter.
Sincerely, Vladimir
Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: ronny.lippold@spark5.de <ronny.lippold@spark5.de> Sent: Tuesday, February 13, 2024 6:50:50 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] concept of ceph and 2 datacenters
hi there, i have a design/concept question, to see, whats outside and which kind of redundancy you use.
actually, we use 2 ceph cluster with rbd-mirror to have an cold-standby clone. but, rbd mirror is not application consistend. so we cannot be sure, that all vms (kvm/proxy) are running. we also waste a lot of hardware.
so now, we think about one big cluster over the two datacenters (two buildings).
my queston is, do you care about ceph redundancy or is one ceph with backups enough for you? i know, with ceph, we are aware of hdd or server failure. but, are software failures a real scenario?
would be great, to get some ideas from you. also about the bandwidth between the 2 datacenters. we are using 2x 6 proxmox server with 2x6x9 osd (sas ssd).
thanks for help, my minds are rotating.
kind regards, ronny
-- Ronny Lippold System Administrator
-- Spark 5 GmbH Rheinstr. 97 64295 Darmstadt Germany -- Fon: +49-6151-8508-050 Fax: +49-6151-8508-111 Mail: ronny.lippold@spark5.de Web: https://www.spark5.de -- Geschäftsführer: Henning Munte, Michael Mylius Amtsgericht Darmstadt, HRB 7809 -- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
hi vladimir, thanks for answering ... of cause, we will build an 3 dc (tiebraker or server) setup. i'm not sure, what to do with "disaster recovery". is it real, that a ceph cluster can be completly broken? kind regards, ronny -- Ronny Lippold System Administrator -- Spark 5 GmbH Rheinstr. 97 64295 Darmstadt Germany -- Fon: +49-6151-8508-050 Fax: +49-6151-8508-111 Mail: ronny.lippold@spark5.de Web: https://www.spark5.de -- Geschäftsführer: Henning Munte, Michael Mylius Amtsgericht Darmstadt, HRB 7809 -- Am 2024-02-14 06:59, schrieb Vladimir Sigunov:
Hi Ronny, This is a good starting point for your design. https://docs.ceph.com/en/latest/rados/operations/stretch-mode/
My personal experience says that 2 DC Ceph deployment could suffer from a 'split brain' situation. If you have any chance to create a 3 DC configuration, I would suggest to consider it. It could be more expensive, but it definitely will be more reliable and fault tolerant.
Sincerely, Vladimir
Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: ronny.lippold@spark5.de <ronny.lippold@spark5.de> Sent: Tuesday, February 13, 2024 6:50:50 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] concept of ceph and 2 datacenters
hi there, i have a design/concept question, to see, whats outside and which kind of redundancy you use.
actually, we use 2 ceph cluster with rbd-mirror to have an cold-standby clone. but, rbd mirror is not application consistend. so we cannot be sure, that all vms (kvm/proxy) are running. we also waste a lot of hardware.
so now, we think about one big cluster over the two datacenters (two buildings).
my queston is, do you care about ceph redundancy or is one ceph with backups enough for you? i know, with ceph, we are aware of hdd or server failure. but, are software failures a real scenario?
would be great, to get some ideas from you. also about the bandwidth between the 2 datacenters. we are using 2x 6 proxmox server with 2x6x9 osd (sas ssd).
thanks for help, my minds are rotating.
kind regards, ronny
-- Ronny Lippold System Administrator
-- Spark 5 GmbH Rheinstr. 97 64295 Darmstadt Germany -- Fon: +49-6151-8508-050 Fax: +49-6151-8508-111 Mail: ronny.lippold@spark5.de Web: https://www.spark5.de -- Geschäftsführer: Henning Munte, Michael Mylius Amtsgericht Darmstadt, HRB 7809 -- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Peter Sabaini
-
ronny.lippold＠spark5.de
-
Vladimir Sigunov