Hi, We have all SSD disks as ceph's backend storage. Consider the cost factor, can we setup the cluster to have only two replicas for objects? thanks & regards Wesley
So can you do it. Yes you can. Should you do it is the bigger question. So my first question would be what type of drives are you using? Enterprise class drives with a low failure rate? Then you have to ask yourself are you feeling lucky? If you do a scrub and 1 drive returns 1 value and another drive returns another value which one is correct? What happens should you have a drive failure and you have any other error? A node failure? Another disk failure? A disk read error? All of these could mean data loss. How important is the data you are storing and do you have a backup of it as you will need that backup at some point. Darren Sent from my iPhone
On 24 Aug 2019, at 14:01, Wesley Peng <weslepeng@gmail.com> wrote:
Hi,
We have all SSD disks as ceph's backend storage. Consider the cost factor, can we setup the cluster to have only two replicas for objects?
thanks & regards Wesley _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Op 24 aug. 2019 om 16:36 heeft Darren Soothill <darren.soothill@suse.com> het volgende geschreven:
So can you do it.
Yes you can.
Should you do it is the bigger question.
So my first question would be what type of drives are you using? Enterprise class drives with a low failure rate?
Doesn’t matter. From my experience: With 2x replication you will loose data at some point. As a consultant I have just seen too many cases of data loss with 2x. Please, don’t do it.
Then you have to ask yourself are you feeling lucky?
If you do a scrub and 1 drive returns 1 value and another drive returns another value which one is correct?
What happens should you have a drive failure and you have any other error? A node failure? Another disk failure? A disk read error? All of these could mean data loss.
How important is the data you are storing and do you have a backup of it as you will need that backup at some point.
Darren
Sent from my iPhone
On 24 Aug 2019, at 14:01, Wesley Peng <weslepeng@gmail.com> wrote:
Hi,
We have all SSD disks as ceph's backend storage. Consider the cost factor, can we setup the cluster to have only two replicas for objects?
thanks & regards Wesley _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ok thanks. Wido den Hollander <wido@42on.com>于2019年8月25日 周日上午4:47写道：
Op 24 aug. 2019 om 16:36 heeft Darren Soothill <darren.soothill@suse.com> het volgende geschreven:
So can you do it.
Yes you can.
Should you do it is the bigger question.
So my first question would be what type of drives are you using? Enterprise class drives with a low failure rate?
Doesn’t matter. From my experience: With 2x replication you will loose data at some point.
As a consultant I have just seen too many cases of data loss with 2x.
Please, don’t do it.
Then you have to ask yourself are you feeling lucky?
If you do a scrub and 1 drive returns 1 value and another drive returns another value which one is correct?
What happens should you have a drive failure and you have any other error? A node failure? Another disk failure? A disk read error? All of these could mean data loss.
How important is the data you are storing and do you have a backup of it as you will need that backup at some point.
Darren
Sent from my iPhone
On 24 Aug 2019, at 14:01, Wesley Peng <weslepeng@gmail.com> wrote:
Hi,
We have all SSD disks as ceph's backend storage. Consider the cost factor, can we setup the cluster to have only two replicas for objects?
thanks & regards Wesley _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If you have decent CPU and RAM on the OSD nodes, you can try Erasure Coding, even just 4:2 should keep the cost per GB/TB lower than 2:1 replica (as that's basically 1.5:1 for cost) and much safer (same protection as 3:1 replica). We use that on our biggest production SSD pool. ________________________________ From: Wesley Peng <weslepeng@gmail.com> Sent: Sunday, 25 August 2019 9:11 PM To: Wido den Hollander <wido@42on.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: ceph's replicas question Ok thanks. Wido den Hollander <wido@42on.com<mailto:wido@42on.com>>于2019年8月25日 周日上午4:47写道：
Op 24 aug. 2019 om 16:36 heeft Darren Soothill <darren.soothill@suse.com<mailto:darren.soothill@suse.com>> het volgende geschreven:
So can you do it.
Yes you can.
Should you do it is the bigger question.
So my first question would be what type of drives are you using? Enterprise class drives with a low failure rate?
Doesn’t matter. From my experience: With 2x replication you will loose data at some point. As a consultant I have just seen too many cases of data loss with 2x. Please, don’t do it.
Then you have to ask yourself are you feeling lucky?
If you do a scrub and 1 drive returns 1 value and another drive returns another value which one is correct?
What happens should you have a drive failure and you have any other error? A node failure? Another disk failure? A disk read error? All of these could mean data loss.
How important is the data you are storing and do you have a backup of it as you will need that backup at some point.
Darren
Sent from my iPhone
On 24 Aug 2019, at 14:01, Wesley Peng <weslepeng@gmail.com<mailto:weslepeng@gmail.com>> wrote:
Hi,
We have all SSD disks as ceph's backend storage. Consider the cost factor, can we setup the cluster to have only two replicas for objects?
thanks & regards Wesley _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hello! Sat, Aug 24, 2019 at 10:47:55PM +0200, wido wrote:
Op 24 aug. 2019 om 16:36 heeft Darren Soothill <darren.soothill@suse.com> het volgende geschreven:
So can you do it.
Yes you can.
Should you do it is the bigger question.
So my first question would be what type of drives are you using? Enterprise class drives with a low failure rate?
Doesn’t matter. From my experience: With 2x replication you will loose data at some point.
As a consultant I have just seen too many cases of data loss with 2x.
Please, don’t do it.
I'd like to ask: do the new features like checksumming fix this problem?
Op 27 aug. 2019 om 11:38 heeft Max Krasilnikov <pseudo@avalon.org.ua> het volgende geschreven:
Hello!
Sat, Aug 24, 2019 at 10:47:55PM +0200, wido wrote:
Op 24 aug. 2019 om 16:36 heeft Darren Soothill <darren.soothill@suse.com> het volgende geschreven:
So can you do it.
Yes you can.
Should you do it is the bigger question.
So my first question would be what type of drives are you using? Enterprise class drives with a low failure rate?
Doesn’t matter. From my experience: With 2x replication you will loose data at some point.
As a consultant I have just seen too many cases of data loss with 2x.
Please, don’t do it.
I'd like to ask: do the new features like checksumming fix this problem?
No, they don’t. Imagine you take a machine down for maintenance and while that machine is down you loose a disk. You lost data. Although you can bring back the other machine you are still missing transactions. This is not fictional, this happens. Same as that during the backfill of a disk you just lost the disk with the remaining copy dies under the load of the backfills. I have seen this happen too many times in the last years. Don’t try to cut corners. Wido
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Aug 27, 2019 at 12:26 PM Wido den Hollander <wido@42on.com> wrote:
Op 27 aug. 2019 om 11:38 heeft Max Krasilnikov <pseudo@avalon.org.ua> het volgende geschreven:
Hello!
Sat, Aug 24, 2019 at 10:47:55PM +0200, wido wrote:
Op 24 aug. 2019 om 16:36 heeft Darren Soothill <darren.soothill@suse.com> het volgende geschreven:
So can you do it.
Yes you can.
Should you do it is the bigger question.
So my first question would be what type of drives are you using? Enterprise class drives with a low failure rate?
Doesn’t matter. From my experience: With 2x replication you will loose data at some point.
As a consultant I have just seen too many cases of data loss with 2x.
Please, don’t do it.
I'd like to ask: do the new features like checksumming fix this problem?
No, they don’t.
Imagine you take a machine down for maintenance and while that machine is down you loose a disk.
You lost data. Although you can bring back the other machine you are still missing transactions.
This is not fictional, this happens.
100% agree, this happens *all the time* with min_size 1. If you really care about your data then 2/1 just doesn't cut it. Paul
Same as that during the backfill of a disk you just lost the disk with the remaining copy dies under the load of the backfills.
I have seen this happen too many times in the last years.
Don’t try to cut corners.
Wido
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
On 27. Aug 2019, at 14:43, Paul Emmerich <paul.emmerich@croit.io> wrote:
100% agree, this happens *all the time* with min_size 1.
If you really care about your data then 2/1 just doesn't cut it.
Just to make this more specific and less fictional: a very easy way to trigger this is by shutting down your whole cluster and starting it up again, including your network equipment. It’s normal that this is a period where cluster activity is quite flaky and this has caused multiple instances of data loss for us when we had clusters with min_size 1. Cheers, Christian -- Christian Theune · ct@flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · http://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
participants (7)
-
Christian Theune
-
Darren Soothill
-
Linh Vu
-
Max Krasilnikov
-
Paul Emmerich
-
Wesley Peng
-
Wido den Hollander