Adam;
Earlier this week, another thread presented 3 white papers in support of running 2x on
NVMe for Ceph.
I searched each to find the section where 2x was discussed. What I found was interesting.
First, there are really only 2 positions here: Micron's and Red Hat's.
Supermicro copies Micron's positon paragraph word for word. Not surprising
considering that they are advertising a Supermicro / Micron solution.
This is Micron's statement:
" NVMe SSDs have high reliability with high MTBR and low bit error rate. 2x
replication is recommended in production when deploying OSDs on NVMe versus the 3x
replication common with legacy storage."
This is Red Hat's statement:
" Given the better MTBF and MTTR of flash-based media, many Ceph customers have
chosen to run 2x replications in
production when deploying OSDs on flash. This differs from magnetic media deployments,
which typically use 3x replication."
Looking at these statements, these acronyms pop out at me: MTBR and MTTR. MTBR is Mean
Time Between Replacements, while MTTR is Mean Time Till Replacement. Essentially; this is
saying that most companies replaces these drives before they have to worry about large
numbers failing.
Regarding MTBF; I can't find any data to support Red Hat's assertion that MTBF is
better for flash. I looked at both Western Digital Gold, and Seagate Exos 12 TB drives,
and found they both list a MTBF of 2.5 million hours. I was unable to find any
information on the MTBF of Micron drives, but the MTBF of Kingston's DC1000B 240GB
drive is 2 million hours.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because
of the cost difference they have to convince buyers that their products are competitive.
Pitch is thus:
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these
other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway,
then you won't have to worry about them failing.
I'll address general concerns of 2x replication in another email.
Thank you,
Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Adam Boyhan [mailto:adamb@medent.com]
Sent: Thursday, February 4, 2021 4:38 AM
To: ceph-users
Subject: [ceph-users] NVMe and 2x Replica
I know there is already a few threads about 2x replication but I wanted to start one
dedicated to discussion on NVMe. There are some older threads, but nothing recent that
addresses how the vendors are now pushing the idea of 2x.
We are in the process of considering Ceph to replace our Nimble setup. We will have two
completely separate clusters at two different sites that we are using rbd-mirror snapshot
replication. The plan would be to run 2x replication on each cluster. 3x is still an
option, but for obvious reasons 2x is enticing.
Both clusters will be spot on to the super micro example in the white paper below.
It seems all the big vendors feel 2x is safe with NVMe but I get the feeling this
community feels otherwise. Trying to wrap my head around were the disconnect is between
the big players and the community. I could be missing something, but even our Supermicro
contact that we worked the config out with was in agreement with 2x on NVMe.
Appreciate the input!
[
https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf |
https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf ]
[
https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referen…
]
[
https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referen…
|
https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referen…
]
[
https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/red…
|
https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/red…
]
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io