I know there is already a few threads about 2x replication but I wanted to start one dedicated to discussion on NVMe. There are some older threads, but nothing recent that addresses how the vendors are now pushing the idea of 2x. We are in the process of considering Ceph to replace our Nimble setup. We will have two completely separate clusters at two different sites that we are using rbd-mirror snapshot replication. The plan would be to run 2x replication on each cluster. 3x is still an option, but for obvious reasons 2x is enticing. Both clusters will be spot on to the super micro example in the white paper below. It seems all the big vendors feel 2x is safe with NVMe but I get the feeling this community feels otherwise. Trying to wrap my head around were the disconnect is between the big players and the community. I could be missing something, but even our Supermicro contact that we worked the config out with was in agreement with 2x on NVMe. Appreciate the input! [ https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf | https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... | https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... ] [ https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redh... | https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redh... ]
Adam; Earlier this week, another thread presented 3 white papers in support of running 2x on NVMe for Ceph. I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution. This is Micron's statement: " NVMe SSDs have high reliability with high MTBR and low bit error rate. 2x replication is recommended in production when deploying OSDs on NVMe versus the 3x replication common with legacy storage." This is Red Hat's statement: " Given the better MTBF and MTTR of flash-based media, many Ceph customers have chosen to run 2x replications in production when deploying OSDs on flash. This differs from magnetic media deployments, which typically use 3x replication." Looking at these statements, these acronyms pop out at me: MTBR and MTTR. MTBR is Mean Time Between Replacements, while MTTR is Mean Time Till Replacement. Essentially; this is saying that most companies replaces these drives before they have to worry about large numbers failing. Regarding MTBF; I can't find any data to support Red Hat's assertion that MTBF is better for flash. I looked at both Western Digital Gold, and Seagate Exos 12 TB drives, and found they both list a MTBF of 2.5 million hours. I was unable to find any information on the MTBF of Micron drives, but the MTBF of Kingston's DC1000B 240GB drive is 2 million hours. Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive. Pitch is thus: Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing. I'll address general concerns of 2x replication in another email. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Adam Boyhan [mailto:adamb@medent.com] Sent: Thursday, February 4, 2021 4:38 AM To: ceph-users Subject: [ceph-users] NVMe and 2x Replica I know there is already a few threads about 2x replication but I wanted to start one dedicated to discussion on NVMe. There are some older threads, but nothing recent that addresses how the vendors are now pushing the idea of 2x. We are in the process of considering Ceph to replace our Nimble setup. We will have two completely separate clusters at two different sites that we are using rbd-mirror snapshot replication. The plan would be to run 2x replication on each cluster. 3x is still an option, but for obvious reasons 2x is enticing. Both clusters will be spot on to the super micro example in the white paper below. It seems all the big vendors feel 2x is safe with NVMe but I get the feeling this community feels otherwise. Trying to wrap my head around were the disconnect is between the big players and the community. I could be missing something, but even our Supermicro contact that we worked the config out with was in agreement with 2x on NVMe. Appreciate the input! [ https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf | https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... | https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... ] [ https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redh... | https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redh... ] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk. * Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen. I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph. My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
All great input and points guys. Helps me lean towards 3 copes a bit more. I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money. From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk. * Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen. I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph. My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
My impression is that cost / TB for a drive may be approaching parity, but the TB /drive is still well below (or at least at densities approaching parity, cost / TB is still quite high). I can get a Micron 15TB SSD for $2600, but why would I when I can get a 18TB Seagate IronWolf for <$600, a 18TB Seagate Exos for <$500, or a 18TB WD Gold for <$600? Personally I wouldn't use drives that big, in our little tiny clusters, but it exemplifies the issues around discussing cost parity. As such a cluster needs more dives for the same total size (thus more nodes), which drives up the cost / TB for a cluster. My 2 cents. Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Adam Boyhan [mailto:adamb@medent.com] Sent: Thursday, February 4, 2021 10:58 AM To: Anthony D'Atri Cc: ceph-users Subject: [ceph-users] Re: NVMe and 2x Replica All great input and points guys. Helps me lean towards 3 copes a bit more. I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money. From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk. * Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen. I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph. My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Why would I when I can get a 18TB Seagate IronWolf for <$600, a 18TB Seagate Exos for <$500, or a 18TB WD Gold for <$600?
IOPS
Some installations don’t care so much about IOPS. Less-tangible factors include: * Time to repair and thus to restore redundancy. When an EC pool of spinners takes a *month* to weight up a drive, that’s a significant operational and data durability / availability concern. * RMAs. They’re a pain, especially if you have to work them through a chassis vendor, who likely will be dilatory and demand unreasonable hoops like attaching a BMC web interface screenshot for every drive. This translates to each RMA being modeled with a certain shipping / person-hour cost, which means that for lower unit-value items it may not be worth the hassle. It is not unreasonable to guesstimate a threshold around USD 500. Soit is not uncommon to just trash failed / DOA spinners — or letting them stack up indefinitely in a corner — instead of recovering their value. As I wrote … in 2019 I think it was, with spinners you have some manner of HBA in the mix. If that HBA is a fussy RAID model, you may have significant added cost for the RoC, onboard RAM, and supercap/BBU. Complexity also comes with neverending firmware bugs and cache management nightmares. Gas gauge firmware… don’t even get me talking about that. And consider how many TB of 3.5” spinners you fit into an RU, compared to 2.5” or EDSFF flash. RUs aren’t free, and SATA HBAs will bottleneck a relatively dense HDD chassis long before a similar number of NVMe drives will bottleneck. Unless perhaps you have the misfortune of a chassis manufacturer who for some reason runs NVMe PCI lanes *though* an HBA.
Taking a month to weight up a drive suggests the cluster doesn't have enough spare IO capacity. And for everyone suggesting EC, I don't understand how anyone really thinks that's a valid alternative with the min allocation / space amplification bug, no one in this community, not even the top developers of the project can provide an accurate space projection for EC usage -- and if you cannot predict how much space an EC configuration will use in the wild because of a bug that isn't well documented nor discussed, you cannot begin to even talk about costs and numbers. I understand '4k' block sizes may fix this issue but not everyone can necessarily run ceph off the latest master. There are a lot of hidden costs in using ceph which can vary depending on usage needs, such as having spare io for recovery operations or ensuring your total cluster disk usage stays below 60%. On Thu, Feb 4, 2021 at 2:25 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Why would I when I can get a 18TB Seagate IronWolf for <$600, a 18TB Seagate Exos for <$500, or a 18TB WD Gold for <$600?
IOPS
Some installations don’t care so much about IOPS.
Less-tangible factors include:
* Time to repair and thus to restore redundancy. When an EC pool of spinners takes a *month* to weight up a drive, that’s a significant operational and data durability / availability concern.
* RMAs. They’re a pain, especially if you have to work them through a chassis vendor, who likely will be dilatory and demand unreasonable hoops like attaching a BMC web interface screenshot for every drive. This translates to each RMA being modeled with a certain shipping / person-hour cost, which means that for lower unit-value items it may not be worth the hassle. It is not unreasonable to guesstimate a threshold around USD 500. Soit is not uncommon to just trash failed / DOA spinners — or letting them stack up indefinitely in a corner — instead of recovering their value.
As I wrote … in 2019 I think it was, with spinners you have some manner of HBA in the mix. If that HBA is a fussy RAID model, you may have significant added cost for the RoC, onboard RAM, and supercap/BBU. Complexity also comes with neverending firmware bugs and cache management nightmares. Gas gauge firmware… don’t even get me talking about that.
And consider how many TB of 3.5” spinners you fit into an RU, compared to 2.5” or EDSFF flash. RUs aren’t free, and SATA HBAs will bottleneck a relatively dense HDD chassis long before a similar number of NVMe drives will bottleneck. Unless perhaps you have the misfortune of a chassis manufacturer who for some reason runs NVMe PCI lanes *though* an HBA.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Steven Pine *E * steven.pine@webair.com | *P * 516.938.4100 x *Webair* | 501 Franklin Avenue Suite 200, Garden City NY, 11530 webair.com [image: Facebook icon] <https://www.facebook.com/WebairInc/> [image: Twitter icon] <https://twitter.com/WebairInc> [image: Linkedin icon] <https://www.linkedin.com/company/webair> NOTICE: This electronic mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged proprietary and confidential information. If the reader of this message is not the intended recipient, or if you are an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and delete it from your computer.
Sorry to jump in here, but would you care to explain why the total disk usage should stay under 60%? This is not something I have heard before and a quick Google search didn't return anything useful. Steven Pine wrote on 04.02.21 20:41:
There are a lot of hidden costs in using ceph which can vary depending on usage needs, such as having spare io for recovery operations or ensuring your total cluster disk usage stays below 60%.
Certainly with a small number of nodes / osd this makes sense as to lose a node could quickly make the cluster storage capacity be full very quickly. On Friday, February 5, 2021, Pascal Ehlert <pascal@hacksrus.net> wrote:
Sorry to jump in here, but would you care to explain why the total disk usage should stay under 60%? This is not something I have heard before and a quick Google search didn't return anything useful.
Steven Pine wrote on 04.02.21 20:41:
There are a lot of hidden costs in using ceph which can vary depending on usage needs, such as having spare io for recovery operations or ensuring your total cluster disk usage stays below 60%.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den fre 5 feb. 2021 kl 07:38 skrev Pascal Ehlert <pascal@hacksrus.net>:
Sorry to jump in here, but would you care to explain why the total disk usage should stay under 60%? This is not something I have heard before and a quick Google search didn't return anything useful.
If you have 3 hosts with 3 drives each and repl=3, then each replica will (at least by default) want to end up on a separate host. If one disk dies, ALL the pgs will want to end up on the last drive of that host, and if it doesn't fit, the recovery will stall. (the cluster will serve IO but in a degraded mode) Same for having 4 hosts with X drives, and if one host dies, all the replicas on that host needs to fit into the 3 remaining hosts, while still ending up below 85-90% when backfill_toofull starts occurring. So if you are to survive planned or unplanned downtime on any single host while the cluster doesn't get full you need extra room, plus the 10-15% margin ceph wants. Of course it will differ a bit if you have 100s of hosts or 50 drives in a host or something extreme like that, but you should have certain levels so that by 50% full you start planning expansion, by 60+% full you execute the plan so that it can get installed before becoming 70% full or something to that effect. Ceph is somewhat like an old raid6 box with a slot or two for hotspares, except you actively use the hotspares because-why-not? So you would still need to have as much space free in your ceph as the hotspares would hold but you use the extra IOPS the hotspares bring in the mean time. -- May the most significant bit of your life be positive.
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money.
To add to this, I have seen real cases as a Ceph consultant where size=2 and min_size=1 on all flash lead to data loss. Picture this: - One node is down (Maintenance, failure, etc, etc) - NVMe device in other node dies - You loose data Although you can bring back the other node which was down but not broken you are missing data. The data on the NVMe devices in there is outdated and thus the PGs will not become active. size=2 is only safe with min_size=2, but that doesn't really provide HA. The same goes with ZFS in mirror, raidz1, etc. If you loose one device the chances are real you loose the other device before the array has healed itself. With Ceph it's slighly more complex, but the same principles apply. No, with NVMe I still would highly advise against using size=2, min_size=1 The question is not if you will loose data, but the question is: When will you loose data? Within one year, 2? 3? 10? Wido
From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk.
* Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen.
I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph.
My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
At the end, this is nothing but a probability stuff Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ? What are the numbers ? On 2/5/21 12:26 PM, Wido den Hollander wrote:
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money.
To add to this, I have seen real cases as a Ceph consultant where size=2 and min_size=1 on all flash lead to data loss.
Picture this:
- One node is down (Maintenance, failure, etc, etc) - NVMe device in other node dies - You loose data
Although you can bring back the other node which was down but not broken you are missing data. The data on the NVMe devices in there is outdated and thus the PGs will not become active.
size=2 is only safe with min_size=2, but that doesn't really provide HA.
The same goes with ZFS in mirror, raidz1, etc. If you loose one device the chances are real you loose the other device before the array has healed itself.
With Ceph it's slighly more complex, but the same principles apply.
No, with NVMe I still would highly advise against using size=2, min_size=1
The question is not if you will loose data, but the question is: When will you loose data? Within one year, 2? 3? 10?
Wido
From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk.
* Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen.
I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph.
My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data
Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ?
If you just look at it from this narrow point of view of fundamental laws of nature, then, yes, 2+1 is safe. As safe as is nuclear power just looking at the laws of physics. So why then did Chernobyl and Fukushima happen? Its because its operated by humans. If you look around, the No. 1 reason for loosing data on ceph or entire clusters is 2+1. Look at the reasons. Its rarely a broken disk. A system designed with no redundancy that offers a margin for error will suffer from every little admin mistake, undetected race condition, bug in ceph or bug in firmware. So, if the savings are worth the sweat, downtime and consultancy budget, why not? Ceph has infinite uptime. During such a long period, low-probability events will happen with probability 1. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Jack <ceph@jack.fr.eu.org> Sent: 05 February 2021 12:48:33 To: ceph-users@ceph.io Subject: [ceph-users] Re: NVMe and 2x Replica At the end, this is nothing but a probability stuff Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ? What are the numbers ? On 2/5/21 12:26 PM, Wido den Hollander wrote:
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money.
To add to this, I have seen real cases as a Ceph consultant where size=2 and min_size=1 on all flash lead to data loss.
Picture this:
- One node is down (Maintenance, failure, etc, etc) - NVMe device in other node dies - You loose data
Although you can bring back the other node which was down but not broken you are missing data. The data on the NVMe devices in there is outdated and thus the PGs will not become active.
size=2 is only safe with min_size=2, but that doesn't really provide HA.
The same goes with ZFS in mirror, raidz1, etc. If you loose one device the chances are real you loose the other device before the array has healed itself.
With Ceph it's slighly more complex, but the same principles apply.
No, with NVMe I still would highly advise against using size=2, min_size=1
The question is not if you will loose data, but the question is: When will you loose data? Within one year, 2? 3? 10?
Wido
From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk.
* Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen.
I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph.
My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This turned into a great thread. Lots of good information and clarification. I am 100% on board with 3 copies for the primary. What does everyone think about possibly only doing 2 copies on the secondary? Keeping in mind that I would keep min=2 which I think will be reasonable for a secondary site. From: "Frank Schilder" <frans@dtu.dk> To: "Jack" <ceph@jack.fr.eu.org>, "ceph-users" <ceph-users@ceph.io> Sent: Friday, February 5, 2021 7:14:52 AM Subject: [ceph-users] Re: NVMe and 2x Replica
Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data
Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ?
If you just look at it from this narrow point of view of fundamental laws of nature, then, yes, 2+1 is safe. As safe as is nuclear power just looking at the laws of physics. So why then did Chernobyl and Fukushima happen? Its because its operated by humans. If you look around, the No. 1 reason for loosing data on ceph or entire clusters is 2+1. Look at the reasons. Its rarely a broken disk. A system designed with no redundancy that offers a margin for error will suffer from every little admin mistake, undetected race condition, bug in ceph or bug in firmware. So, if the savings are worth the sweat, downtime and consultancy budget, why not? Ceph has infinite uptime. During such a long period, low-probability events will happen with probability 1. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Jack <ceph@jack.fr.eu.org> Sent: 05 February 2021 12:48:33 To: ceph-users@ceph.io Subject: [ceph-users] Re: NVMe and 2x Replica At the end, this is nothing but a probability stuff Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ? What are the numbers ? On 2/5/21 12:26 PM, Wido den Hollander wrote:
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money.
To add to this, I have seen real cases as a Ceph consultant where size=2 and min_size=1 on all flash lead to data loss.
Picture this:
- One node is down (Maintenance, failure, etc, etc) - NVMe device in other node dies - You loose data
Although you can bring back the other node which was down but not broken you are missing data. The data on the NVMe devices in there is outdated and thus the PGs will not become active.
size=2 is only safe with min_size=2, but that doesn't really provide HA.
The same goes with ZFS in mirror, raidz1, etc. If you loose one device the chances are real you loose the other device before the array has healed itself.
With Ceph it's slighly more complex, but the same principles apply.
No, with NVMe I still would highly advise against using size=2, min_size=1
The question is not if you will loose data, but the question is: When will you loose data? Within one year, 2? 3? 10?
Wido
From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk.
* Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen.
I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph.
My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I don't run a secondary site and don't know if short windows of read-only access are terrible. From the data security point of view, min_size 2 is fine. Its the min_size 1 that really is dangerous, because it accepts non-redundant writes. Even if you loose the second site entirely, you can always re-sync from scratch - assuming decent network bandwidth. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Adam Boyhan <adamb@medent.com> Sent: 05 February 2021 13:58:34 To: Frank Schilder Cc: Jack; ceph-users Subject: Re: [ceph-users] Re: NVMe and 2x Replica This turned into a great thread. Lots of good information and clarification. I am 100% on board with 3 copies for the primary. What does everyone think about possibly only doing 2 copies on the secondary? Keeping in mind that I would keep min=2 which I think will be reasonable for a secondary site. ________________________________ From: "Frank Schilder" <frans@dtu.dk> To: "Jack" <ceph@jack.fr.eu.org>, "ceph-users" <ceph-users@ceph.io> Sent: Friday, February 5, 2021 7:14:52 AM Subject: [ceph-users] Re: NVMe and 2x Replica
Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data
Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ?
If you just look at it from this narrow point of view of fundamental laws of nature, then, yes, 2+1 is safe. As safe as is nuclear power just looking at the laws of physics. So why then did Chernobyl and Fukushima happen? Its because its operated by humans. If you look around, the No. 1 reason for loosing data on ceph or entire clusters is 2+1. Look at the reasons. Its rarely a broken disk. A system designed with no redundancy that offers a margin for error will suffer from every little admin mistake, undetected race condition, bug in ceph or bug in firmware. So, if the savings are worth the sweat, downtime and consultancy budget, why not? Ceph has infinite uptime. During such a long period, low-probability events will happen with probability 1. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Jack <ceph@jack.fr.eu.org> Sent: 05 February 2021 12:48:33 To: ceph-users@ceph.io Subject: [ceph-users] Re: NVMe and 2x Replica At the end, this is nothing but a probability stuff Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ? What are the numbers ? On 2/5/21 12:26 PM, Wido den Hollander wrote:
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money.
To add to this, I have seen real cases as a Ceph consultant where size=2 and min_size=1 on all flash lead to data loss.
Picture this:
- One node is down (Maintenance, failure, etc, etc) - NVMe device in other node dies - You loose data
Although you can bring back the other node which was down but not broken you are missing data. The data on the NVMe devices in there is outdated and thus the PGs will not become active.
size=2 is only safe with min_size=2, but that doesn't really provide HA.
The same goes with ZFS in mirror, raidz1, etc. If you loose one device the chances are real you loose the other device before the array has healed itself.
With Ceph it's slighly more complex, but the same principles apply.
No, with NVMe I still would highly advise against using size=2, min_size=1
The question is not if you will loose data, but the question is: When will you loose data? Within one year, 2? 3? 10?
Wido
From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk.
* Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen.
I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph.
My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Those are my thoughts as well. We have 40Gbit/s of dedicated dark fiber that we manage between the two sites. From: "Frank Schilder" <frans@dtu.dk> To: "adamb" <adamb@medent.com> Cc: "Jack" <ceph@jack.fr.eu.org>, "ceph-users" <ceph-users@ceph.io> Sent: Friday, February 5, 2021 10:19:06 AM Subject: Re: [ceph-users] Re: NVMe and 2x Replica I don't run a secondary site and don't know if short windows of read-only access are terrible. From the data security point of view, min_size 2 is fine. Its the min_size 1 that really is dangerous, because it accepts non-redundant writes. Even if you loose the second site entirely, you can always re-sync from scratch - assuming decent network bandwidth. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Adam Boyhan <adamb@medent.com> Sent: 05 February 2021 13:58:34 To: Frank Schilder Cc: Jack; ceph-users Subject: Re: [ceph-users] Re: NVMe and 2x Replica This turned into a great thread. Lots of good information and clarification. I am 100% on board with 3 copies for the primary. What does everyone think about possibly only doing 2 copies on the secondary? Keeping in mind that I would keep min=2 which I think will be reasonable for a secondary site. ________________________________ From: "Frank Schilder" <frans@dtu.dk> To: "Jack" <ceph@jack.fr.eu.org>, "ceph-users" <ceph-users@ceph.io> Sent: Friday, February 5, 2021 7:14:52 AM Subject: [ceph-users] Re: NVMe and 2x Replica
Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data
Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ?
If you just look at it from this narrow point of view of fundamental laws of nature, then, yes, 2+1 is safe. As safe as is nuclear power just looking at the laws of physics. So why then did Chernobyl and Fukushima happen? Its because its operated by humans. If you look around, the No. 1 reason for loosing data on ceph or entire clusters is 2+1. Look at the reasons. Its rarely a broken disk. A system designed with no redundancy that offers a margin for error will suffer from every little admin mistake, undetected race condition, bug in ceph or bug in firmware. So, if the savings are worth the sweat, downtime and consultancy budget, why not? Ceph has infinite uptime. During such a long period, low-probability events will happen with probability 1. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Jack <ceph@jack.fr.eu.org> Sent: 05 February 2021 12:48:33 To: ceph-users@ceph.io Subject: [ceph-users] Re: NVMe and 2x Replica At the end, this is nothing but a probability stuff Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ? What are the numbers ? On 2/5/21 12:26 PM, Wido den Hollander wrote:
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money.
To add to this, I have seen real cases as a Ceph consultant where size=2 and min_size=1 on all flash lead to data loss.
Picture this:
- One node is down (Maintenance, failure, etc, etc) - NVMe device in other node dies - You loose data
Although you can bring back the other node which was down but not broken you are missing data. The data on the NVMe devices in there is outdated and thus the PGs will not become active.
size=2 is only safe with min_size=2, but that doesn't really provide HA.
The same goes with ZFS in mirror, raidz1, etc. If you loose one device the chances are real you loose the other device before the array has healed itself.
With Ceph it's slighly more complex, but the same principles apply.
No, with NVMe I still would highly advise against using size=2, min_size=1
The question is not if you will loose data, but the question is: When will you loose data? Within one year, 2? 3? 10?
Wido
From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk.
* Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen.
I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph.
My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is raid1 dangerous ? Is raid5 dangerous ? They both allow non-redondant writes On 2/5/21 4:19 PM, Frank Schilder wrote:
I don't run a secondary site and don't know if short windows of read-only access are terrible. From the data security point of view, min_size 2 is fine. Its the min_size 1 that really is dangerous, because it accepts non-redundant writes.
Even if you loose the second site entirely, you can always re-sync from scratch - assuming decent network bandwidth.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Adam Boyhan <adamb@medent.com> Sent: 05 February 2021 13:58:34 To: Frank Schilder Cc: Jack; ceph-users Subject: Re: [ceph-users] Re: NVMe and 2x Replica
This turned into a great thread. Lots of good information and clarification.
I am 100% on board with 3 copies for the primary.
What does everyone think about possibly only doing 2 copies on the secondary? Keeping in mind that I would keep min=2 which I think will be reasonable for a secondary site.
________________________________ From: "Frank Schilder" <frans@dtu.dk> To: "Jack" <ceph@jack.fr.eu.org>, "ceph-users" <ceph-users@ceph.io> Sent: Friday, February 5, 2021 7:14:52 AM Subject: [ceph-users] Re: NVMe and 2x Replica
Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data
Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ?
If you just look at it from this narrow point of view of fundamental laws of nature, then, yes, 2+1 is safe. As safe as is nuclear power just looking at the laws of physics. So why then did Chernobyl and Fukushima happen? Its because its operated by humans. If you look around, the No. 1 reason for loosing data on ceph or entire clusters is 2+1.
Look at the reasons. Its rarely a broken disk. A system designed with no redundancy that offers a margin for error will suffer from every little admin mistake, undetected race condition, bug in ceph or bug in firmware. So, if the savings are worth the sweat, downtime and consultancy budget, why not?
Ceph has infinite uptime. During such a long period, low-probability events will happen with probability 1.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Jack <ceph@jack.fr.eu.org> Sent: 05 February 2021 12:48:33 To: ceph-users@ceph.io Subject: [ceph-users] Re: NVMe and 2x Replica
At the end, this is nothing but a probability stuff
Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data
Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ?
What are the numbers ?
On 2/5/21 12:26 PM, Wido den Hollander wrote:
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money.
To add to this, I have seen real cases as a Ceph consultant where size=2 and min_size=1 on all flash lead to data loss.
Picture this:
- One node is down (Maintenance, failure, etc, etc) - NVMe device in other node dies - You loose data
Although you can bring back the other node which was down but not broken you are missing data. The data on the NVMe devices in there is outdated and thus the PGs will not become active.
size=2 is only safe with min_size=2, but that doesn't really provide HA.
The same goes with ZFS in mirror, raidz1, etc. If you loose one device the chances are real you loose the other device before the array has healed itself.
With Ceph it's slighly more complex, but the same principles apply.
No, with NVMe I still would highly advise against using size=2, min_size=1
The question is not if you will loose data, but the question is: When will you loose data? Within one year, 2? 3? 10?
Wido
From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk.
* Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen.
I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph.
My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Analogies between a distributed system and one that isn’t can be a bit strained or nuanced. The question really isn’t IF a given solution is dangerous, but HOW dangerous it is. There is always a long tail ; one picks a point along it based on capex, business needs, etc. I sometimes read that RAID1 == R2, but I’ve always considered it RN. After I asked HP to support R3 with their HBAs they did, I like to think I had something to do with that but it may have been coincidence. For RAID5, look up “write hole”. min_size = 1 has occasional utility when *very temporarily* set to allow recovery from a bad situation(1), but as a permanent topology it’s Russian Roulette. Spin enough times and .... 1: These aren’t as common as they used to be.
On Feb 5, 2021, at 8:09 AM, Jack <ceph@jack.fr.eu.org> wrote:
Is raid1 dangerous ? Is raid5 dangerous ?
They both allow non-redondant writes
On 2/5/21 4:19 PM, Frank Schilder wrote: I don't run a secondary site and don't know if short windows of read-only access are terrible. From the data security point of view, min_size 2 is fine. Its the min_size 1 that really is dangerous, because it accepts non-redundant writes. Even if you loose the second site entirely, you can always re-sync from scratch - assuming decent network bandwidth. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Adam Boyhan <adamb@medent.com> Sent: 05 February 2021 13:58:34 To: Frank Schilder Cc: Jack; ceph-users Subject: Re: [ceph-users] Re: NVMe and 2x Replica This turned into a great thread. Lots of good information and clarification. I am 100% on board with 3 copies for the primary. What does everyone think about possibly only doing 2 copies on the secondary? Keeping in mind that I would keep min=2 which I think will be reasonable for a secondary site. ________________________________ From: "Frank Schilder" <frans@dtu.dk> To: "Jack" <ceph@jack.fr.eu.org>, "ceph-users" <ceph-users@ceph.io> Sent: Friday, February 5, 2021 7:14:52 AM Subject: [ceph-users] Re: NVMe and 2x Replica
Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data
Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ? If you just look at it from this narrow point of view of fundamental laws of nature, then, yes, 2+1 is safe. As safe as is nuclear power just looking at the laws of physics. So why then did Chernobyl and Fukushima happen? Its because its operated by humans. If you look around, the No. 1 reason for loosing data on ceph or entire clusters is 2+1. Look at the reasons. Its rarely a broken disk. A system designed with no redundancy that offers a margin for error will suffer from every little admin mistake, undetected race condition, bug in ceph or bug in firmware. So, if the savings are worth the sweat, downtime and consultancy budget, why not? Ceph has infinite uptime. During such a long period, low-probability events will happen with probability 1. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
From: Jack <ceph@jack.fr.eu.org> Sent: 05 February 2021 12:48:33 To: ceph-users@ceph.io Subject: [ceph-users] Re: NVMe and 2x Replica At the end, this is nothing but a probability stuff Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ? What are the numbers ?
On 2/5/21 12:26 PM, Wido den Hollander wrote:
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money.
To add to this, I have seen real cases as a Ceph consultant where size=2 and min_size=1 on all flash lead to data loss.
Picture this:
- One node is down (Maintenance, failure, etc, etc) - NVMe device in other node dies - You loose data
Although you can bring back the other node which was down but not broken you are missing data. The data on the NVMe devices in there is outdated and thus the PGs will not become active.
size=2 is only safe with min_size=2, but that doesn't really provide HA.
The same goes with ZFS in mirror, raidz1, etc. If you loose one device the chances are real you loose the other device before the array has healed itself.
With Ceph it's slighly more complex, but the same principles apply.
No, with NVMe I still would highly advise against using size=2, min_size=1
The question is not if you will loose data, but the question is: When will you loose data? Within one year, 2? 3? 10?
Wido
From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk.
* Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen.
I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph.
My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Why would you use RAID underneath Ceph? The only reason I've seen to do that is if you don't have enough CPU to run enough OSDs. On Fri, Feb 5, 2021 at 11:09 AM Jack <ceph@jack.fr.eu.org> wrote:
Is raid1 dangerous ? Is raid5 dangerous ?
They both allow non-redondant writes
On 2/5/21 4:19 PM, Frank Schilder wrote:
I don't run a secondary site and don't know if short windows of read-only access are terrible. From the data security point of view, min_size 2 is fine. Its the min_size 1 that really is dangerous, because it accepts non-redundant writes.
Even if you loose the second site entirely, you can always re-sync from scratch - assuming decent network bandwidth.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Adam Boyhan <adamb@medent.com> Sent: 05 February 2021 13:58:34 To: Frank Schilder Cc: Jack; ceph-users Subject: Re: [ceph-users] Re: NVMe and 2x Replica
This turned into a great thread. Lots of good information and clarification.
I am 100% on board with 3 copies for the primary.
What does everyone think about possibly only doing 2 copies on the secondary? Keeping in mind that I would keep min=2 which I think will be reasonable for a secondary site.
________________________________ From: "Frank Schilder" <frans@dtu.dk> To: "Jack" <ceph@jack.fr.eu.org>, "ceph-users" <ceph-users@ceph.io> Sent: Friday, February 5, 2021 7:14:52 AM Subject: [ceph-users] Re: NVMe and 2x Replica
Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data
Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ?
If you just look at it from this narrow point of view of fundamental laws of nature, then, yes, 2+1 is safe. As safe as is nuclear power just looking at the laws of physics. So why then did Chernobyl and Fukushima happen? Its because its operated by humans. If you look around, the No. 1 reason for loosing data on ceph or entire clusters is 2+1.
Look at the reasons. Its rarely a broken disk. A system designed with no redundancy that offers a margin for error will suffer from every little admin mistake, undetected race condition, bug in ceph or bug in firmware. So, if the savings are worth the sweat, downtime and consultancy budget, why not?
Ceph has infinite uptime. During such a long period, low-probability events will happen with probability 1.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Jack <ceph@jack.fr.eu.org> Sent: 05 February 2021 12:48:33 To: ceph-users@ceph.io Subject: [ceph-users] Re: NVMe and 2x Replica
At the end, this is nothing but a probability stuff
Picture this, using size=3, min_size=2: - One node is down for maintenance - You loose a couple of devices - You loose data
Is it likely that a nvme device dies during a short maintenance window ? Is it likely that two devices dies at the same time ?
What are the numbers ?
On 2/5/21 12:26 PM, Wido den Hollander wrote:
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now. Somewhat surprised the salesmen aren't pitching 3x replication as it makes them more money.
To add to this, I have seen real cases as a Ceph consultant where size=2 and min_size=1 on all flash lead to data loss.
Picture this:
- One node is down (Maintenance, failure, etc, etc) - NVMe device in other node dies - You loose data
Although you can bring back the other node which was down but not broken you are missing data. The data on the NVMe devices in there is outdated and thus the PGs will not become active.
size=2 is only safe with min_size=2, but that doesn't really provide HA.
The same goes with ZFS in mirror, raidz1, etc. If you loose one device the chances are real you loose the other device before the array has healed itself.
With Ceph it's slighly more complex, but the same principles apply.
No, with NVMe I still would highly advise against using size=2, min_size=1
The question is not if you will loose data, but the question is: When will you loose data? Within one year, 2? 3? 10?
Wido
From: "Anthony D'Atri" <anthony.datri@gmail.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, February 4, 2021 12:47:27 PM Subject: [ceph-users] Re: NVMe and 2x Replica
I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution.
FWIW, at Cephalocon another vendor made a similar claim during a talk.
* Failure rates are averages, not minima. Some drives will always fail sooner * Firmware and other design flaws can result in much higher rates of failure or insidious UREs that can result in partial data unavailability or loss * Latent soft failures may not be detected until a deep scrub succeeds, which could be weeks later * In a distributed system, there are up/down/failure scenarios where the location of even one good / canonical / latest copy of data is unclear, especially when drive or HBA cache is in play. * One of these is a power failure. Sure PDU / PSU redundancy helps, but stuff happens, like a DC underprovisioning amps, so that a spike in user traffic results in the whole row going down :-x Various unpleasant things can happen.
I was championing R3 even pre-Ceph when I was using ZFS or HBA RAID. As others have written, as drives get larger the time to fill them with replica data increases, as does the chance of overlapping failures. I’ve experieneced R2 overlapping failures more than once, with and before Ceph.
My sense has been that not many people run R2 for data they care about, and as has been written recently 2,2 EC is safer with the same raw:usable ratio. I’ve figured that vendors make R2 statements like these as a selling point to assert lower TCO. My first response is often “How much would it cost you directly, and indirectly in terms of user / customer goodwill, to loose data?”.
Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive.
^this. I’m watching the QLC arena with interest for the potential to narrow the CapEx gap. Durability has been one concern, though I’m seeing newer products claiming that eg. ZNS improves that. It also seems that there are something like what, *4* separate EDSFF / ruler form factors, I really want to embrace those eg. for object clusters, but I’m VERY wary of the longevity of competing standards and any single-source for chassies or drives.
Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing.
Refresh timelines. You’re funny ;) Every time, every single time, that I’ve worked in an organization that claims a 3 (or 5, or whatever) hardware refresh cycle, it hasn’t happened. When you start getting close, the capex doesn’t materialize, or the opex cost of DC hands and operational oversight. “How do you know that the drives will start failing or getting slower? Let’s revisit this in 6 months”. Etc.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It seems all the big vendors feel 2x is safe with NVMe but I get the feeling this community feels otherwise
Definitely! As someone who works for a big vendor (and I have since I worked at Fusion-IO way back in the old days), IMO the correct way to phrase this would probably be that "someone in technical marketing at the big vendors" was convinced that 2x was safe enough to put in a white paper or sales document. They (we, I guess, since I'm one of these types of people) are focused on performance and cost numbers and as much as I hate to admit it, it can get in the way of long-term reliability settings sometimes. This doesn't mean that they are "wrong" -- these documents are primarily meant to show the capabilities of their hardware, with a bill of materials containing their part numbers. It is expected that end users will adjust a few things when it comes to a production environment. The idea that NVMe is safer than spinning rust drives is not necessarily true -- and it's beside the point. You are just as likely to run into a weird situation where an OSD or pg acts up or disappears for non-hardware reasons. Unless you can live with "nine fives" instead of "five nines" (say, a caching type of application where you can re-generate the data), use a size of at least 3 -- and if you can't afford this much storage then look at erasure coding schemes. All of this is IMO of course, Mark On Thu, Feb 4, 2021 at 4:38 AM Adam Boyhan <adamb@medent.com> wrote:
I know there is already a few threads about 2x replication but I wanted to start one dedicated to discussion on NVMe. There are some older threads, but nothing recent that addresses how the vendors are now pushing the idea of 2x.
We are in the process of considering Ceph to replace our Nimble setup. We will have two completely separate clusters at two different sites that we are using rbd-mirror snapshot replication. The plan would be to run 2x replication on each cluster. 3x is still an option, but for obvious reasons 2x is enticing.
Both clusters will be spot on to the super micro example in the white paper below.
It seems all the big vendors feel 2x is safe with NVMe but I get the feeling this community feels otherwise. Trying to wrap my head around were the disconnect is between the big players and the community. I could be missing something, but even our Supermicro contact that we worked the config out with was in agreement with 2x on NVMe.
Appreciate the input!
[ https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf | https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf ]
[ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... | https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... ]
[ https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redh... | https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redh... ] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have just one more suggestion for you:
but even our Supermicro contact that we worked the config out with was in agreement with 2x on NVMe
These kinds of settings aren't set in stone, it is a one line command to rebalance (admittedly you wouldn't want to just do this casually). I don't know your situation in any detail, but perhaps you could start with size 3 and put off the size 2 decision until your cluster is maybe 30% full... then you could make a final decision to either add more storage or rebalance to size 2. You can also have different size settings for different pools depending on how important the data is. Mark On Thu, Feb 4, 2021 at 4:38 AM Adam Boyhan <adamb@medent.com> wrote:
I know there is already a few threads about 2x replication but I wanted to start one dedicated to discussion on NVMe. There are some older threads, but nothing recent that addresses how the vendors are now pushing the idea of 2x.
We are in the process of considering Ceph to replace our Nimble setup. We will have two completely separate clusters at two different sites that we are using rbd-mirror snapshot replication. The plan would be to run 2x replication on each cluster. 3x is still an option, but for obvious reasons 2x is enticing.
Both clusters will be spot on to the super micro example in the white paper below.
It seems all the big vendors feel 2x is safe with NVMe but I get the feeling this community feels otherwise. Trying to wrap my head around were the disconnect is between the big players and the community. I could be missing something, but even our Supermicro contact that we worked the config out with was in agreement with 2x on NVMe.
Appreciate the input!
[ https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf | https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf ]
[ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... | https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referenc... ]
[ https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redh... | https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redh... ] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (12)
-
Adam Boyhan
-
Anthony D'Atri
-
Brian :
-
DHilsbos＠performair.com
-
Frank Schilder
-
Jack
-
Janne Johansson
-
Mark Lehrer
-
Nathan Fish
-
Pascal Ehlert
-
Steven Pine
-
Wido den Hollander