State of SMR support in Ceph?
Dear Cephalopodians, seeing the recent moves of major HDD vendors to sell SMR disks targeted for use in consumer NAS devices (including RAID systems), I got curious and wonder what the current status of SMR support in Bluestore is. Of course, I'd expect disk vendors to give us host-managed SMR disks for data center use cases (and to tell us when actually they do so...), but in that case, Bluestore surely needs some new intelligence for best performance in the shingled ages. I had a quick look at the repository and could only make out that libzbc has been added some years ago, but no activity after this (also no tickets in the issue tracker). Is this still something on the roadmap? It would be wonderful for backup / archiving / mostly data ingest / cold storage clusters to be able to use cheaper and larger disks once they become available :-). I'm still positively optimistic for such use cases even though my personal experience with SMR has not been so well up to now: I bought a (not cleanly labelled...) DM-SMR, used it for BTRFS archiving (btrbk, i.e. btrfs-send and -receive), and after it was filled once, it got excruciatinly slow (less than a few kiB/s even when changing only a few 100 MB after prolonged idle). But then, deleting btrfs snapshots of a desktop OS is pure random read/write access, and there are no optimizations for that use case in btrfs (and the drive did not even support TRIM/DISCARD), and I read on the btrfs list now that it can work well even in arrays mostly used for cold storage if the balancing is throttled ;-). Cheers, Oliver
Den ons 6 maj 2020 kl 00:58 skrev Oliver Freyermuth < freyermuth@physik.uni-bonn.de>:
Dear Cephalopodians, seeing the recent moves of major HDD vendors to sell SMR disks targeted for use in consumer NAS devices (including RAID systems), I got curious and wonder what the current status of SMR support in Bluestore is. Of course, I'd expect disk vendors to give us host-managed SMR disks for data center use cases (and to tell us when actually they do so...), but in that case, Bluestore surely needs some new intelligence for best performance in the shingled ages.
I've only done filestore on SMRs, and it did work for a while, in normal cases for us, but it broke down horribly as soon as recovery needed to be done. I have no idea if filestore was doing the worst ever for SMRs, or if bluestore will do better or if patches are going to help bluestore become useful, but all in all, I can't say anything else to people wanting to experiment with SMRs than "if you must use SMRs, make sure you test the most evil of corner cases". As you noted, one can easily get into <1M/s with SMRs by doing something else than long linear writes, and you don't want to be in a place where several hundred TBs of data is doing recovery at that speed. To me, SMR is a con, its a trick to sell cheap crap to people who can't or won't test properly. Doesn't matter if its ceph recovery/backfill, btrfs deletes or someones NAS raid sync job that places the final straw on the camels back and breaks it, it's the fact that filesystems do lots more than just easy nice long linear writes. No matter if it is fsck, defrags or ceph PG splits/reshardings, there will be disk meta-operations that needs to be done which includes tons of random small writes, and SMR drives will punish you for them when you need the drive up the most. 8-( If I had some very special system which used cheap disks to pretend to be a tape device and only did 10G sized reads/writes like a tape would do, then I could see a use case for SMR. -- May the most significant bit of your life be positive.
Dear Janne, Am 06.05.20 um 09:18 schrieb Janne Johansson:
Den ons 6 maj 2020 kl 00:58 skrev Oliver Freyermuth <freyermuth@physik.uni-bonn.de <mailto:freyermuth@physik.uni-bonn.de>>:
Dear Cephalopodians, seeing the recent moves of major HDD vendors to sell SMR disks targeted for use in consumer NAS devices (including RAID systems), I got curious and wonder what the current status of SMR support in Bluestore is. Of course, I'd expect disk vendors to give us host-managed SMR disks for data center use cases (and to tell us when actually they do so...), but in that case, Bluestore surely needs some new intelligence for best performance in the shingled ages.
I've only done filestore on SMRs, and it did work for a while, in normal cases for us, but it broke down horribly as soon as recovery needed to be done. I have no idea if filestore was doing the worst ever for SMRs, or if bluestore will do better or if patches are going to help bluestore become useful, but all in all, I can't say anything else to people wanting to experiment with SMRs than "if you must use SMRs, make sure you test the most evil of corner cases".
Thanks for the input and especially the hands-on experience! That's very helpful (and "expensive" to gather), so thanks for sharing! After my "small-scale" experiences, I would indeed have expected exactly that. My sincere hope is that this hardware will become useable by making use of Copy-on-Write semantics to align writes into larger, consecutive batches.
As you noted, one can easily get into <1M/s with SMRs by doing something else than long linear writes, and you don't want to be in a place where several hundred TBs of data is doing recovery at that speed.
To me, SMR is a con, its a trick to sell cheap crap to people who can't or won't test properly. Doesn't matter if its ceph recovery/backfill, btrfs deletes or someones NAS raid sync job that places the final straw on the camels back and breaks it, it's the fact that filesystems do lots more than just easy nice long linear writes. No matter if it is fsck, defrags or ceph PG splits/reshardings, there will be disk meta-operations that needs to be done which includes tons of random small writes, and SMR drives will punish you for them when you need the drive up the most. 8-(
If I had some very special system which used cheap disks to pretend to be a tape device and only did 10G sized reads/writes like a tape would do, then I could see a use case for SMR.
I agree that in many cases SMR is not the correct hardware to use, and never will be. Indeed, I also agree that in most cases the "trick to sell cheap crap to people who can't or won't test properly" applies, even more with disk-managed SMR which in some cases gives you zero control and maximum frustration. Still, my hope would be that especially for archiving purposes (think of a pure Ceph-RGW cluster fed with Restic, Duplicati or similar tools), we can make good use of the cheaper hardware (but then, this would of course need to be host-managed SMR, and the file system should know about it). I currently only know of Dropbox who are actively doing that (and I guess they can easily, since they deduplicate data and probably rarely delete), and they seem to have developed their own file system to deal with this essentially. It would be cool to have this with Ceph. You might also think about having a separate pool for "colder" objects which is SMR-backed (likely coupled with SSDs / NVMes for WAL / BlockDB). In short, we'd never even think about using it with CephFS in our HPC cluster (unless some admin-controllable write-once-read-many use cases evolve, which we could think about for centrally managed high-energy physics data), or RBD in our virtualization cluster. We're more interested in it for our backup cluster which mostly sees data ingest and the chunking into larger batches is even done client-side (Restic, Duplicati etc.). Of course, your point about resharding and PG splits fully applies, so this for sure needs careful development (and testing!) to reduce the randomness as far as possible (if we want to make use of this hardware for the use cases it may fit). Cheers and thanks for your input, Oliver
-- May the most significant bit of your life be positive.
We’re prototyping a native SMR object store now to run alongside Ceph, which has been our only object store backend for the last 3 years and is somewhat problematic on some metrics. I believe trying to use SMR drives with a file system in the architecture (as in Ceph) is a non starter, the solution is really to treat them like tape drives. Using a sequential access model across the volume is everything. So the tape metaphor is important down to how you handle free space collection. Unfortunately, trying to expose this new architecture using a libRADOS API isn’t something we’re interested in doing since we don’t use Ceph for anything above the libRADOS layer of the stack, but our code will be open source if someone else wants to take a crack at it. It wouldn’t be easy. Our implementation is based on reusing the sequential access version of the SCSI protocol, and our host transport is iSER (iSCSI over RDMA), so it’s not a natural overlay to libRADOS to put it mildly, particularly since we’re doing EC well above this layer of our application stack. Steve Cranage Principal Architect, Co-Founder DeepSpace Storage [cid:image001.png@01D3FCBC.58FDB6F0] From: Oliver Freyermuth<mailto:freyermuth@physik.uni-bonn.de> Sent: Wednesday, May 6, 2020 5:28 AM To: Janne Johansson<mailto:icepic.dz@gmail.com> Cc: ceph-users<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: State of SMR support in Ceph? Dear Janne, Am 06.05.20 um 09:18 schrieb Janne Johansson:
Den ons 6 maj 2020 kl 00:58 skrev Oliver Freyermuth <freyermuth@physik.uni-bonn.de <mailto:freyermuth@physik.uni-bonn.de>>:
Dear Cephalopodians, seeing the recent moves of major HDD vendors to sell SMR disks targeted for use in consumer NAS devices (including RAID systems), I got curious and wonder what the current status of SMR support in Bluestore is. Of course, I'd expect disk vendors to give us host-managed SMR disks for data center use cases (and to tell us when actually they do so...), but in that case, Bluestore surely needs some new intelligence for best performance in the shingled ages.
I've only done filestore on SMRs, and it did work for a while, in normal cases for us, but it broke down horribly as soon as recovery needed to be done. I have no idea if filestore was doing the worst ever for SMRs, or if bluestore will do better or if patches are going to help bluestore become useful, but all in all, I can't say anything else to people wanting to experiment with SMRs than "if you must use SMRs, make sure you test the most evil of corner cases".
Thanks for the input and especially the hands-on experience! That's very helpful (and "expensive" to gather), so thanks for sharing! After my "small-scale" experiences, I would indeed have expected exactly that. My sincere hope is that this hardware will become useable by making use of Copy-on-Write semantics to align writes into larger, consecutive batches.
As you noted, one can easily get into <1M/s with SMRs by doing something else than long linear writes, and you don't want to be in a place where several hundred TBs of data is doing recovery at that speed.
To me, SMR is a con, its a trick to sell cheap crap to people who can't or won't test properly. Doesn't matter if its ceph recovery/backfill, btrfs deletes or someones NAS raid sync job that places the final straw on the camels back and breaks it, it's the fact that filesystems do lots more than just easy nice long linear writes. No matter if it is fsck, defrags or ceph PG splits/reshardings, there will be disk meta-operations that needs to be done which includes tons of random small writes, and SMR drives will punish you for them when you need the drive up the most. 8-(
If I had some very special system which used cheap disks to pretend to be a tape device and only did 10G sized reads/writes like a tape would do, then I could see a use case for SMR.
I agree that in many cases SMR is not the correct hardware to use, and never will be. Indeed, I also agree that in most cases the "trick to sell cheap crap to people who can't or won't test properly" applies, even more with disk-managed SMR which in some cases gives you zero control and maximum frustration. Still, my hope would be that especially for archiving purposes (think of a pure Ceph-RGW cluster fed with Restic, Duplicati or similar tools), we can make good use of the cheaper hardware (but then, this would of course need to be host-managed SMR, and the file system should know about it). I currently only know of Dropbox who are actively doing that (and I guess they can easily, since they deduplicate data and probably rarely delete), and they seem to have developed their own file system to deal with this essentially. It would be cool to have this with Ceph. You might also think about having a separate pool for "colder" objects which is SMR-backed (likely coupled with SSDs / NVMes for WAL / BlockDB). In short, we'd never even think about using it with CephFS in our HPC cluster (unless some admin-controllable write-once-read-many use cases evolve, which we could think about for centrally managed high-energy physics data), or RBD in our virtualization cluster. We're more interested in it for our backup cluster which mostly sees data ingest and the chunking into larger batches is even done client-side (Restic, Duplicati etc.). Of course, your point about resharding and PG splits fully applies, so this for sure needs careful development (and testing!) to reduce the randomness as far as possible (if we want to make use of this hardware for the use cases it may fit). Cheers and thanks for your input, Oliver
-- May the most significant bit of your life be positive.
Take a look at the available SMR drives: https://www.ixsystems.com/community/resources/list-of-known-smr-drives.141/ I wouldn’t put a single one of those drives into a Ceph system. You won’t save any money. In fact, it’s likely a sink-hole for labor on your part. In fact, you couldn’t pay me to take them. The Seagate 8TB units listed above are the same price as their Enterprise. Clearly, manufacturing costs went down, and this allows a (temporary) margin boost for the manufacturer over the near term. Their intended use cases are not for a Ceph system. Performance is lower. Therefore, cost is higher. They’re meant for desktops and cheap long term storage. I’d let Backblaze test, and wait a year or two for the fail rate reports. And then, I’d take into account their primary use case...long term backup and desktop use. When I built out Ceph, I spent some time looking at performance in mixed-use, cost, reliability. I looked at the user application(s) from an IOPS and bandwidth perspective. Then I plotted out the demands of the application(s), as well as user trends, and added 20%. Then I threw in 20% growth per month. I extended it out 2 years. Then I doubled it. What’s my time worth? What’s the time worth for 160 developers? Rock-solid performance and reliability is worth it. Then I figured out the cost, got my funding, ordered, and built it. I warned the user community up front about IOPS performance, drastic application demands require changes. The users have done an excellent job staying within their requirements. We didn’t want an inefficient application that is expensive to implement. Seagate Exos 7e8’s are ~$25/TB. 8TB, 16TB. I have 200 Seagate Enterprise drives. In 2.5 years, I’ve had 2 drives fail. I have about 160 users, and dozens of automated platforms that use the system. I make the users aware of the IOPS demands. They make their applications and tests more efficient over time. They schedule and queue. I like to sleep at night. I spend about 20 minutes a week doing maintenance, mainly my Ceph dashboard and some Ansible output. I talk to the users, I spend time with them when they’re planning. It is time for a boost. I’ve followed costs and DWPD on available enterprise SSD’s. With my same application and user data, updated, projected, I need an enterprise SSD with DWPD around 3. 5 and 10 are too expensive. 1 is too low. I won’t use “read-optimizes” with DWPD of 1. I didn’t even look at consumer grade. Micron MAX 5200 (SATA) at DWPD of 3 is perfect. Cost is within reason when I consider performance and my time. I’m about to add 56 of these, need to finish my testing.
Am 06.05.20 um 16:12 schrieb brad.swanson@adtran.com:
Take a look at the available SMR drives: https://www.ixsystems.com/community/resources/list-of-known-smr-drives.141/
Thanks for this nice overview link! Indeed, my question was more thinking about "the next years". While currently, SMR is certainly not to be used in enterprise systems, unless your use cases and software stack really matches it, all the different energy-assisted technologies (MAMR, HAMR, something-else-AMR) are likely to stay for the next years, unless we manage to change physics. So it would be useful to have a way to exploit this technology, even if it (currently) will cause headaches, but Ceph has already managed to overcome many of the headaches admins had to endure in the past (e.g. RAID controller intricacies).
I wouldn’t put a single one of those drives into a Ceph system. You won’t save any money. In fact, it’s likely a sink-hole for labor on your part. In fact, you couldn’t pay me to take them. The Seagate 8TB units listed above are the same price as their Enterprise. Clearly, manufacturing costs went down, and this allows a (temporary) margin boost for the manufacturer over the near term.
Their intended use cases are not for a Ceph system. Performance is lower. Therefore, cost is higher. They’re meant for desktops and cheap long term storage. I’d let Backblaze test, and wait a year or two for the fail rate reports. And then, I’d take into account their primary use case...long term backup and desktop use.
The only use case I'd consider would indeed be tape-like: Long-term backups, the only writing activity is influx of data, balancing (if needed) and purging old backups (after applying some chunking such as restic / duplicati and others do).
When I built out Ceph, I spent some time looking at performance in mixed-use, cost, reliability. I looked at the user application(s) from an IOPS and bandwidth perspective. Then I plotted out the demands of the application(s), as well as user trends, and added 20%. Then I threw in 20% growth per month. I extended it out 2 years. Then I doubled it. What’s my time worth? What’s the time worth for 160 developers? Rock-solid performance and reliability is worth it.
We did similar estimates indeed for our CephFS use case. And you are fully right, SMR (and potentially, any *AMR) is out of the picture for this use case.
Then I figured out the cost, got my funding, ordered, and built it. I warned the user community up front about IOPS performance, drastic application demands require changes. The users have done an excellent job staying within their requirements. We didn’t want an inefficient application that is expensive to implement.
Seagate Exos 7e8’s are ~$25/TB. 8TB, 16TB. I have 200 Seagate Enterprise drives. In 2.5 years, I’ve had 2 drives fail. I have about 160 users, and dozens of automated platforms that use the system. I make the users aware of the IOPS demands. They make their applications and tests more efficient over time. They schedule and queue. I like to sleep at night. I spend about 20 minutes a week doing maintenance, mainly my Ceph dashboard and some Ansible output. I talk to the users, I spend time with them when they’re planning.
I also agree on this (we are similar scale with our computing cluster, albeit our Exos drives were less happy, we have about one failure per month, but our cooling is likely worse than yours).
It is time for a boost. I’ve followed costs and DWPD on available enterprise SSD’s. With my same application and user data, updated, projected, I need an enterprise SSD with DWPD around 3. 5 and 10 are too expensive. 1 is too low. I won’t use “read-optimizes” with DWPD of 1. I didn’t even look at consumer grade.
Micron MAX 5200 (SATA) at DWPD of 3 is perfect. Cost is within reason when I consider performance and my time. I’m about to add 56 of these, need to finish my testing.
Thanks for this interesting report! :-)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
brad.swanson＠adtran.com
-
Cranage, Steve
-
Janne Johansson
-
Oliver Freyermuth