5 May
2020
5 May
'20
10:57 p.m.
Dear Cephalopodians,
seeing the recent moves of major HDD vendors to sell SMR disks targeted for use in
consumer NAS devices (including RAID systems),
I got curious and wonder what the current status of SMR support in Bluestore is.
Of course, I'd expect disk vendors to give us host-managed SMR disks for data center
use cases (and to tell us when actually they do so...),
but in that case, Bluestore surely needs some new intelligence for best performance in the
shingled ages.
I had a quick look at the repository and could only make out that libzbc has been added
some years ago,
but no activity after this (also no tickets in the issue tracker). Is this still something
on the roadmap?
It would be wonderful for backup / archiving / mostly data ingest / cold storage clusters
to be able to use cheaper and larger disks once they become available :-).
I'm still positively optimistic for such use cases even though my personal experience
with SMR has not been so well up to now:
I bought a (not cleanly labelled...) DM-SMR, used it for BTRFS archiving (btrbk, i.e.
btrfs-send and -receive),
and after it was filled once, it got excruciatinly slow (less than a few kiB/s even when
changing only a few 100 MB after prolonged idle).
But then, deleting btrfs snapshots of a desktop OS is pure random read/write access, and
there are no optimizations for that use case in btrfs (and the drive did not even support
TRIM/DISCARD),
and I read on the btrfs list now that it can work well even in arrays mostly used for cold
storage if the balancing is throttled ;-).
Cheers,
Oliver
6 May
6 May
7:18 a.m.
Den ons 6 maj 2020 kl 00:58 skrev Oliver Freyermuth <
freyermuth(a)physik.uni-bonn.de>gt;:
Dear Cephalopodians,
seeing the recent moves of major HDD vendors to sell SMR disks targeted
for use in consumer NAS devices (including RAID systems),
I got curious and wonder what the current status of SMR support in
Bluestore is.
Of course, I'd expect disk vendors to give us host-managed SMR disks for
data center use cases (and to tell us when actually they do so...),
but in that case, Bluestore surely needs some new intelligence for best
performance in the shingled ages.
I've only done filestore on SMRs, and it did work for a while, in normal
cases for us, but it broke down horribly as soon as recovery needed to be
done.
I have no idea if filestore was doing the worst ever for SMRs, or if
bluestore will do better or if patches are going to help bluestore become
useful, but all in all, I can't say anything else to people wanting to
experiment with SMRs than "if you must use SMRs, make sure you test the
most evil of corner cases".
As you noted, one can easily get into <1M/s with SMRs by doing something
else than long linear writes, and you don't want to be in a place where
several hundred TBs of data is doing recovery at that speed.
To me, SMR is a con, its a trick to sell cheap crap to people who can't or
won't test properly. Doesn't matter if its ceph recovery/backfill, btrfs
deletes or someones NAS raid sync job that places the final straw on the
camels back and breaks it, it's the fact that filesystems do lots more than
just easy nice long linear writes. No matter if it is fsck, defrags or ceph
PG splits/reshardings, there will be disk meta-operations that needs to be
done which includes tons of random small writes, and SMR drives will punish
you for them when you need the drive up the most. 8-(
If I had some very special system which used cheap disks to pretend to be a
tape device and only did 10G sized reads/writes like a tape would do, then
I could see a use case for SMR.
--
May the most significant bit of your life be positive.
11:28 a.m.
Dear Janne,
Am 06.05.20 um 09:18 schrieb Janne Johansson:
Den ons 6 maj 2020 kl 00:58 skrev Oliver Freyermuth
<freyermuth(a)physik.uni-bonn.de <mailto:freyermuth@physik.uni-bonn.de>>:
Dear Cephalopodians,
seeing the recent moves of major HDD vendors to sell SMR disks targeted for use in
consumer NAS devices (including RAID systems),
I got curious and wonder what the current status of SMR support in Bluestore is.
Of course, I'd expect disk vendors to give us host-managed SMR disks for data
center use cases (and to tell us when actually they do so...),
but in that case, Bluestore surely needs some new intelligence for best performance
in the shingled ages.
I've only done filestore on SMRs, and it did work for a while, in normal cases for
us, but it broke down horribly as soon as recovery needed to be done.
I have no idea if filestore was doing the worst ever for SMRs, or if bluestore will do
better or if patches are going to help bluestore become useful, but all in all, I
can't say anything else to people wanting to experiment with SMRs than "if you
must use SMRs, make sure you test the most evil of corner cases".
Thanks for the input and especially the hands-on experience! That's very helpful (and
"expensive" to gather), so thanks for sharing!
After my "small-scale" experiences, I would indeed have expected exactly that.
My sincere hope is that this hardware will become useable by making use of Copy-on-Write
semantics
to align writes into larger, consecutive batches.
As you noted, one can easily get into <1M/s with SMRs by doing something else than
long linear writes, and you don't want to be in a place where several hundred TBs of
data is doing recovery at that speed.
To me, SMR is a con, its a trick to sell cheap crap to people who can't or won't
test properly. Doesn't matter if its ceph recovery/backfill, btrfs deletes or someones
NAS raid sync job that places the final straw on the camels back and breaks it, it's
the fact that filesystems do lots more than just easy nice long linear writes. No matter
if it is fsck, defrags or ceph PG splits/reshardings, there will be disk meta-operations
that needs to be done which includes tons of random small writes, and SMR drives will
punish you for them when you need the drive up the most. 8-(
If I had some very special system which used cheap disks to pretend to be a tape device
and only did 10G sized reads/writes like a tape would do, then I could see a use case for
SMR.
I agree that in many cases SMR is not the correct hardware to use, and never will be.
Indeed, I also agree that in most cases the "trick to sell cheap crap to people who
can't or won't test properly"
applies, even more with disk-managed SMR which in some cases gives you zero control and
maximum frustration.
Still, my hope would be that especially for archiving purposes (think of a pure Ceph-RGW
cluster fed with Restic, Duplicati or similar tools), we can make good use of the cheaper
hardware
(but then, this would of course need to be host-managed SMR, and the file system should
know about it). I currently only know of Dropbox who are actively doing that
(and I guess they can easily, since they deduplicate data and probably rarely delete), and
they seem to have developed their own file system to deal with this essentially.
It would be cool to have this with Ceph. You might also think about having a separate pool
for "colder" objects which is SMR-backed
(likely coupled with SSDs / NVMes for WAL / BlockDB). In short, we'd never even think
about using it with CephFS in our HPC cluster
(unless some admin-controllable write-once-read-many use cases evolve, which we could
think about for centrally managed high-energy physics data),
or RBD in our virtualization cluster.
We're more interested in it for our backup cluster which mostly sees data ingest and
the chunking into larger batches is even done client-side (Restic, Duplicati etc.).
Of course, your point about resharding and PG splits fully applies, so this for sure needs
careful development (and testing!) to reduce the randomness as far as possible
(if we want to make use of this hardware for the use cases it may fit).
Cheers and thanks for your input,
Oliver
--
May the most significant bit of your life be positive.
1:53 p.m.
We’re prototyping a native SMR object store now to run alongside Ceph, which has been our
only object store backend for the last 3 years and is somewhat problematic on some
metrics. I believe trying to use SMR drives with a file system in the architecture (as in
Ceph) is a non starter, the solution is really to treat them like tape drives. Using a
sequential access model across the volume is everything. So the tape metaphor is important
down to how you handle free space collection.
Unfortunately, trying to expose this new architecture using a libRADOS API isn’t something
we’re interested in doing since we don’t use Ceph for anything above the libRADOS layer of
the stack, but our code will be open source if someone else wants to take a crack at it.
It wouldn’t be easy. Our implementation is based on reusing the sequential access version
of the SCSI protocol, and our host transport is iSER (iSCSI over RDMA), so it’s not a
natural overlay to libRADOS to put it mildly, particularly since we’re doing EC well above
this layer of our application stack.
Steve Cranage
Principal Architect, Co-Founder
DeepSpace Storage
[cid:image001.png@01D3FCBC.58FDB6F0]
From: Oliver Freyermuth<mailto:freyermuth@physik.uni-bonn.de>
Sent: Wednesday, May 6, 2020 5:28 AM
To: Janne Johansson<mailto:icepic.dz@gmail.com>
Cc: ceph-users<mailto:ceph-users@ceph.io>
Subject: [ceph-users] Re: State of SMR support in Ceph?
Dear Janne,
Am 06.05.20 um 09:18 schrieb Janne Johansson:
Den ons 6 maj 2020 kl 00:58 skrev Oliver Freyermuth
<freyermuth(a)physik.uni-bonn.de <mailto:freyermuth@physik.uni-bonn.de>>:
Dear Cephalopodians,
seeing the recent moves of major HDD vendors to sell SMR disks targeted for use in
consumer NAS devices (including RAID systems),
I got curious and wonder what the current status of SMR support in Bluestore is.
Of course, I'd expect disk vendors to give us host-managed SMR disks for data
center use cases (and to tell us when actually they do so...),
but in that case, Bluestore surely needs some new intelligence for best performance
in the shingled ages.
I've only done filestore on SMRs, and it did work for a while, in normal cases for
us, but it broke down horribly as soon as recovery needed to be done.
I have no idea if filestore was doing the worst ever for SMRs, or if bluestore will do
better or if patches are going to help bluestore become useful, but all in all, I
can't say anything else to people wanting to experiment with SMRs than "if you
must use SMRs, make sure you test the most evil of corner cases".
Thanks for the input and especially the hands-on experience! That's very helpful (and
"expensive" to gather), so thanks for sharing!
After my "small-scale" experiences, I would indeed have expected exactly that.
My sincere hope is that this hardware will become useable by making use of Copy-on-Write
semantics
to align writes into larger, consecutive batches.
As you noted, one can easily get into <1M/s with SMRs by doing something else than
long linear writes, and you don't want to be in a place where several hundred TBs of
data is doing recovery at that speed.
To me, SMR is a con, its a trick to sell cheap crap to people who can't or won't
test properly. Doesn't matter if its ceph recovery/backfill, btrfs deletes or someones
NAS raid sync job that places the final straw on the camels back and breaks it, it's
the fact that filesystems do lots more than just easy nice long linear writes. No matter
if it is fsck, defrags or ceph PG splits/reshardings, there will be disk meta-operations
that needs to be done which includes tons of random small writes, and SMR drives will
punish you for them when you need the drive up the most. 8-(
If I had some very special system which used cheap disks to pretend to be a tape device
and only did 10G sized reads/writes like a tape would do, then I could see a use case for
SMR.
I agree that in many cases SMR is not the correct hardware to use, and never will be.
Indeed, I also agree that in most cases the "trick to sell cheap crap to people who
can't or won't test properly"
applies, even more with disk-managed SMR which in some cases gives you zero control and
maximum frustration.
Still, my hope would be that especially for archiving purposes (think of a pure Ceph-RGW
cluster fed with Restic, Duplicati or similar tools), we can make good use of the cheaper
hardware
(but then, this would of course need to be host-managed SMR, and the file system should
know about it). I currently only know of Dropbox who are actively doing that
(and I guess they can easily, since they deduplicate data and probably rarely delete), and
they seem to have developed their own file system to deal with this essentially.
It would be cool to have this with Ceph. You might also think about having a separate pool
for "colder" objects which is SMR-backed
(likely coupled with SSDs / NVMes for WAL / BlockDB). In short, we'd never even think
about using it with CephFS in our HPC cluster
(unless some admin-controllable write-once-read-many use cases evolve, which we could
think about for centrally managed high-energy physics data),
or RBD in our virtualization cluster.
We're more interested in it for our backup cluster which mostly sees data ingest and
the chunking into larger batches is even done client-side (Restic, Duplicati etc.).
Of course, your point about resharding and PG splits fully applies, so this for sure needs
careful development (and testing!) to reduce the randomness as far as possible
(if we want to make use of this hardware for the use cases it may fit).
Cheers and thanks for your input,
Oliver
--
May the most significant bit of your life be positive.
2:12 p.m.
Take a look at the available SMR drives:
https://www.ixsystems.com/community/resources/list-of-known-smr-drives.141/
I wouldn’t put a single one of those drives into a Ceph system. You won’t save any money.
In fact, it’s likely a sink-hole for labor on your part. In fact, you couldn’t pay me to
take them. The Seagate 8TB units listed above are the same price as their Enterprise.
Clearly, manufacturing costs went down, and this allows a (temporary) margin boost for the
manufacturer over the near term.
Their intended use cases are not for a Ceph system. Performance is lower. Therefore,
cost is higher. They’re meant for desktops and cheap long term storage.
I’d let Backblaze test, and wait a year or two for the fail rate reports. And then, I’d
take into account their primary use case...long term backup and desktop use.
When I built out Ceph, I spent some time looking at performance in mixed-use, cost,
reliability.
I looked at the user application(s) from an IOPS and bandwidth perspective. Then I
plotted out the demands of the application(s), as well as user trends, and added 20%.
Then I threw in 20% growth per month. I extended it out 2 years. Then I doubled it.
What’s my time worth? What’s the time worth for 160 developers? Rock-solid performance
and reliability is worth it.
Then I figured out the cost, got my funding, ordered, and built it. I warned the user
community up front about IOPS performance, drastic application demands require changes.
The users have done an excellent job staying within their requirements. We didn’t want an
inefficient application that is expensive to implement.
Seagate Exos 7e8’s are ~$25/TB. 8TB, 16TB.
I have 200 Seagate Enterprise drives. In 2.5 years, I’ve had 2 drives fail. I have about
160 users, and dozens of automated platforms that use the system. I make the users aware
of the IOPS demands. They make their applications and tests more efficient over time.
They schedule and queue. I like to sleep at night. I spend about 20 minutes a week doing
maintenance, mainly my Ceph dashboard and some Ansible output.
I talk to the users, I spend time with them when they’re planning.
It is time for a boost. I’ve followed costs and DWPD on available enterprise SSD’s.
With my same application and user data, updated, projected, I need an enterprise SSD with
DWPD around 3. 5 and 10 are too expensive. 1 is too low. I won’t use “read-optimizes”
with DWPD of 1.
I didn’t even look at consumer grade.
Micron MAX 5200 (SATA) at DWPD of 3 is perfect. Cost is within reason when I consider
performance and my time. I’m about to add 56 of these, need to finish my testing.
3:29 p.m.
Am 06.05.20 um 16:12 schrieb brad.swanson(a)adtran.com:
Take a look at the available SMR drives:
https://www.ixsystems.com/community/resources/list-of-known-smr-drives.141/
Thanks for this nice overview link!
Indeed, my question was more thinking about "the next years". While currently,
SMR is certainly not to be used in enterprise systems,
unless your use cases and software stack really matches it, all the different
energy-assisted technologies (MAMR, HAMR, something-else-AMR) are likely to stay for the
next years,
unless we manage to change physics. So it would be useful to have a way to exploit this
technology, even if it (currently) will cause headaches,
but Ceph has already managed to overcome many of the headaches admins had to endure in the
past (e.g. RAID controller intricacies).
I wouldn’t put a single one of those drives into a
Ceph system. You won’t save any money. In fact, it’s likely a sink-hole for labor on
your part. In fact, you couldn’t pay me to take them. The Seagate 8TB units listed above
are the same price as their Enterprise. Clearly, manufacturing costs went down, and this
allows a (temporary) margin boost for the manufacturer over the near term.
Their intended use cases are not for a Ceph system. Performance is lower. Therefore,
cost is higher. They’re meant for desktops and cheap long term storage.
I’d let Backblaze test, and wait a year or two for the fail rate reports. And then, I’d
take into account their primary use case...long term backup and desktop use.
The only use case I'd consider would indeed be tape-like: Long-term backups, the only
writing activity is influx of data, balancing (if needed)
and purging old backups (after applying some chunking such as restic / duplicati and
others do).
When I built out Ceph, I spent some time looking at
performance in mixed-use, cost, reliability.
I looked at the user application(s) from an IOPS and bandwidth perspective. Then I
plotted out the demands of the application(s), as well as user trends, and added 20%.
Then I threw in 20% growth per month. I extended it out 2 years. Then I doubled it.
What’s my time worth? What’s the time worth for 160 developers? Rock-solid performance
and reliability is worth it.
We did similar estimates indeed for our CephFS use case. And you are fully right, SMR (and
potentially, any *AMR) is out of the picture for this use case.
Then I figured out the cost, got my funding, ordered,
and built it. I warned the user community up front about IOPS performance, drastic
application demands require changes. The users have done an excellent job staying within
their requirements. We didn’t want an inefficient application that is expensive to
implement.
Seagate Exos 7e8’s are ~$25/TB. 8TB, 16TB.
I have 200 Seagate Enterprise drives. In 2.5 years, I’ve had 2 drives fail. I have
about 160 users, and dozens of automated platforms that use the system. I make the users
aware of the IOPS demands. They make their applications and tests more efficient over
time. They schedule and queue. I like to sleep at night. I spend about 20 minutes a
week doing maintenance, mainly my Ceph dashboard and some Ansible output.
I talk to the users, I spend time with them when they’re planning.
I also agree on this (we are similar scale with our computing cluster, albeit our Exos
drives were less happy, we have about one failure per month, but our cooling is likely
worse than yours).
It is time for a boost. I’ve followed costs and DWPD
on available enterprise SSD’s. With my same application and user data, updated,
projected, I need an enterprise SSD with DWPD around 3. 5 and 10 are too expensive. 1 is
too low. I won’t use “read-optimizes” with DWPD of 1.
I didn’t even look at consumer grade.
Micron MAX 5200 (SATA) at DWPD of 3 is perfect. Cost is within reason when I consider
performance and my time. I’m about to add 56 of these, need to finish my testing.
Thanks for this interesting report! :-)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1458
days inactive
1459
days old
5 comments
4 participants
participants (4)
-
brad.swanson＠adtran.com -
Cranage, Steve -
Janne Johansson -
Oliver Freyermuth