22 Sep
2020
22 Sep
'20
10:44 p.m.
Hi!
After almost a year of development in my spare time I present my own software-defined
block storage system: Vitastor - https://vitastor.io
I designed it similar to Ceph in many ways, it also has Pools, PGs, OSDs, different coding
schemes, rebalancing and so on. However it's much simpler and much faster. In a test
cluster with SATA SSDs it achieved Q1T1 latency of 0.14ms which is especially great
compared to Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops
saturation" parallel load benchmark it reached 895k read / 162k write iops, compared
to Ceph's 480k / 100k on the same hardware, but the most interesting part was CPU
usage: Ceph OSDs were using 40 CPU cores out of 64 on each node and Vitastor was only
using 4.
Of course it's an early pre-release which means that, for example, it lacks snapshot
support and other useful features. However the base is finished - it works and runs QEMU
VMs. I like the design and I plan to develop it further.
There are more details in the README file which currently opens from the domain
https://vitastor.io
Sorry if it was a bit off-topic, I just thought it could be interesting for you :)
--
With best regards,
Vitaliy Filippov
23 Sep
23 Sep
12:07 a.m.
On Wed, Sep 23, 2020 at 3:44 AM <vitalif(a)yourcmc.ru> wrote:
Hi!
After almost a year of development in my spare time I present my own software-defined
block storage system: Vitastor - https://vitastor.io
I designed it similar to Ceph in many ways, it also has Pools, PGs, OSDs, different
coding schemes, rebalancing and so on. However it's much simpler and much faster. In a
test cluster with SATA SSDs it achieved Q1T1 latency of 0.14ms which is especially great
compared to Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops
saturation" parallel load benchmark it reached 895k read / 162k write iops, compared
to Ceph's 480k / 100k on the same hardware, but the most interesting part was CPU
usage: Ceph OSDs were using 40 CPU cores out of 64 on each node and Vitastor was only
using 4.
Of course it's an early pre-release which means that, for example, it lacks snapshot
support and other useful features. However the base is finished - it works and runs QEMU
VMs. I like the design and I plan to develop it further.
There are more details in the README file which currently opens from the domain
https://vitastor.io
Very interesting.
Could you please add more details to the README file, as listed below?
1. Network benchmarks, in terms of achievable throughput and latency.
2. The type of the switch you used, and if there was any latency
tuning, please state it.
3. The network MTU.
4. The utilization figures for SSDs and network interfaces during each test.
Also, given that the scope of the project only includes block storage,
I think it would be fair to ask for a comparison with DRBD 9 and
possibly Linstor, not only with Ceph.
--
Alexander E. Patrakov
CV: http://pc.cd/PLz7
12:24 a.m.
On 23/09/2020 8:44 am, vitalif(a)yourcmc.ru wrote:
There are more details in the README file which
currently opens from the domainhttps://vitastor.io
that redirects to https://yourcmc.ru/git/vitalif/vitastor
Is that your own site?
--
Lindsay
4:29 a.m.
Yes
Not RBD, it has an own qemu driver
23 сентября 2020 г. 3:24:23 GMT+03:00, Lindsay Mathieson
<lindsay.mathieson(a)gmail.com> пишет:
On 23/09/2020 8:44 am, vitalif(a)yourcmc.ru wrote:
--
With best regards,
Vitaliy Filippov
There are more details in the README file which
currently opens from
the domainhttps://vitastor.io
that redirects to https://yourcmc.ru/git/vitalif/vitastor
Is that your own site?
--
Lindsay
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:59 a.m.
It succeeds to load with LD_PRELOAD as, as I understand, block_register() gets called.
QAPI and new QAPI-based block device syntax don't work though because they're
based on IDLs built into QEMU... QAPI will require patching, yeah. It would be nicer if
QAPI supported plugins too... :-)
--
With best regards,
Vitaliy Filippov
5:37 a.m.
I love how it’s not possible to delete inodes yet. Data loss would be a thing of the
past!
Jokes aside, interesting project.
Sent from mobile
Op 23 sep. 2020 om 00:45 heeft vitalif(a)yourcmc.ru het
volgende geschreven:
Hi!
After almost a year of development in my spare time I present my own software-defined
block storage system: Vitastor - https://vitastor.io
I designed it similar to Ceph in many ways, it also has Pools, PGs, OSDs, different
coding schemes, rebalancing and so on. However it's much simpler and much faster. In a
test cluster with SATA SSDs it achieved Q1T1 latency of 0.14ms which is especially great
compared to Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops
saturation" parallel load benchmark it reached 895k read / 162k write iops, compared
to Ceph's 480k / 100k on the same hardware, but the most interesting part was CPU
usage: Ceph OSDs were using 40 CPU cores out of 64 on each node and Vitastor was only
using 4.
Of course it's an early pre-release which means that, for example, it lacks snapshot
support and other useful features. However the base is finished - it works and runs QEMU
VMs. I like the design and I plan to develop it further.
There are more details in the README file which currently opens from the domain
https://vitastor.io
Sorry if it was a bit off-topic, I just thought it could be interesting for you :)
--
With best regards,
Vitaliy Filippov
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:48 a.m.
Vitaliy you are crazy ;) But really cool work. Why not combine efforts
with ceph? Especially with something as important as SDS and PB's of
clients data stored on it, everyone with a little bit of brain chooses a
solution from a 'reliable' source. For me it was decisive to learn that
CERN and NASA were using this on a large scale. I do not have the
expertise nor time (like probably 90% of ceph users) to test how they
have been testing and using ceph.
I often see opensource projects that could benefit from cooperation.
Some teams totally lack the expertise that others have, and vice versa.
Providing the community with 10 or 20 'shitty' projects instead of 3
'good' projects.
I think opensource projects should more often embrace a sort of modular
development solution. Where others can change functionality by replacing
just a module. If I ever get my idea funded, I would make it like this.
-----Original Message-----
Cc: dev(a)ceph.io; ceph-users(a)ceph.io
Subject: [ceph-users] Re: Vitastor, a fast Ceph-like block storage for
VMs
I love how it’s not possible to delete inodes yet. Data loss would be a
thing of the past!
Jokes aside, interesting project.
Sent from mobile
Op 23 sep. 2020 om 00:45 heeft vitalif(a)yourcmc.ru het
volgende
geschreven:
Hi!
After almost a year of development in my spare time I present my own
software-defined block storage system: Vitastor - https://vitastor.io
I designed it similar to Ceph in many ways, it also has Pools, PGs,
OSDs,
different coding schemes, rebalancing and so on. However it's much
simpler and much faster. In a test cluster with SATA SSDs it achieved
Q1T1 latency of 0.14ms which is especially great compared to Ceph RBD's
1ms for writes and 0.57ms for reads. In an "iops saturation" parallel
load benchmark it reached 895k read / 162k write iops, compared to
Ceph's 480k / 100k on the same hardware, but the most interesting part
was CPU usage: Ceph OSDs were using 40 CPU cores out of 64 on each node
and Vitastor was only using 4.
Of course it's an early pre-release which means that, for example, it
lacks
snapshot support and other useful features. However the base is
finished - it works and runs QEMU VMs. I like the design and I plan to
develop it further.
There are more details in the README file which currently opens from
the domain https://vitastor.io
Sorry if it was a bit off-topic, I just thought it could be
interesting for you :)
--
With best regards,
Vitaliy Filippov
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
10:06 a.m.
Thanks Marc :)
It's easier to write code than to cooperate :) I can do whatever I want in my own
project.
Ceph is rather complex. For example, I failed to find bottlenecks in OSD when I tried to
profile it - I'm not an expert of course, but still... The only bottleneck I found was
cephx_sign_messages=true by default. Now I always disable it. In fact I don't think
Ceph needs those signatures at all because 99.9% of setups live in private networks. Ceph
has ~1M lines of code. Vitastor has 22k :). Bluestore is complicated, SeaStore seems like
it may also end up being complicated, there are a lot of other architectural things like
RBD cache, RBD object map, immediate commit semantics for all writes and so on that
can't be easily fixed. It would take MUCH more than a year to fix everything.
Ceph is great for object storage, but 1ms write latency in an NVMe cluster is something
that annoyed me so much that I basically had to try to reinvent the wheel. So I hope my
wheel will make its way into production at some point :)
> Vitaliy you are crazy ;) But really cool work. Why not combine efforts
> with ceph? Especially with something as important as SDS and PB's of
> clients data stored on it, everyone with a little bit of brain chooses a
> solution from a 'reliable' source. For me it was decisive to learn that
> CERN and NASA were using this on a large scale. I do not have the
> expertise nor time (like probably 90% of ceph users) to test how they
> have been testing and using ceph.
>
> I often see opensource projects that could benefit from cooperation.
> Some teams totally lack the expertise that others have, and vice versa.
> Providing the community with 10 or 20 'shitty' projects instead of 3
> 'good' projects.
> I think opensource projects should more often embrace a sort of modular
> development solution. Where others can change functionality by replacing
> just a module. If I ever get my idea funded, I would make it like this.
1323
days inactive
1324
days old
9 comments
6 participants
participants (6)
-
Alexander E. Patrakov -
Lindsay Mathieson -
Marc Roos -
vitalif＠yourcmc.ru -
William Edwards -
Виталий Филиппов