Vitastor, a fast Ceph-like block storage for VMs
Hi! After almost a year of development in my spare time I present my own software-defined block storage system: Vitastor - https://vitastor.io I designed it similar to Ceph in many ways, it also has Pools, PGs, OSDs, different coding schemes, rebalancing and so on. However it's much simpler and much faster. In a test cluster with SATA SSDs it achieved Q1T1 latency of 0.14ms which is especially great compared to Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops saturation" parallel load benchmark it reached 895k read / 162k write iops, compared to Ceph's 480k / 100k on the same hardware, but the most interesting part was CPU usage: Ceph OSDs were using 40 CPU cores out of 64 on each node and Vitastor was only using 4. Of course it's an early pre-release which means that, for example, it lacks snapshot support and other useful features. However the base is finished - it works and runs QEMU VMs. I like the design and I plan to develop it further. There are more details in the README file which currently opens from the domain https://vitastor.io Sorry if it was a bit off-topic, I just thought it could be interesting for you :) -- With best regards, Vitaliy Filippov
On Wed, Sep 23, 2020 at 3:44 AM <vitalif@yourcmc.ru> wrote:
Hi!
After almost a year of development in my spare time I present my own software-defined block storage system: Vitastor - https://vitastor.io
I designed it similar to Ceph in many ways, it also has Pools, PGs, OSDs, different coding schemes, rebalancing and so on. However it's much simpler and much faster. In a test cluster with SATA SSDs it achieved Q1T1 latency of 0.14ms which is especially great compared to Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops saturation" parallel load benchmark it reached 895k read / 162k write iops, compared to Ceph's 480k / 100k on the same hardware, but the most interesting part was CPU usage: Ceph OSDs were using 40 CPU cores out of 64 on each node and Vitastor was only using 4.
Of course it's an early pre-release which means that, for example, it lacks snapshot support and other useful features. However the base is finished - it works and runs QEMU VMs. I like the design and I plan to develop it further.
There are more details in the README file which currently opens from the domain https://vitastor.io
Very interesting. Could you please add more details to the README file, as listed below? 1. Network benchmarks, in terms of achievable throughput and latency. 2. The type of the switch you used, and if there was any latency tuning, please state it. 3. The network MTU. 4. The utilization figures for SSDs and network interfaces during each test. Also, given that the scope of the project only includes block storage, I think it would be fair to ask for a comparison with DRBD 9 and possibly Linstor, not only with Ceph. -- Alexander E. Patrakov CV: http://pc.cd/PLz7
On 23/09/2020 8:44 am, vitalif@yourcmc.ru wrote:
After almost a year of development in my spare time I present my own software-defined block storage system: Vitastor -https://vitastor.io
Interesting, thanks. It supports qemu connecting via rbd? -- Lindsay
On 23/09/2020 8:44 am, vitalif@yourcmc.ru wrote:
There are more details in the README file which currently opens from the domainhttps://vitastor.io
that redirects to https://yourcmc.ru/git/vitalif/vitastor Is that your own site? -- Lindsay
Yes Not RBD, it has an own qemu driver 23 сентября 2020 г. 3:24:23 GMT+03:00, Lindsay Mathieson <lindsay.mathieson@gmail.com> пишет:
There are more details in the README file which currently opens from
On 23/09/2020 8:44 am, vitalif@yourcmc.ru wrote: the domainhttps://vitastor.io
that redirects to https://yourcmc.ru/git/vitalif/vitastor
Is that your own site?
-- Lindsay _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- With best regards, Vitaliy Filippov
It succeeds to load with LD_PRELOAD as, as I understand, block_register() gets called. QAPI and new QAPI-based block device syntax don't work though because they're based on IDLs built into QEMU... QAPI will require patching, yeah. It would be nicer if QAPI supported plugins too... :-) -- With best regards, Vitaliy Filippov
I love how it’s not possible to delete inodes yet. Data loss would be a thing of the past! Jokes aside, interesting project. Sent from mobile
Op 23 sep. 2020 om 00:45 heeft vitalif@yourcmc.ru het volgende geschreven:
Hi!
After almost a year of development in my spare time I present my own software-defined block storage system: Vitastor - https://vitastor.io
I designed it similar to Ceph in many ways, it also has Pools, PGs, OSDs, different coding schemes, rebalancing and so on. However it's much simpler and much faster. In a test cluster with SATA SSDs it achieved Q1T1 latency of 0.14ms which is especially great compared to Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops saturation" parallel load benchmark it reached 895k read / 162k write iops, compared to Ceph's 480k / 100k on the same hardware, but the most interesting part was CPU usage: Ceph OSDs were using 40 CPU cores out of 64 on each node and Vitastor was only using 4.
Of course it's an early pre-release which means that, for example, it lacks snapshot support and other useful features. However the base is finished - it works and runs QEMU VMs. I like the design and I plan to develop it further.
There are more details in the README file which currently opens from the domain https://vitastor.io
Sorry if it was a bit off-topic, I just thought it could be interesting for you :)
-- With best regards, Vitaliy Filippov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Vitaliy you are crazy ;) But really cool work. Why not combine efforts with ceph? Especially with something as important as SDS and PB's of clients data stored on it, everyone with a little bit of brain chooses a solution from a 'reliable' source. For me it was decisive to learn that CERN and NASA were using this on a large scale. I do not have the expertise nor time (like probably 90% of ceph users) to test how they have been testing and using ceph. I often see opensource projects that could benefit from cooperation. Some teams totally lack the expertise that others have, and vice versa. Providing the community with 10 or 20 'shitty' projects instead of 3 'good' projects. I think opensource projects should more often embrace a sort of modular development solution. Where others can change functionality by replacing just a module. If I ever get my idea funded, I would make it like this. -----Original Message----- Cc: dev@ceph.io; ceph-users@ceph.io Subject: [ceph-users] Re: Vitastor, a fast Ceph-like block storage for VMs I love how it’s not possible to delete inodes yet. Data loss would be a thing of the past! Jokes aside, interesting project. Sent from mobile
Op 23 sep. 2020 om 00:45 heeft vitalif@yourcmc.ru het volgende geschreven:
Hi!
After almost a year of development in my spare time I present my own software-defined block storage system: Vitastor - https://vitastor.io
I designed it similar to Ceph in many ways, it also has Pools, PGs, OSDs, different coding schemes, rebalancing and so on. However it's much simpler and much faster. In a test cluster with SATA SSDs it achieved Q1T1 latency of 0.14ms which is especially great compared to Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops saturation" parallel load benchmark it reached 895k read / 162k write iops, compared to Ceph's 480k / 100k on the same hardware, but the most interesting part was CPU usage: Ceph OSDs were using 40 CPU cores out of 64 on each node and Vitastor was only using 4.
Of course it's an early pre-release which means that, for example, it lacks snapshot support and other useful features. However the base is finished - it works and runs QEMU VMs. I like the design and I plan to develop it further.
There are more details in the README file which currently opens from the domain https://vitastor.io
Sorry if it was a bit off-topic, I just thought it could be interesting for you :)
-- With best regards, Vitaliy Filippov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Marc :) It's easier to write code than to cooperate :) I can do whatever I want in my own project. Ceph is rather complex. For example, I failed to find bottlenecks in OSD when I tried to profile it - I'm not an expert of course, but still... The only bottleneck I found was cephx_sign_messages=true by default. Now I always disable it. In fact I don't think Ceph needs those signatures at all because 99.9% of setups live in private networks. Ceph has ~1M lines of code. Vitastor has 22k :). Bluestore is complicated, SeaStore seems like it may also end up being complicated, there are a lot of other architectural things like RBD cache, RBD object map, immediate commit semantics for all writes and so on that can't be easily fixed. It would take MUCH more than a year to fix everything. Ceph is great for object storage, but 1ms write latency in an NVMe cluster is something that annoyed me so much that I basically had to try to reinvent the wheel. So I hope my wheel will make its way into production at some point :)
Vitaliy you are crazy ;) But really cool work. Why not combine efforts with ceph? Especially with something as important as SDS and PB's of clients data stored on it, everyone with a little bit of brain chooses a solution from a 'reliable' source. For me it was decisive to learn that CERN and NASA were using this on a large scale. I do not have the expertise nor time (like probably 90% of ceph users) to test how they have been testing and using ceph.
I often see opensource projects that could benefit from cooperation. Some teams totally lack the expertise that others have, and vice versa. Providing the community with 10 or 20 'shitty' projects instead of 3 'good' projects. I think opensource projects should more often embrace a sort of modular development solution. Where others can change functionality by replacing just a module. If I ever get my idea funded, I would make it like this.
participants (6)
-
Alexander E. Patrakov
-
Lindsay Mathieson
-
Marc Roos
-
vitalif＠yourcmc.ru
-
William Edwards
-
Виталий Филиппов