Hi Roman, Yes, you're right - OSDs list all objects during peering and take the latest full version of each object. Full version is a version that has at least min_size parts for XOR/EC, or any version for replicated setups which is OK because writes are atomic. If there is a newer "committed" ("stabilized") version of the object, but it doesn't have sufficient number of parts, the object is marked as incomplete. By the way, did you receive my reply that I sent directly to your email? - it could've ended up in spam because yourcmc.ru is my own mail server. > Hi Vitaliy, > > Awesome what you did. I like the minimalistic approach focused on > performance very much. A long way you've passed implementing the > whole stack from client to storage. > > Could you please describe peering process and object recovery in more > details? You write in the README that there is no PGLog in your > storage. After a brief glance into osd_ops.h and blockstore_impl.cpp I > see there is a 'version' field. I can assume that every object is > marked with a version and recovery happens taking the whole up-to-date > object, right? > > -- > Roman > > On 2020-09-23 00:44, vitalif(a)yourcmc.ru wrote: > >> Hi! >> After almost a year of development in my spare time I present my own >> software-defined block storage system: Vitastor - https://vitastor.io >> I designed it similar to Ceph in many ways, it also has Pools, PGs, >> OSDs, different coding schemes, rebalancing and so on. However it's >> much simpler and much faster. In a test cluster with SATA SSDs it >> achieved Q1T1 latency of 0.14ms which is especially great compared to >> Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops >> saturation" parallel load benchmark it reached 895k read / 162k write >> iops, compared to Ceph's 480k / 100k on the same hardware, but the >> most interesting part was CPU usage: Ceph OSDs were using 40 CPU cores >> out of 64 on each node and Vitastor was only using 4. >> Of course it's an early pre-release which means that, for example, it >> lacks snapshot support and other useful features. However the base is >> finished - it works and runs QEMU VMs. I like the design and I plan to >> develop it further. >> There are more details in the README file which currently opens from >> the domain https://vitastor.io >> Sorry if it was a bit off-topic, I just thought it could be >> interesting for you :)