I suspect it's impossible to notice cgo overhead with Ceph :))

Hi John,

Thanks for the quick response. I am aware of go-ceph. My team has used it successfully for several control-plane services. However, the performance penalty imposed by the Cgo runtime makes go-ceph not a good fit for the data-plane services we'd like to build in the future. The idea for the library I want to create/start is to have serialization and deserialization, of both the RADOS client protocol and of the messenger protocol wrapping it, done natively in Go. Hence, it would be a huge help to get some pointers to good sources of information about the RADOS client protocol.

Thanks,

Sanford

On Sun, Sep 27, 2020 at 10:30 AM John Mulligan <phlogistonjohn@asynchrono.us> wrote:

...

On Saturday, September 26, 2020 1:56:21 PM EDT Sanford Miller wrote:

...

Hi Team,

I'm interested in creating a Go-native version of librados.

I'd be curious if you are aware of go-ceph [1], the project to bind ceph libraries to Go, much like the Python bindings in the ceph tree. There are librados bindings in the "rados" subpackage.

If you were aware of go-ceph and have chosen not to make use of it, as a maintainer of that project, I'm curious to hear about why.

...

To that end, I'm looking for sources of information about the RADOS client protocol (beyond what's on https://docs.ceph.com/en/latest/dev/rados-client-protocol/). Could you point me to some good places to start?

Thanks for the help!

[1] - https://github.com/ceph/go-ceph

-- Sanford Miller Engineer I smiller@digitalocean.com

_______________________________

We're Hiring! | @digitalocean |linkedin

Right now, the focus of my project is a subset of the Ceph wire protocol and of the RADOS client protocol. The idea is to create a library that implements the basic operations for interacting with a RADOS cluster natively in Go. Pure Go will be easily maintainable by my company, which is high on Go expertise and low on knowledge of C++, and will also be integratable into our existing systems/conventions (such as using context.Context). The library will also hopefully make it easier to build services that are particularly sensitive to the performance of the operations that they perform mostly frequently against the cluster. I'm starting with getting and putting a RADOS object, and will see where to go from there. In the short term, I need help with the semantics of the members of the MOSDOp/MOSDOpReply classes. Where might I find information about the appropriate values to set for client_inc, flags, hobj, features, and bdata_encode? On Sun, Sep 27, 2020 at 12:12 PM John Mulligan <phlogistonjohn@asynchrono.us> wrote:

On Sunday, September 27, 2020 12:41:32 PM EDT vitalif@yourcmc.ru wrote:

...

I suspect it's impossible to notice cgo overhead with Ceph :))

I would be a bit somewhat surprised to hear if it really was cgo itself too. But I have some additional thoughts on that below.

...

...

Hi John,

Thanks for the quick response. I am aware of go-ceph. My team has used it successfully for several control-plane services. However, the performance penalty imposed by the Cgo runtime makes go-ceph not a good fit for the data-plane services we'd like to build in the future.

Sanford, do you have any benchmarks or any pointers to what methods you tested and were too slow for your use cases? It may be that the go-ceph implementation performs too many data copies. Some of the methods that work with strings rather than byte slices as buffers need to do copies between C/C+ + memory to Go memory and we may be incurring too large of a performance penalty for that.

Even if you don't use go-ceph for your future projects we'd like to know where we can improve. Right now, I've been more focused on expanding our coverage of the functionality in the lib{rados,rbd,cephfs} wrapper packages, but being efficient is important too.

...

...

The idea for the library I want to create/start is to have serialization and deserialization, of both the RADOS client protocol and of the messenger protocol wrapping it, done natively in Go. Hence, it would be a huge help to get some pointers to good sources of information about the RADOS client protocol.

I'll certainly want to keep tabs on any effort here if you go forward with it. For my use cases using wrappers gets us to our goals faster but there's certainly some trade offs we make when we use cgo. One big one for me is the lack of deeper integration with goroutines and the difficulty using common Go patterns like deadlines on context.Context.

Thanks!

-- Sanford Miller Engineer I smiller@digitalocean.com ------------------------------ We're Hiring! <https://kmlnsr.me/do-sig-YV4oLB.html> | @digitalocean <https://twitter.com/digitalocean> |linkedin <https://www.linkedin.com/in/sanford-miller-750044142/>