26 Sep
2020
26 Sep
'20
11:26 p.m.
Hi Team,
I'm interested in creating a Go-native version of librados. To that end,
I'm looking for sources of information about the RADOS client protocol
(beyond what's on https://docs.ceph.com/en/latest/dev/rados-client-protocol/).
Could you point me to some good places to start?
Thanks for the help!
--
Sanford Miller
Engineer I
7735176031
smiller(a)digitalocean.com
------------------------------
We're Hiring! <https://kmlnsr.me/do-sig-YV4oLB.html> | @digitalocean
<https://twitter.com/digitalocean> |linkedin
<https://www.linkedin.com/in/sanford-miller-750044142/>
27 Sep
27 Sep
9 p.m.
On Saturday, September 26, 2020 1:56:21 PM EDT Sanford Miller wrote:
Hi Team,
I'm interested in creating a Go-native version of librados.
I'd be curious if you are aware of go-ceph [1], the project to bind ceph
libraries to Go, much like the Python bindings in the ceph tree. There are
librados bindings in the "rados" subpackage.
If you were aware of go-ceph and have chosen not to make use of it, as a
maintainer of that project, I'm curious to hear about why.
To that end,
I'm looking for sources of information about the RADOS client protocol
(beyond what's on
https://docs.ceph.com/en/latest/dev/rados-client-protocol/). Could you
point me to some good places to start?
Thanks for the help!
[1] - https://github.com/ceph/go-ceph
9:54 p.m.
Hi John,
Thanks for the quick response. I am aware of go-ceph. My team has used it
successfully for several control-plane services. However, the performance
penalty imposed by the Cgo runtime makes go-ceph not a good fit for the
data-plane services we'd like to build in the future. The idea for the
library I want to create/start is to have serialization and
deserialization, of both the RADOS client protocol and of the messenger
protocol wrapping it, done natively in Go. Hence, it would be a huge help
to get some pointers to good sources of information about the RADOS client
protocol.
Thanks,
Sanford
On Sun, Sep 27, 2020 at 10:30 AM John Mulligan <phlogistonjohn(a)asynchrono.us>
wrote:
On Saturday, September 26, 2020 1:56:21 PM EDT Sanford
Miller wrote:
--
Sanford Miller
Engineer I
smiller(a)digitalocean.com
------------------------------
We're Hiring! <https://kmlnsr.me/do-sig-YV4oLB.html> | @digitalocean
<https://twitter.com/digitalocean> |linkedin
<https://www.linkedin.com/in/sanford-miller-750044142/>
Hi Team,
I'm interested in creating a Go-native version of librados.
I'd be curious if you are aware of go-ceph [1], the project to bind ceph
libraries to Go, much like the Python bindings in the ceph tree. There are
librados bindings in the "rados" subpackage.
If you were aware of go-ceph and have chosen not to make use of it, as a
maintainer of that project, I'm curious to hear about why.
To that end,
I'm looking for sources of information about the RADOS client protocol
(beyond what's on
https://docs.ceph.com/en/latest/dev/rados-client-protocol/). Could you
point me to some good places to start?
Thanks for the help!
[1] - https://github.com/ceph/go-ceph
10:11 p.m.
I suspect it's impossible to notice cgo overhead with Ceph :))
> Hi John,
>
> Thanks for the quick response. I am aware of go-ceph. My team has used it
successfully for several
> control-plane services. However, the performance penalty imposed by the Cgo runtime
makes go-ceph
> not a good fit for the data-plane services we'd like to build in the future. The
idea for the
> library I want to create/start is to have serialization and deserialization, of both
the RADOS
> client protocol and of the messenger protocol wrapping it, done natively in Go.
Hence, it would be
> a huge help to get some pointers to good sources of information about the RADOS
client protocol.
>
> Thanks,
>
> Sanford
>
> On Sun, Sep 27, 2020 at 10:30 AM John Mulligan <phlogistonjohn(a)asynchrono.us>
wrote:
>
>> On Saturday, September 26, 2020 1:56:21 PM EDT Sanford Miller wrote:
>>> Hi Team,
>>>
>>> I'm interested in creating a Go-native version of librados.
>>
>> I'd be curious if you are aware of go-ceph [1], the project to bind ceph
>> libraries to Go, much like the Python bindings in the ceph tree. There are
>> librados bindings in the "rados" subpackage.
>>
>> If you were aware of go-ceph and have chosen not to make use of it, as a
>> maintainer of that project, I'm curious to hear about why.
>>
>>> To that end,
>>> I'm looking for sources of information about the RADOS client protocol
>>> (beyond what's on
>>> https://docs.ceph.com/en/latest/dev/rados-client-protocol/). Could you
>>> point me to some good places to start?
>>>
>>> Thanks for the help!
>>
>> [1] - https://github.com/ceph/go-ceph
>
> --
> Sanford Miller
> Engineer I
> smiller(a)digitalocean.com
>
> _______________________________
>
> We're Hiring! | @digitalocean |linkedin
10:42 p.m.
On Sunday, September 27, 2020 12:41:32 PM EDT vitalif(a)yourcmc.ru wrote:
I suspect it's impossible to notice cgo overhead
with Ceph :))
I would be a bit somewhat surprised to hear if it really was cgo itself too.
But I have some additional thoughts on that below.
> Hi John,
>
> Thanks for the quick response. I am aware of go-ceph. My team has used it
> successfully for several control-plane services. However, the performance
> penalty imposed by the Cgo runtime makes go-ceph not a good fit for the
> data-plane services we'd like to build in the future.
Sanford, do you have any benchmarks or any pointers to what methods you tested
and were too slow for your use cases? It may be that the go-ceph
implementation performs too many data copies. Some of the methods that work
with strings rather than byte slices as buffers need to do copies between C/C+
+ memory to Go memory and we may be incurring too large of a performance
penalty for that.
Even if you don't use go-ceph for your future projects we'd like to know where
we can improve. Right now, I've been more focused on expanding our coverage of
the functionality in the lib{rados,rbd,cephfs} wrapper packages, but being
efficient is important too.
> The idea for the
> library I want to create/start is to have serialization and
> deserialization, of both the RADOS client protocol and of the messenger
> protocol wrapping it, done natively in Go. Hence, it would be a huge help
> to get some pointers to good sources of information about the RADOS
> client protocol.
I'll certainly want to keep tabs on any effort here if you go forward with it.
For my use cases using wrappers gets us to our goals faster but there's
certainly some trade offs we make when we use cgo. One big one for me is the
lack of deeper integration with goroutines and the difficulty using common Go
patterns like deadlines on context.Context.
Thanks!
1 Oct
1 Oct
7:57 p.m.
Right now, the focus of my project is a subset of the Ceph wire protocol
and of the RADOS client protocol. The idea is to create a library that
implements the basic operations for interacting with a RADOS cluster
natively in Go. Pure Go will be easily maintainable by my company, which is
high on Go expertise and low on knowledge of C++, and will also be
integratable into our existing systems/conventions (such as using
context.Context). The library will also hopefully make it easier to build
services that are particularly sensitive to the performance of the
operations that they perform mostly frequently against the cluster.
I'm starting with getting and putting a RADOS object, and will see where to
go from there. In the short term, I need help with the semantics of the
members of the MOSDOp/MOSDOpReply classes. Where might I find information
about the appropriate values to set for client_inc, flags, hobj, features,
and bdata_encode?
On Sun, Sep 27, 2020 at 12:12 PM John Mulligan <phlogistonjohn(a)asynchrono.us>
wrote:
On Sunday, September 27, 2020 12:41:32 PM EDT
vitalif(a)yourcmc.ru wrote:
--
Sanford Miller
Engineer I
smiller(a)digitalocean.com
------------------------------
We're Hiring! <https://kmlnsr.me/do-sig-YV4oLB.html> | @digitalocean
<https://twitter.com/digitalocean> |linkedin
<https://www.linkedin.com/in/sanford-miller-750044142/>
I suspect it's impossible to notice cgo
overhead with Ceph :))
I would be a bit somewhat surprised to hear if it really was cgo itself
too.
But I have some additional thoughts on that below.
> Hi John,
>
> Thanks for the quick response. I am aware of go-ceph. My team has used
it
> successfully for several control-plane
services. However, the
performance
> penalty imposed by the Cgo runtime makes
go-ceph not a good fit for the
> data-plane services we'd like to build in the future.
Sanford, do you have any benchmarks or any pointers to what methods you
tested
and were too slow for your use cases? It may be that the go-ceph
implementation performs too many data copies. Some of the methods that
work
with strings rather than byte slices as buffers need to do copies between
C/C+
+ memory to Go memory and we may be incurring too large of a performance
penalty for that.
Even if you don't use go-ceph for your future projects we'd like to know
where
we can improve. Right now, I've been more focused on expanding our
coverage of
the functionality in the lib{rados,rbd,cephfs} wrapper packages, but being
efficient is important too.
> The idea for the
> library I want to create/start is to have serialization and
> deserialization, of both the RADOS client protocol and of the messenger
> protocol wrapping it, done natively in Go. Hence, it would be a huge
help
> to get some pointers to good sources of
information about the RADOS
> client protocol.
I'll certainly want to keep tabs on any effort here if you go forward with
it.
For my use cases using wrappers gets us to our goals faster but there's
certainly some trade offs we make when we use cgo. One big one for me is
the
lack of deeper integration with goroutines and the difficulty using common
Go
patterns like deadlines on context.Context.
Thanks!
1301
days inactive
1306
days old
5 comments
3 participants
participants (3)
-
John Mulligan -
Sanford Miller -
vitalif＠yourcmc.ru