22 Jul
2019
22 Jul
'19
2:37 p.m.
hi Chunmei,
i am reviewing your change of
https://github.com/ceph/ceph/compare/master...liu-chunmei:ceph_seastar_alie….
it looks good in general. i think the simplest way to co-locate
different versions of alien-common, ceph-common and crimson-common is
to introduce different namespaces. because we need to have
alien-common and crimson-common in the same binary, and to have all of
these three versions in the same repository.
but this divergence concerns me, as it introduces yet another
condition in the shared infrastructure in our code base. and in the
long run, this #ifdef won't go away if we want to go this way, so i
need to at least give it a try. what is "it"? to port rocksdb to
seastar. as seastar offers "seastar::thread" which makes it relative
simpler to wrap the blocking calls with ucontext. and rocksdb offers a
abstraction machinery allowing one to port it to a new platform. and
seastar is a "platform" to some degree, i'd say.
will update you guys with my progress and findings.
--
Regards
Kefu Chai
23 Jul
23 Jul
10:21 p.m.
Hi kefu,
Even we use seastar::thread, I think we still need #ifdef macro to build these code
together, am I right? The same binary still contain both blocking code and unblocking
code.
Let's discuss it in the meeting.
-Chunmei
-----Original Message-----
From: kefu chai [mailto:tchaikov@gmail.com]
Sent: Monday, July 22, 2019 7:37 AM
To: Liu, Chunmei <chunmei.liu(a)intel.com>om>; dev(a)ceph.io
Subject: [crimson] bluestore in an alien world
hi Chunmei,
i am reviewing your change of
https://github.com/ceph/ceph/compare/master...liu-
chunmei:ceph_seastar_alien_store.
it looks good in general. i think the simplest way to co-locate different versions
of alien-common, ceph-common and crimson-common is to introduce different
namespaces. because we need to have alien-common and crimson-common in
the same binary, and to have all of these three versions in the same repository.
but this divergence concerns me, as it introduces yet another condition in the
shared infrastructure in our code base. and in the long run, this #ifdef won't go
away if we want to go this way, so i need to at least give it a try. what is
"it"? to
port rocksdb to seastar. as seastar offers "seastar::thread" which makes it
relative simpler to wrap the blocking calls with ucontext. and rocksdb offers a
abstraction machinery allowing one to port it to a new platform. and seastar is a
"platform" to some degree, i'd say.
will update you guys with my progress and findings.
--
Regards
Kefu Chai
26 Jul
26 Jul
4 p.m.
On Wed, Jul 24, 2019 at 6:21 AM Liu, Chunmei <chunmei.liu(a)intel.com> wrote:
Hi kefu,
Even we use seastar::thread, I think we still need #ifdef macro to build these code
together, am I right? The same binary still contain both blocking code and unblocking
code.
Let's discuss it in the meeting.
-Chunmei
> -----Original Message-----
> From: kefu chai [mailto:tchaikov@gmail.com]
> Sent: Monday, July 22, 2019 7:37 AM
> To: Liu, Chunmei <chunmei.liu(a)intel.com>om>; dev(a)ceph.io
> Subject: [crimson] bluestore in an alien world
>
> hi Chunmei,
>
> i am reviewing your change of
> https://github.com/ceph/ceph/compare/master...liu-
> chunmei:ceph_seastar_alien_store.
> it looks good in general. i think the simplest way to co-locate different versions
> of alien-common, ceph-common and crimson-common is to introduce different
> namespaces. because we need to have alien-common and crimson-common in
> the same binary, and to have all of these three versions in the same repository.
>
> but this divergence concerns me, as it introduces yet another condition in the
> shared infrastructure in our code base. and in the long run, this #ifdef won't
go
> away if we want to go this way, so i need to at least give it a try. what is
"it"? to
> port rocksdb to seastar. as seastar offers "seastar::thread" which makes
it
> relative simpler to wrap the blocking calls with ucontext. and rocksdb offers a
> abstraction machinery allowing one to port it to a new platform. and seastar is a
> "platform" to some degree, i'd say.
>
> will update you guys with my progress and findings.
i am noting down the takeaways in last crimson standup:
the problem we want to resolve:
- we already have two "common" libraries compiled from the same source
tree: ceph-common and crimson-common. because crimson is pretty much
single-threaded, all mutex, atomic and other synchronizing primitives
are defined as no-op if WITH_SEASTAR is defined. and these two
libraries share the same set of symbol names.
- because rocksdb is desgined around the semantic of blocking calls.
and it does not support seastar at this moment. if we want to continue
using bluestore in crimson-osd, we will have to put bluestore in a
world where the blocked calls are allowed. this world or environment
is dubbed the alien world from seastar's point of view.
- and crimson is targeting fast storage devices. the assumptions made
by rocksdb do not hold anymore there.
some possible solutions
- to run bluestore in a separated process and use mmap to do the IPC
between crimson-osd and bluestore. we can use a ringbuffer to manage
the shared memory, the way how crimson-osd talks to bluestore will be
quite like io-uring. if we can have a customized allocator to so we
can point the msgr to the allocator to avoid memcpy. if we cannot
think of anything that could hurt the performance significantly, this
approach would be
- to run bluestore in the same process of crimson-osd, but we will
allocate some dedicated threads (and CPU cores) to it. we could use
ceph::thread::ThreadPool for this purpose. for instance, we will have
3 ConfigProxy backends.
1. the classic ConfigProxy used by classic OSD and other daemons and
command line utilities. the ConfigProxy normally resides in a global
CephContext.
2. the ceph::common::ConfigProxy solely used by crimson OSD. it is
rewritten using seastar. it's a sharded service. normally we just
access the config proxy directly in crimson, like
'local_conf().get_val< uint64_t>("name")' instead of using
something
like 'cct->_conf.get_val<uint64_t>("name")'
3. the ConfigProxy used by bluestore living in the alien world. its
interface will be exactly the same as the classic one, but it will
call into its crimson counterpart using the `seastar::alien::submit()`
call.
in addition to WITH_SEASTAR macro, we can introduce yet another
macro allowing us to call into the facilities offered by
crimson-common. and we can use inline namespace to differentiate the
2nd from 3rd implementations. as they will need to be co-located in
the same process. and without using different names, we'd violate ODR.
- to hide bluestore in a library which links against ceph-common
library. but the libblustore won't expose any ceph-common symbols to
crimson-osd. but we need to figure out how to maintain the internal
status of ceph-common. as it not quite self-contained in the sense
that it need to access the logging, config and other facilities
offered by crimson-osd.
- to port rocksdb to seastar: to be specific, this approach will use
seastar's green thread to implement the Mutex, CondVar and Thread in
rocksdb, and implement all blocking calls using seastar's
counterparts. if this approach is proved to be workable. the next
problem would be to upstream this change. and in a long run, the
rocksdb backed bluestore will be replaced by seastore if seastore is
capable of supporting relatively slow devices as well.
- seastore: a completely rewritten object store backend targeting fast
NVMe devices. but it will take longer to get there.
>
>
> --
> Regards
> Kefu Chai
--
Regards
Kefu Chai
9:26 p.m.
- to run bluestore in the same process of crimson-osd,
but we will
allocate some dedicated threads (and CPU cores) to it. we could use
ceph::thread::ThreadPool for this purpose. for instance, we will have
3 ConfigProxy backends.
1. the classic ConfigProxy used by classic OSD and other daemons and
command line utilities. the ConfigProxy normally resides in a global
CephContext.
2. the ceph::common::ConfigProxy solely used by crimson OSD. it is
rewritten using seastar. it's a sharded service. normally we just
access the config proxy directly in crimson, like
'local_conf().get_val< uint64_t>("name")' instead of using
something
like 'cct->_conf.get_val<uint64_t>("name")'
3. the ConfigProxy used by bluestore living in the alien world. its
interface will be exactly the same as the classic one, but it will
call into its crimson counterpart using the `seastar::alien::submit()`
call.
I'm not sure this is quite right. I think that the seastar config
would have a reference over to the alien config machinery in order to
inject config changes and do the initial setup, but the alien side
needn't have a reference to the crimson one.
in addition to WITH_SEASTAR macro, we can introduce
yet another
macro allowing us to call into the facilities offered by
crimson-common. and we can use inline namespace to differentiate the
2nd from 3rd implementations. as they will need to be co-located in
the same process. and without using different names, we'd violate ODR.
- to hide bluestore in a library which links against ceph-common
library. but the libblustore won't expose any ceph-common symbols to
crimson-osd. but we need to figure out how to maintain the internal
status of ceph-common. as it not quite self-contained in the sense
that it need to access the logging, config and other facilities
offered by crimson-osd.
The library option seems promising to me if we go this direction. It
can even export an interface which is entirely agnostic of the config
machinery (maybe take a serialized representation of the config
values?) and write to a different log file at first.
- to port rocksdb to seastar: to be specific, this
approach will use
seastar's green thread to implement the Mutex, CondVar and Thread in
rocksdb, and implement all blocking calls using seastar's
counterparts. if this approach is proved to be workable. the next
problem would be to upstream this change. and in a long run, the
rocksdb backed bluestore will be replaced by seastore if seastore is
capable of supporting relatively slow devices as well.
I've started to look at your rocksdb port. It does look like the
parts we'd need to adapt are appropriately factored out in rocksdb,
and I bet we'd get interest from upstream. We might want to take
their temperature sooner rather than later? We'd also have to perform
essentially the same refactor in Bluestore in order to break the
bluestore logic apart from the IO/blocking/locking portions. I guess
this exists in some form with the BlockDevice interface, but we'll
also have to introduce something like rocksdb's lock replacement.
This path would get us a much more cooperative (probably more
performant as well, particularly in high density hosts) bluestore in
the long run, so it might be worth the work.
- seastore: a completely rewritten object store
backend targeting fast
NVMe devices. but it will take longer to get there.
I think we're going to do this no matter what. I think
alien/bluestore choice is about how we want to test crimson prior to
developing seastore and possibly for handling devices inappropriate
for seastore?
-Sam
29 Jul
29 Jul
1:19 p.m.
On Sat, Jul 27, 2019 at 5:26 AM Sam Just <sjust(a)redhat.com> wrote:
i was thinking about the implementation of ConfigProxy::get_val<>().
but yeah, if we 1) have a separated copy of ConfigValue on the alien
side, 2) let the alien side work in the passive mode, and 3) use the
ThreadPool::submit() to inject config changes into alien's
ConfigProxy, what'd be a lot easier.
yeah, probably we just need a "keyhole" for updating the alien side's
config settings. this option actually is a variant of the previous
one. the only difference is that, we need to use different namespaces
to differentiate the symbols in bluestore from those in
crimson-common.
good idea! will do so early tomorrow!
that's also my impression. the way how i see it is just because we
haven't started scoping it or had a low level design.
- to run bluestore in the same process of
crimson-osd, but we will
allocate some dedicated threads (and CPU cores) to it. we could use
ceph::thread::ThreadPool for this purpose. for instance, we will have
3 ConfigProxy backends.
1. the classic ConfigProxy used by classic OSD and other daemons and
command line utilities. the ConfigProxy normally resides in a global
CephContext.
2. the ceph::common::ConfigProxy solely used by crimson OSD. it is
rewritten using seastar. it's a sharded service. normally we just
access the config proxy directly in crimson, like
'local_conf().get_val< uint64_t>("name")' instead of using
something
like 'cct->_conf.get_val<uint64_t>("name")'
3. the ConfigProxy used by bluestore living in the alien world. its
interface will be exactly the same as the classic one, but it will
call into its crimson counterpart using the `seastar::alien::submit()`
call.
I'm not sure this is quite right. I think that the seastar config
would have a reference over to the alien config machinery in order to
inject config changes and do the initial setup, but the alien side
needn't have a reference to the crimson one. in addition to WITH_SEASTAR macro, we can
introduce yet another
macro allowing us to call into the facilities offered by
crimson-common. and we can use inline namespace to differentiate the
2nd from 3rd implementations. as they will need to be co-located in
the same process. and without using different names, we'd violate ODR.
- to hide bluestore in a library which links against ceph-common
library. but the libblustore won't expose any ceph-common symbols to
crimson-osd. but we need to figure out how to maintain the internal
status of ceph-common. as it not quite self-contained in the sense
that it need to access the logging, config and other facilities
offered by crimson-osd.
The library option seems promising to me if we go this direction. It
can even export an interface which is entirely agnostic of the config
machinery (maybe take a serialized representation of the config
values?) and write to a different log file at first. - to port rocksdb to seastar: to be specific,
this approach will use
seastar's green thread to implement the Mutex, CondVar and Thread in
rocksdb, and implement all blocking calls using seastar's
counterparts. if this approach is proved to be workable. the next
problem would be to upstream this change. and in a long run, the
rocksdb backed bluestore will be replaced by seastore if seastore is
capable of supporting relatively slow devices as well.
I've started to look at your rocksdb port. It does look like the
parts we'd need to adapt are appropriately factored out in rocksdb,
and I bet we'd get interest from upstream. We might want to take
their temperature sooner rather than later? We'd also have to perform essentially the same refactor in Bluestore in order to
break the
bluestore logic apart from the IO/blocking/locking portions. I guess
this exists in some form with the BlockDevice interface, but we'll
also have to introduce something like rocksdb's lock replacement.
This path would get us a much more cooperative (probably more
performant as well, particularly in high density hosts) bluestore in
the long run, so it might be worth the work.
thanks. your insights are inspiring!
- seastore: a completely rewritten object store
backend targeting fast
NVMe devices. but it will take longer to get there.
I think we're going to do this no matter what. I think
alien/bluestore choice is about how we want to test crimson prior to
developing seastore and possibly for handling devices inappropriate
for seastore? -Sam
--
Regards
Kefu Chai
1 Aug
1 Aug
7:44 a.m.
On Mon, Jul 29, 2019 at 9:19 PM kefu chai <tchaikov(a)gmail.com> wrote:
that's also my impression. the way how i see it is just because we
haven't started scoping it or had a low level design.
--
Regards
Kefu Chai
On Sat, Jul 27, 2019 at 5:26 AM Sam Just <sjust(a)redhat.com> wrote:
i was thinking about the implementation of ConfigProxy::get_val<>().
but yeah, if we 1) have a separated copy of ConfigValue on the alien
side, 2) let the alien side work in the passive mode, and 3) use the
ThreadPool::submit() to inject config changes into alien's
ConfigProxy, what'd be a lot easier.
yeah, probably we just need a "keyhole" for updating the alien side's
config settings. this option actually is a variant of the previous
one. the only difference is that, we need to use different namespaces
to differentiate the symbols in bluestore from those in
crimson-common.
good idea! will do so early tomorrow!
the test of `env_seastar_test` passed, so it kinda works. and i also
wrote a post in https://www.facebook.com/groups/rocksdb.dev/ to get
the opinions from the upstream community.
- to run bluestore in the same process of
crimson-osd, but we will
allocate some dedicated threads (and CPU cores) to it. we could use
ceph::thread::ThreadPool for this purpose. for instance, we will have
3 ConfigProxy backends.
1. the classic ConfigProxy used by classic OSD and other daemons and
command line utilities. the ConfigProxy normally resides in a global
CephContext.
2. the ceph::common::ConfigProxy solely used by crimson OSD. it is
rewritten using seastar. it's a sharded service. normally we just
access the config proxy directly in crimson, like
'local_conf().get_val< uint64_t>("name")' instead of using
something
like 'cct->_conf.get_val<uint64_t>("name")'
3. the ConfigProxy used by bluestore living in the alien world. its
interface will be exactly the same as the classic one, but it will
call into its crimson counterpart using the `seastar::alien::submit()`
call.
I'm not sure this is quite right. I think that the seastar config
would have a reference over to the alien config machinery in order to
inject config changes and do the initial setup, but the alien side
needn't have a reference to the crimson one. in addition to WITH_SEASTAR macro, we can
introduce yet another
macro allowing us to call into the facilities offered by
crimson-common. and we can use inline namespace to differentiate the
2nd from 3rd implementations. as they will need to be co-located in
the same process. and without using different names, we'd violate ODR.
- to hide bluestore in a library which links against ceph-common
library. but the libblustore won't expose any ceph-common symbols to
crimson-osd. but we need to figure out how to maintain the internal
status of ceph-common. as it not quite self-contained in the sense
that it need to access the logging, config and other facilities
offered by crimson-osd.
The library option seems promising to me if we go this direction. It
can even export an interface which is entirely agnostic of the config
machinery (maybe take a serialized representation of the config
values?) and write to a different log file at first. - to port rocksdb to seastar: to be specific,
this approach will use
seastar's green thread to implement the Mutex, CondVar and Thread in
rocksdb, and implement all blocking calls using seastar's
counterparts. if this approach is proved to be workable. the next
problem would be to upstream this change. and in a long run, the
rocksdb backed bluestore will be replaced by seastore if seastore is
capable of supporting relatively slow devices as well.
I've started to look at your rocksdb port. It does look like the
parts we'd need to adapt are appropriately factored out in rocksdb,
and I bet we'd get interest from upstream. We might want to take
their temperature sooner rather than later? We'd also have to perform essentially the same refactor in Bluestore in
order to break the
bluestore logic apart from the IO/blocking/locking portions. I guess
this exists in some form with the BlockDevice interface, but we'll
also have to introduce something like rocksdb's lock replacement.
This path would get us a much more cooperative (probably more
performant as well, particularly in high density hosts) bluestore in
the long run, so it might be worth the work.
thanks. your insights are inspiring! - seastore: a completely rewritten object store
backend targeting fast
NVMe devices. but it will take longer to get there.
I think we're going to do this no matter what. I think
alien/bluestore choice is about how we want to test crimson prior to
developing seastore and possibly for handling devices inappropriate
for seastore? -Sam
--
Regards
Kefu Chai
20 Aug
20 Aug
2:57 p.m.
On Thu, Aug 1, 2019 at 3:44 PM kefu chai <tchaikov(a)gmail.com> wrote:
that's also my impression. the way how i see it is just because we
haven't started scoping it or had a low level design.
--
Regards
Kefu Chai
--
Regards
Kefu Chai
On Mon, Jul 29, 2019 at 9:19 PM kefu chai <tchaikov(a)gmail.com> wrote:
just a quick update.
i was testing the seastar port of rocksdb, its performance does not
look promising in comparison with that of classic rocksdb:
db_bench --benchmarks="fillseq"
seastar rocksdb:
fillseq : 390.297 micros/op 2562 ops/sec; 0.3 MB/s
classic
fillseq : 30.836 micros/op 32429 ops/sec; 3.6 MB/s
i will try to understand this discrepancy. if it turns out to be a
dead end, we will have to focus on one of the options above unless we
have a concrete plan of seastore.
On Sat, Jul 27, 2019 at 5:26 AM Sam Just <sjust(a)redhat.com> wrote:
i was thinking about the implementation of ConfigProxy::get_val<>().
but yeah, if we 1) have a separated copy of ConfigValue on the alien
side, 2) let the alien side work in the passive mode, and 3) use the
ThreadPool::submit() to inject config changes into alien's
ConfigProxy, what'd be a lot easier.
yeah, probably we just need a "keyhole" for updating the alien side's
config settings. this option actually is a variant of the previous
one. the only difference is that, we need to use different namespaces
to differentiate the symbols in bluestore from those in
crimson-common.
good idea! will do so early tomorrow!
the test of `env_seastar_test` passed, so it kinda works. and i also
wrote a post in https://www.facebook.com/groups/rocksdb.dev/ to get
the opinions from the upstream community.
- to run bluestore in the same process of
crimson-osd, but we will
allocate some dedicated threads (and CPU cores) to it. we could use
ceph::thread::ThreadPool for this purpose. for instance, we will have
3 ConfigProxy backends.
1. the classic ConfigProxy used by classic OSD and other daemons and
command line utilities. the ConfigProxy normally resides in a global
CephContext.
2. the ceph::common::ConfigProxy solely used by crimson OSD. it is
rewritten using seastar. it's a sharded service. normally we just
access the config proxy directly in crimson, like
'local_conf().get_val< uint64_t>("name")' instead of using
something
like 'cct->_conf.get_val<uint64_t>("name")'
3. the ConfigProxy used by bluestore living in the alien world. its
interface will be exactly the same as the classic one, but it will
call into its crimson counterpart using the `seastar::alien::submit()`
call.
I'm not sure this is quite right. I think that the seastar config
would have a reference over to the alien config machinery in order to
inject config changes and do the initial setup, but the alien side
needn't have a reference to the crimson one. in addition to WITH_SEASTAR macro, we can
introduce yet another
macro allowing us to call into the facilities offered by
crimson-common. and we can use inline namespace to differentiate the
2nd from 3rd implementations. as they will need to be co-located in
the same process. and without using different names, we'd violate ODR.
- to hide bluestore in a library which links against ceph-common
library. but the libblustore won't expose any ceph-common symbols to
crimson-osd. but we need to figure out how to maintain the internal
status of ceph-common. as it not quite self-contained in the sense
that it need to access the logging, config and other facilities
offered by crimson-osd.
The library option seems promising to me if we go this direction. It
can even export an interface which is entirely agnostic of the config
machinery (maybe take a serialized representation of the config
values?) and write to a different log file at first. - to port rocksdb to seastar: to be specific,
this approach will use
seastar's green thread to implement the Mutex, CondVar and Thread in
rocksdb, and implement all blocking calls using seastar's
counterparts. if this approach is proved to be workable. the next
problem would be to upstream this change. and in a long run, the
rocksdb backed bluestore will be replaced by seastore if seastore is
capable of supporting relatively slow devices as well.
I've started to look at your rocksdb port. It does look like the
parts we'd need to adapt are appropriately factored out in rocksdb,
and I bet we'd get interest from upstream. We might want to take
their temperature sooner rather than later? We'd also have to perform essentially the same refactor in Bluestore in
order to break the
bluestore logic apart from the IO/blocking/locking portions. I guess
this exists in some form with the BlockDevice interface, but we'll
also have to introduce something like rocksdb's lock replacement.
This path would get us a much more cooperative (probably more
performant as well, particularly in high density hosts) bluestore in
the long run, so it might be worth the work.
thanks. your insights are inspiring! - seastore: a completely rewritten object store
backend targeting fast
NVMe devices. but it will take longer to get there.
I think we're going to do this no matter what. I think
alien/bluestore choice is about how we want to test crimson prior to
developing seastore and possibly for handling devices inappropriate
for seastore? -Sam
--
Regards
Kefu Chai
1717
days inactive
1746
days old
6 comments
3 participants
participants (3)
-
kefu chai -
Liu, Chunmei -
Sam Just