25 Jun
2019
25 Jun
'19
4:54 a.m.
Hi,
The ceph-iscsi project [1] has gained a lot of new features and
functionality along with the iSCSI management features that were added
to Ceph Dashboard in Nautilus. The Dashboard is tightly coupled to the
ceph-iscsi config version and needs to be updated in parallel, e.g. when
new functionality is added or existing behavior changes (e.g. here [2],
[3]).
Currently, all ceph-iscsi development is done in the "master" branch of
ceph-iscsi, while the dashboard is developed in the Ceph git repo and
thus is developed and maintained in multiple branches (e.g. "master" and
"nautilus").
This makes it challenging to keep these two components in sync and to
facilitate both maintaining a "stable" version while allowing new
features to be merged. To reduce this complexity and to better interlock
the testing and development of the dashboard and ceph-iscsi, I would
like to propose two possible solutions:
1) merge the ceph-iscsi code base into the ceph git repository. This
way, the development of new features and maintenance would take place in
distinct branches, and in close synchronization between the dashboard
and the ceph-iscsi component. This might also be beneficial for creating
unit tests that test both components without having to assemble the
pieces from various places beforehand. It would also ensure that a
matching ceph-iscsi package is always built and released along with the
corresponding Ceph version, thus offloading the ceph-iscsi devs from the
build and release work. From a version numbering perspective, this
should not be a problem - the current ceph-iscsi packages are of version
"3.0", so they could easily be upgraded to the Ceph versioning scheme.
It would also help with keeping the documentation [4] in sync with the
functionality (the ceph-iscsi docs are actually part of the main Ceph
documentation in the git repo already). This approach is more work, as
it requires finding a way to merge the ceph-iscsi git repo into the ceph
git repo (ideally by preserving the history), and because the packaging
and build scripts need to be updated to build the respective DEB and RPM
packages.
2) As an alternative that requires lesser effort (but also has fewer
benefits), we should at least start using branches within the ceph-iscsi
git repo, so we can do bug fixing on a stable version separate from new
feature development that may introduce incompatible changes. I'd propose
to name the branches according to the Ceph release they support, e.g. by
creating a "nautilus" branch. But the packaging and releasing would
still be disconnected from the Ceph releases (and the work would
increase along with the branches we need to build and support).
I clearly would be in favor of the first proposal, but would need help
in getting this implemented.
Thoughts, concerns, objections?
Would anybody be willing and interested in making this happen?
Thanks!
Lenz
[1] https://github.com/ceph/ceph-iscsi/
[2] https://github.com/ceph/ceph-iscsi/pull/84/
[3] https://github.com/ceph/ceph/pull/28720
[4] http://docs.ceph.com/docs/master/rbd/iscsi-overview/
--
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
25 Jun
25 Jun
5:02 a.m.
On Tue, Jun 25, 2019 at 7:55 AM Lenz Grimmer <lgrimmer(a)suse.com> wrote:
Hi,
The ceph-iscsi project [1] has gained a lot of new features and
functionality along with the iSCSI management features that were added
to Ceph Dashboard in Nautilus. The Dashboard is tightly coupled to the
ceph-iscsi config version and needs to be updated in parallel, e.g. when
new functionality is added or existing behavior changes (e.g. here [2],
[3]).
Currently, all ceph-iscsi development is done in the "master" branch of
ceph-iscsi, while the dashboard is developed in the Ceph git repo and
thus is developed and maintained in multiple branches (e.g. "master" and
"nautilus").
This makes it challenging to keep these two components in sync and to
facilitate both maintaining a "stable" version while allowing new
features to be merged. To reduce this complexity and to better interlock
the testing and development of the dashboard and ceph-iscsi, I would
like to propose two possible solutions:
1) merge the ceph-iscsi code base into the ceph git repository. This
way, the development of new features and maintenance would take place in
distinct branches, and in close synchronization between the dashboard
and the ceph-iscsi component. This might also be beneficial for creating
unit tests that test both components without having to assemble the
pieces from various places beforehand. It would also ensure that a
matching ceph-iscsi package is always built and released along with the
corresponding Ceph version, thus offloading the ceph-iscsi devs from the
build and release work. From a version numbering perspective, this
should not be a problem - the current ceph-iscsi packages are of version
"3.0", so they could easily be upgraded to the Ceph versioning scheme.
It would also help with keeping the documentation [4] in sync with the
functionality (the ceph-iscsi docs are actually part of the main Ceph
documentation in the git repo already). This approach is more work, as
it requires finding a way to merge the ceph-iscsi git repo into the ceph
git repo (ideally by preserving the history), and because the packaging
and build scripts need to be updated to build the respective DEB and RPM
packages.
As we have talked about before, I don't like the idea of merging
everything and the kitchen sink into the "ceph" repo. We would need to
ensure backwards compatibility regardless (be it ceph-iscsi upgrades
one node at a time or the ceph-mgr/dashboard being upgraded first
before ceph-iscsi). Those issues can be addressed by creating a decent
set of unit test cases that simulate the REST interactions expected
from the XYZ version of the dashboard to ceph-iscsi. This isn't the
only sub-project that is tied to the core Ceph project repo (e.g.
ceph-ansible, rook, ceph-csi, go-ceph, ...).
2) As an alternative that requires lesser effort (but
also has fewer
benefits), we should at least start using branches within the ceph-iscsi
git repo, so we can do bug fixing on a stable version separate from new
feature development that may introduce incompatible changes. I'd propose
to name the branches according to the Ceph release they support, e.g. by
creating a "nautilus" branch. But the packaging and releasing would
still be disconnected from the Ceph releases (and the work would
increase along with the branches we need to build and support).
As part of that same discussion, I also agreed that we should create a
"3.x" stable branch along with a new 3.1 release once the last few
known issues are merged. As of today, I believe it's only a single PR
that just needs to get tested and merged.
I clearly would be in favor of the first proposal, but
would need help
in getting this implemented.
Thoughts, concerns, objections?
Would anybody be willing and interested in making this happen?
Thanks!
Lenz
[1] https://github.com/ceph/ceph-iscsi/
[2] https://github.com/ceph/ceph-iscsi/pull/84/
[3] https://github.com/ceph/ceph/pull/28720
[4] http://docs.ceph.com/docs/master/rbd/iscsi-overview/
--
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
--
Jason
5:31 a.m.
On 6/25/19 7:02 AM, Jason Dillaman wrote:
On Tue, Jun 25, 2019 at 7:55 AM Lenz Grimmer
<lgrimmer(a)suse.com> wrote:
Very much agreed! The ceph repo is already far bigger than I'd prefer.
My current ceph working directory is 34GB! (granted most of that is
build). Compile times have ballooned over the years too. Personally
I'd rather go the other way and start removing things that aren't
explicitly required.
>
>> 2) As an alternative that requires lesser effort (but also has fewer
>> benefits), we should at least start using branches within the ceph-iscsi
>> git repo, so we can do bug fixing on a stable version separate from new
>> feature development that may introduce incompatible changes. I'd propose
>> to name the branches according to the Ceph release they support, e.g. by
>> creating a "nautilus" branch. But the packaging and releasing would
>> still be disconnected from the Ceph releases (and the work would
>> increase along with the branches we need to build and support).
> As part of that same discussion, I also agreed that we should create a
> "3.x" stable branch along with a new 3.1 release once the last few
> known issues are merged. As of today, I believe it's only a single PR
> that just needs to get tested and merged.
>
>> I clearly would be in favor of the first proposal, but would need help
>> in getting this implemented.
>>
>> Thoughts, concerns, objections?
>> Would anybody be willing and interested in making this happen?
>>
>> Thanks!
>>
>> Lenz
>>
>> [1] https://github.com/ceph/ceph-iscsi/
>> [2] https://github.com/ceph/ceph-iscsi/pull/84/
>> [3] https://github.com/ceph/ceph/pull/28720
>> [4] http://docs.ceph.com/docs/master/rbd/iscsi-overview/
>> --
>> SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
>> GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
>>
>> _______________________________________________
>> Dev mailing list -- dev(a)ceph.io
>> To unsubscribe send an email to dev-leave(a)ceph.io
>
>
> --
> Jason
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
Hi,
The ceph-iscsi project [1] has gained a lot of new features and
functionality along with the iSCSI management features that were added
to Ceph Dashboard in Nautilus. The Dashboard is tightly coupled to the
ceph-iscsi config version and needs to be updated in parallel, e.g. when
new functionality is added or existing behavior changes (e.g. here [2],
[3]).
Currently, all ceph-iscsi development is done in the "master" branch of
ceph-iscsi, while the dashboard is developed in the Ceph git repo and
thus is developed and maintained in multiple branches (e.g. "master" and
"nautilus").
This makes it challenging to keep these two components in sync and to
facilitate both maintaining a "stable" version while allowing new
features to be merged. To reduce this complexity and to better interlock
the testing and development of the dashboard and ceph-iscsi, I would
like to propose two possible solutions:
1) merge the ceph-iscsi code base into the ceph git repository. This
way, the development of new features and maintenance would take place in
distinct branches, and in close synchronization between the dashboard
and the ceph-iscsi component. This might also be beneficial for creating
unit tests that test both components without having to assemble the
pieces from various places beforehand. It would also ensure that a
matching ceph-iscsi package is always built and released along with the
corresponding Ceph version, thus offloading the ceph-iscsi devs from the
build and release work. From a version numbering perspective, this
should not be a problem - the current ceph-iscsi packages are of version
"3.0", so they could easily be upgraded to the Ceph versioning scheme.
It would also help with keeping the documentation [4] in sync with the
functionality (the ceph-iscsi docs are actually part of the main Ceph
documentation in the git repo already). This approach is more work, as
it requires finding a way to merge the ceph-iscsi git repo into the ceph
git repo (ideally by preserving the history), and because the packaging
and build scripts need to be updated to build the respective DEB and RPM
packages.
As we have talked about before, I don't like the idea of merging
everything and the kitchen sink into the "ceph" repo. We would need to
ensure backwards compatibility regardless (be it ceph-iscsi upgrades
one node at a time or the ceph-mgr/dashboard being upgraded first
before ceph-iscsi). Those issues can be addressed by creating a decent
set of unit test cases that simulate the REST interactions expected
from the XYZ version of the dashboard to ceph-iscsi. This isn't the
only sub-project that is tied to the core Ceph project repo (e.g.
ceph-ansible, rook, ceph-csi, go-ceph, ...).
7:08 a.m.
New subject: Removing things not explicitly required? (was: Proposal: ceph-iscsi branch/release concept)
Hi Mark,
On 6/25/19 2:31 PM, Mark Nelson wrote:
Very much agreed! The ceph repo is already far bigger
than I'd prefer.
My current ceph working directory is 34GB! (granted most of that is
build). Compile times have ballooned over the years too. Personally
I'd rather go the other way and start removing things that aren't
explicitly required.
Of course that's something that should be considered/evaluated and I
actually share your pains and concerns regarding the overall size. In
the case of ceph-iscsi, I think the benefits of adding it would outweigh
the additional overhead, but I'm aware it's a fine balance and we don't
seem to have a formalized way in which sub-projects could be evaluated
for promotion/inclusion into the main git repo (or the other way around).
It probably makes sense to check the code base for unnecessary things on
a regular basis. Do you have any specific components in mind? How do you
define "explicitly required"? And by "removing", do you mean deleting
them for good or moving then out of the ceph git repo into a separate one?
Lenz
--
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
7:16 a.m.
New subject: Removing things not explicitly required?
On 6/25/19 9:08 AM, Lenz Grimmer wrote:
Hi Mark,
On 6/25/19 2:31 PM, Mark Nelson wrote:
Some kind of formal process for sub-projects probably makes sense!
Very much agreed! The ceph repo is already far
bigger than I'd prefer.
My current ceph working directory is 34GB! (granted most of that is
build). Compile times have ballooned over the years too. Personally
I'd rather go the other way and start removing things that aren't
explicitly required.
Of course that's something that should be
considered/evaluated and I
actually share your pains and concerns regarding the overall size. In
the case of ceph-iscsi, I think the benefits of adding it would outweigh
the additional overhead, but I'm aware it's a fine balance and we don't
seem to have a formalized way in which sub-projects could be evaluated
for promotion/inclusion into the main git repo (or the other way around).
It probably makes sense to check the code base for unnecessary things on
a regular basis. Do you have any specific components in mind? How do you
define "explicitly required"? And by "removing", do you mean
deleting
them for good or moving then out of the ceph git repo into a separate one?
I'd rather not single anything out, but I think having a formal process
for determining inclusion is a very good idea. No need to delete
anything, but I would very much like to see a smaller ceph git repo in
the future by moving/splitting things that aren't especially tightly
coupled. Other folks are more tolerant of large git repos though, so my
opinion may be in the minority.
Mark
>
> Lenz
>
>
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
6:46 a.m.
Hi Jason,
On 6/25/19 2:02 PM, Jason Dillaman wrote:
As we have talked about before, I don't like the
idea of merging
everything and the kitchen sink into the "ceph" repo.
Sorry if this sounds like a rehash. I know we talked about this before,
but I had a few more thoughts about this in the meanwhile, and I didn't
recall we had come to a definitive agreement that was publicly shared/noted.
We would need to ensure backwards compatibility
regardless (be it
ceph-iscsi upgrades one node at a time or the ceph-mgr/dashboard
being upgraded first before ceph-iscsi). Those issues can be
addressed by creating a decent set of unit test cases that simulate
the REST interactions expected from the XYZ version of the dashboard
to ceph-iscsi.
Agreed, and adding such tests is something that should be done
regardless of where the code resides.
This isn't the only sub-project that is tied to
the core Ceph project
repo (e.g. ceph-ansible, rook, ceph-csi, go-ceph, ...).
Ack. There are many projects in the Ceph ecosystem and I wouldn't
suggest to merge "everything and the kitchen sink", just because it
relates to Ceph in some form or another.
My rationale for this particular proposal was that the ceph-iscsi code
is fairly limited in scope and size and not dependent on any other
external/special tools, while it does have a tight relationship to the
dashboard. To me, it simply feels like it would benefit from a much
closer integration than the other examples you mentioned above, as it
provides an "access protocol handler" that extends the possibilities in
which data can be stored in Ceph (similar to RGW).
As part of that same discussion, I also agreed that we
should create
a "3.x" stable branch along with a new 3.1 release once the last few
known issues are merged. As of today, I believe it's only a single PR
that just needs to get tested and merged.
OK, thanks - fair enough. That would already help to minimize confusion,
as long as we also clarify this in the Ceph docs (e.g. which version to
use and where to obtain the packages that match the corresponding Ceph
version).
Lenz
--
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
7:23 a.m.
On 6/25/19 1:54 PM, Lenz Grimmer wrote:
1) merge the ceph-iscsi code base into the ceph git
repository.
FWIW, there have been a number of articles that promote the "mono repo"
approach and list a number of good reasons for going down that route:
https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-…
https://blog.shippable.com/our-journey-to-microservices-and-a-mono-reposito…
https://github.com/babel/babel/blob/master/doc/design/monorepo.md
Worth a read ;)
Lenz
--
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
10:49 a.m.
Interesting debate you brought up here, Lenz! Wouldn't want to divert
from the original question, but...
The interesting thing is that, as the paper says, BECAUSE of that
monolithic approach Google couldn't keep on using P4 (as it no longer
scaled enough) and had to build their own VCS.
Besides, P4 (not sure if Piper follows the same approach) made really
easy to setup up workspaces with cherry-picked dirs to sync (instead
of whole repo clones). So IMHO the debate should not be only about
mono-repo vs. multi-repos, but centralized mono-repo vs DVCS
multi-repos.
In my experience this is also about the practices & cultures
allowed/fostered by mono-repo vs. multi-repo ecosystems:
- code, dirs & files vs. architecture, modules & interfaces
- atomic (backward-breaking) changes vs. extending functionality while
honoring APIs
- code sharing/reuse/ownership vs. functionality sharing/reuse/ownership *
- grep vs. documentation
- #include dependency management vs. declarative dep. mgmt
- build everything vs. delta builds
- E2E testing vs. integration testing
- 1 tool for everything (VCS) vs. pipelines (VCS, CI, artifact/release
management, CD, etc.)
* mono-repos make really hard to share/spread code beyond the repo,
BTW, here's (https://www.youtube.com/watch?v=W71BTkUbdqE) a recorded
talk about that paper, and it contains a few 'gems' ("Google solves
dependencies by statically linking everything", "can do massive
backward-incompatible changes... atomically").
Ernesto
On Tue, Jun 25, 2019 at 4:24 PM Lenz Grimmer <lgrimmer(a)suse.com> wrote:
>
> On 6/25/19 1:54 PM, Lenz Grimmer wrote:
>
> > 1) merge the ceph-iscsi code base into the ceph git repository.
>
> FWIW, there have been a number of articles that promote the "mono repo"
> approach and list a number of good reasons for going down that route:
>
>
https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-…
>
> https://blog.shippable.com/our-journey-to-microservices-and-a-mono-reposito…
>
> https://github.com/babel/babel/blob/master/doc/design/monorepo.md
>
> Worth a read ;)
>
> Lenz
>
> --
> SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
> GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
>
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
27 Jun
27 Jun
2 a.m.
Hi Ernesto,
thanks for your feedback!
On 6/25/19 7:49 PM, Ernesto Puerta wrote:
Interesting debate you brought up here, Lenz!
Wouldn't want to divert
from the original question, but...
The interesting thing is that, as the paper says, BECAUSE of that
monolithic approach Google couldn't keep on using P4 (as it no longer
scaled enough) and had to build their own VCS.
Because they can :)
I personally think that git still scales quite well for a project of the
size of Ceph (in terms of contributors and code base).
Besides, P4 (not sure if Piper follows the same
approach) made really
easy to setup up workspaces with cherry-picked dirs to sync (instead
of whole repo clones). So IMHO the debate should not be only about
mono-repo vs. multi-repos, but centralized mono-repo vs DVCS
multi-repos.
But that would likely require more drastic changes, wouldn't it?
In my experience this is also about the practices
& cultures
allowed/fostered by mono-repo vs. multi-repo ecosystems:
- code, dirs & files vs. architecture, modules & interfaces
- atomic (backward-breaking) changes vs. extending functionality while
honoring APIs
- code sharing/reuse/ownership vs. functionality sharing/reuse/ownership *
- grep vs. documentation
- #include dependency management vs. declarative dep. mgmt
- build everything vs. delta builds
- E2E testing vs. integration testing
- 1 tool for everything (VCS) vs. pipelines (VCS, CI, artifact/release
management, CD, etc.)
* mono-repos make really hard to share/spread code beyond the repo,
BTW, here's (https://www.youtube.com/watch?v=W71BTkUbdqE) a recorded
talk about that paper, and it contains a few 'gems' ("Google solves
dependencies by statically linking everything", "can do massive
backward-incompatible changes... atomically").
Thanks for sharing your insights! I guess we would have to start with
collecting current pain points and concerns and then figure out if any
of these approaches would help to alleviate them. In my case, I outlined
the reasons for my proposal in my initial message. Having worked on
openATTIC as an independent project before, we really enjoy working on
the dashboard as a "built-in" component. Less surprises...
Lenz
--
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
5:23 a.m.
On Thu, Jun 27, 2019 at 5:00 AM Lenz Grimmer <lgrimmer(a)suse.com> wrote:
Hi Ernesto,
thanks for your feedback!
On 6/25/19 7:49 PM, Ernesto Puerta wrote:
This is a relief to hear (to collect pain points), because the
ceph-volume experience has been tremendously hard: it made testing
slower, it needed to release faster because fixes/improvements
were produced at a rate that the release cycle couldn't accommodate
(this has since slowed down though), and lastly the backporting of
everything in master all the way down to Luminous took
out a good chunk of our time that we never anticipated.
There has been talk of pulling ceph-volume out of the tree but I
frankly doubt that the enormous amount of work it would take would
benefit us. We wouldn't get the "we no longer need to backport" until
Nautilus (!) is no longer
supported.
Having a conversation about this is great though, happy to add more
details of what worked well and what didn't in the case of
ceph-volume.
>
> Lenz
>
> --
> SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
> GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
>
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
Interesting debate you brought up here, Lenz!
Wouldn't want to divert
from the original question, but...
The interesting thing is that, as the paper says, BECAUSE of that
monolithic approach Google couldn't keep on using P4 (as it no longer
scaled enough) and had to build their own VCS.
Because they can :)
I personally think that git still scales quite well for a project of the
size of Ceph (in terms of contributors and code base).
Besides, P4 (not sure if Piper follows the same
approach) made really
easy to setup up workspaces with cherry-picked dirs to sync (instead
of whole repo clones). So IMHO the debate should not be only about
mono-repo vs. multi-repos, but centralized mono-repo vs DVCS
multi-repos.
But that would likely require more drastic changes, wouldn't it?
In my experience this is also about the practices
& cultures
allowed/fostered by mono-repo vs. multi-repo ecosystems:
- code, dirs & files vs. architecture, modules & interfaces
- atomic (backward-breaking) changes vs. extending functionality while
honoring APIs
- code sharing/reuse/ownership vs. functionality sharing/reuse/ownership *
- grep vs. documentation
- #include dependency management vs. declarative dep. mgmt
- build everything vs. delta builds
- E2E testing vs. integration testing
- 1 tool for everything (VCS) vs. pipelines (VCS, CI, artifact/release
management, CD, etc.)
* mono-repos make really hard to share/spread code beyond the repo,
BTW, here's (https://www.youtube.com/watch?v=W71BTkUbdqE) a recorded
talk about that paper, and it contains a few 'gems' ("Google solves
dependencies by statically linking everything", "can do massive
backward-incompatible changes... atomically").
Thanks for sharing your insights! I guess we would have to start with
collecting current pain points and concerns and then figure out if any
of these approaches would help to alleviate them. In my case, I outlined
the reasons for my proposal in my initial message. Having worked on
openATTIC as an independent project before, we really enjoy working on
the dashboard as a "built-in" component. Less surprises...
1765
days inactive
1767
days old
9 comments
5 participants
participants (5)
-
Alfredo Deza -
Ernesto Puerta -
Jason Dillaman -
Lenz Grimmer -
Mark Nelson