Proposal: ceph-iscsi branch/release concept
Hi, The ceph-iscsi project [1] has gained a lot of new features and functionality along with the iSCSI management features that were added to Ceph Dashboard in Nautilus. The Dashboard is tightly coupled to the ceph-iscsi config version and needs to be updated in parallel, e.g. when new functionality is added or existing behavior changes (e.g. here [2], [3]). Currently, all ceph-iscsi development is done in the "master" branch of ceph-iscsi, while the dashboard is developed in the Ceph git repo and thus is developed and maintained in multiple branches (e.g. "master" and "nautilus"). This makes it challenging to keep these two components in sync and to facilitate both maintaining a "stable" version while allowing new features to be merged. To reduce this complexity and to better interlock the testing and development of the dashboard and ceph-iscsi, I would like to propose two possible solutions: 1) merge the ceph-iscsi code base into the ceph git repository. This way, the development of new features and maintenance would take place in distinct branches, and in close synchronization between the dashboard and the ceph-iscsi component. This might also be beneficial for creating unit tests that test both components without having to assemble the pieces from various places beforehand. It would also ensure that a matching ceph-iscsi package is always built and released along with the corresponding Ceph version, thus offloading the ceph-iscsi devs from the build and release work. From a version numbering perspective, this should not be a problem - the current ceph-iscsi packages are of version "3.0", so they could easily be upgraded to the Ceph versioning scheme. It would also help with keeping the documentation [4] in sync with the functionality (the ceph-iscsi docs are actually part of the main Ceph documentation in the git repo already). This approach is more work, as it requires finding a way to merge the ceph-iscsi git repo into the ceph git repo (ideally by preserving the history), and because the packaging and build scripts need to be updated to build the respective DEB and RPM packages. 2) As an alternative that requires lesser effort (but also has fewer benefits), we should at least start using branches within the ceph-iscsi git repo, so we can do bug fixing on a stable version separate from new feature development that may introduce incompatible changes. I'd propose to name the branches according to the Ceph release they support, e.g. by creating a "nautilus" branch. But the packaging and releasing would still be disconnected from the Ceph releases (and the work would increase along with the branches we need to build and support). I clearly would be in favor of the first proposal, but would need help in getting this implemented. Thoughts, concerns, objections? Would anybody be willing and interested in making this happen? Thanks! Lenz [1] https://github.com/ceph/ceph-iscsi/ [2] https://github.com/ceph/ceph-iscsi/pull/84/ [3] https://github.com/ceph/ceph/pull/28720 [4] http://docs.ceph.com/docs/master/rbd/iscsi-overview/ -- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
On Tue, Jun 25, 2019 at 7:55 AM Lenz Grimmer <lgrimmer@suse.com> wrote:
Hi,
The ceph-iscsi project [1] has gained a lot of new features and functionality along with the iSCSI management features that were added to Ceph Dashboard in Nautilus. The Dashboard is tightly coupled to the ceph-iscsi config version and needs to be updated in parallel, e.g. when new functionality is added or existing behavior changes (e.g. here [2], [3]).
Currently, all ceph-iscsi development is done in the "master" branch of ceph-iscsi, while the dashboard is developed in the Ceph git repo and thus is developed and maintained in multiple branches (e.g. "master" and "nautilus").
This makes it challenging to keep these two components in sync and to facilitate both maintaining a "stable" version while allowing new features to be merged. To reduce this complexity and to better interlock the testing and development of the dashboard and ceph-iscsi, I would like to propose two possible solutions:
1) merge the ceph-iscsi code base into the ceph git repository. This way, the development of new features and maintenance would take place in distinct branches, and in close synchronization between the dashboard and the ceph-iscsi component. This might also be beneficial for creating unit tests that test both components without having to assemble the pieces from various places beforehand. It would also ensure that a matching ceph-iscsi package is always built and released along with the corresponding Ceph version, thus offloading the ceph-iscsi devs from the build and release work. From a version numbering perspective, this should not be a problem - the current ceph-iscsi packages are of version "3.0", so they could easily be upgraded to the Ceph versioning scheme. It would also help with keeping the documentation [4] in sync with the functionality (the ceph-iscsi docs are actually part of the main Ceph documentation in the git repo already). This approach is more work, as it requires finding a way to merge the ceph-iscsi git repo into the ceph git repo (ideally by preserving the history), and because the packaging and build scripts need to be updated to build the respective DEB and RPM packages.
As we have talked about before, I don't like the idea of merging everything and the kitchen sink into the "ceph" repo. We would need to ensure backwards compatibility regardless (be it ceph-iscsi upgrades one node at a time or the ceph-mgr/dashboard being upgraded first before ceph-iscsi). Those issues can be addressed by creating a decent set of unit test cases that simulate the REST interactions expected from the XYZ version of the dashboard to ceph-iscsi. This isn't the only sub-project that is tied to the core Ceph project repo (e.g. ceph-ansible, rook, ceph-csi, go-ceph, ...).
2) As an alternative that requires lesser effort (but also has fewer benefits), we should at least start using branches within the ceph-iscsi git repo, so we can do bug fixing on a stable version separate from new feature development that may introduce incompatible changes. I'd propose to name the branches according to the Ceph release they support, e.g. by creating a "nautilus" branch. But the packaging and releasing would still be disconnected from the Ceph releases (and the work would increase along with the branches we need to build and support).
As part of that same discussion, I also agreed that we should create a "3.x" stable branch along with a new 3.1 release once the last few known issues are merged. As of today, I believe it's only a single PR that just needs to get tested and merged.
I clearly would be in favor of the first proposal, but would need help in getting this implemented.
Thoughts, concerns, objections? Would anybody be willing and interested in making this happen?
Thanks!
Lenz
[1] https://github.com/ceph/ceph-iscsi/ [2] https://github.com/ceph/ceph-iscsi/pull/84/ [3] https://github.com/ceph/ceph/pull/28720 [4] http://docs.ceph.com/docs/master/rbd/iscsi-overview/ -- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason
On 6/25/19 7:02 AM, Jason Dillaman wrote:
On Tue, Jun 25, 2019 at 7:55 AM Lenz Grimmer <lgrimmer@suse.com> wrote:
Hi,
The ceph-iscsi project [1] has gained a lot of new features and functionality along with the iSCSI management features that were added to Ceph Dashboard in Nautilus. The Dashboard is tightly coupled to the ceph-iscsi config version and needs to be updated in parallel, e.g. when new functionality is added or existing behavior changes (e.g. here [2], [3]).
Currently, all ceph-iscsi development is done in the "master" branch of ceph-iscsi, while the dashboard is developed in the Ceph git repo and thus is developed and maintained in multiple branches (e.g. "master" and "nautilus").
This makes it challenging to keep these two components in sync and to facilitate both maintaining a "stable" version while allowing new features to be merged. To reduce this complexity and to better interlock the testing and development of the dashboard and ceph-iscsi, I would like to propose two possible solutions:
1) merge the ceph-iscsi code base into the ceph git repository. This way, the development of new features and maintenance would take place in distinct branches, and in close synchronization between the dashboard and the ceph-iscsi component. This might also be beneficial for creating unit tests that test both components without having to assemble the pieces from various places beforehand. It would also ensure that a matching ceph-iscsi package is always built and released along with the corresponding Ceph version, thus offloading the ceph-iscsi devs from the build and release work. From a version numbering perspective, this should not be a problem - the current ceph-iscsi packages are of version "3.0", so they could easily be upgraded to the Ceph versioning scheme. It would also help with keeping the documentation [4] in sync with the functionality (the ceph-iscsi docs are actually part of the main Ceph documentation in the git repo already). This approach is more work, as it requires finding a way to merge the ceph-iscsi git repo into the ceph git repo (ideally by preserving the history), and because the packaging and build scripts need to be updated to build the respective DEB and RPM packages. As we have talked about before, I don't like the idea of merging everything and the kitchen sink into the "ceph" repo. We would need to ensure backwards compatibility regardless (be it ceph-iscsi upgrades one node at a time or the ceph-mgr/dashboard being upgraded first before ceph-iscsi). Those issues can be addressed by creating a decent set of unit test cases that simulate the REST interactions expected from the XYZ version of the dashboard to ceph-iscsi. This isn't the only sub-project that is tied to the core Ceph project repo (e.g. ceph-ansible, rook, ceph-csi, go-ceph, ...).
Very much agreed! The ceph repo is already far bigger than I'd prefer. My current ceph working directory is 34GB! (granted most of that is build). Compile times have ballooned over the years too. Personally I'd rather go the other way and start removing things that aren't explicitly required.
2) As an alternative that requires lesser effort (but also has fewer benefits), we should at least start using branches within the ceph-iscsi git repo, so we can do bug fixing on a stable version separate from new feature development that may introduce incompatible changes. I'd propose to name the branches according to the Ceph release they support, e.g. by creating a "nautilus" branch. But the packaging and releasing would still be disconnected from the Ceph releases (and the work would increase along with the branches we need to build and support). As part of that same discussion, I also agreed that we should create a "3.x" stable branch along with a new 3.1 release once the last few known issues are merged. As of today, I believe it's only a single PR that just needs to get tested and merged.
I clearly would be in favor of the first proposal, but would need help in getting this implemented.
Thoughts, concerns, objections? Would anybody be willing and interested in making this happen?
Thanks!
Lenz
[1] https://github.com/ceph/ceph-iscsi/ [2] https://github.com/ceph/ceph-iscsi/pull/84/ [3] https://github.com/ceph/ceph/pull/28720 [4] http://docs.ceph.com/docs/master/rbd/iscsi-overview/ -- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Mark, On 6/25/19 2:31 PM, Mark Nelson wrote:
Very much agreed! The ceph repo is already far bigger than I'd prefer. My current ceph working directory is 34GB! (granted most of that is build). Compile times have ballooned over the years too. Personally I'd rather go the other way and start removing things that aren't explicitly required.
Of course that's something that should be considered/evaluated and I actually share your pains and concerns regarding the overall size. In the case of ceph-iscsi, I think the benefits of adding it would outweigh the additional overhead, but I'm aware it's a fine balance and we don't seem to have a formalized way in which sub-projects could be evaluated for promotion/inclusion into the main git repo (or the other way around). It probably makes sense to check the code base for unnecessary things on a regular basis. Do you have any specific components in mind? How do you define "explicitly required"? And by "removing", do you mean deleting them for good or moving then out of the ceph git repo into a separate one? Lenz -- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
On 6/25/19 9:08 AM, Lenz Grimmer wrote:
Hi Mark,
On 6/25/19 2:31 PM, Mark Nelson wrote:
Very much agreed! The ceph repo is already far bigger than I'd prefer. My current ceph working directory is 34GB! (granted most of that is build). Compile times have ballooned over the years too. Personally I'd rather go the other way and start removing things that aren't explicitly required. Of course that's something that should be considered/evaluated and I actually share your pains and concerns regarding the overall size. In the case of ceph-iscsi, I think the benefits of adding it would outweigh the additional overhead, but I'm aware it's a fine balance and we don't seem to have a formalized way in which sub-projects could be evaluated for promotion/inclusion into the main git repo (or the other way around).
Some kind of formal process for sub-projects probably makes sense!
It probably makes sense to check the code base for unnecessary things on a regular basis. Do you have any specific components in mind? How do you define "explicitly required"? And by "removing", do you mean deleting them for good or moving then out of the ceph git repo into a separate one?
I'd rather not single anything out, but I think having a formal process for determining inclusion is a very good idea. No need to delete anything, but I would very much like to see a smaller ceph git repo in the future by moving/splitting things that aren't especially tightly coupled. Other folks are more tolerant of large git repos though, so my opinion may be in the minority. Mark
Lenz
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Jason, On 6/25/19 2:02 PM, Jason Dillaman wrote:
As we have talked about before, I don't like the idea of merging everything and the kitchen sink into the "ceph" repo.
Sorry if this sounds like a rehash. I know we talked about this before, but I had a few more thoughts about this in the meanwhile, and I didn't recall we had come to a definitive agreement that was publicly shared/noted.
We would need to ensure backwards compatibility regardless (be it ceph-iscsi upgrades one node at a time or the ceph-mgr/dashboard being upgraded first before ceph-iscsi). Those issues can be addressed by creating a decent set of unit test cases that simulate the REST interactions expected from the XYZ version of the dashboard to ceph-iscsi.
Agreed, and adding such tests is something that should be done regardless of where the code resides.
This isn't the only sub-project that is tied to the core Ceph project repo (e.g. ceph-ansible, rook, ceph-csi, go-ceph, ...).
Ack. There are many projects in the Ceph ecosystem and I wouldn't suggest to merge "everything and the kitchen sink", just because it relates to Ceph in some form or another. My rationale for this particular proposal was that the ceph-iscsi code is fairly limited in scope and size and not dependent on any other external/special tools, while it does have a tight relationship to the dashboard. To me, it simply feels like it would benefit from a much closer integration than the other examples you mentioned above, as it provides an "access protocol handler" that extends the possibilities in which data can be stored in Ceph (similar to RGW).
As part of that same discussion, I also agreed that we should create a "3.x" stable branch along with a new 3.1 release once the last few known issues are merged. As of today, I believe it's only a single PR that just needs to get tested and merged.
OK, thanks - fair enough. That would already help to minimize confusion, as long as we also clarify this in the Ceph docs (e.g. which version to use and where to obtain the packages that match the corresponding Ceph version). Lenz -- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
On 6/25/19 1:54 PM, Lenz Grimmer wrote:
1) merge the ceph-iscsi code base into the ceph git repository.
FWIW, there have been a number of articles that promote the "mono repo" approach and list a number of good reasons for going down that route: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-l... https://blog.shippable.com/our-journey-to-microservices-and-a-mono-repositor... https://github.com/babel/babel/blob/master/doc/design/monorepo.md Worth a read ;) Lenz -- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
Interesting debate you brought up here, Lenz! Wouldn't want to divert from the original question, but... The interesting thing is that, as the paper says, BECAUSE of that monolithic approach Google couldn't keep on using P4 (as it no longer scaled enough) and had to build their own VCS. Besides, P4 (not sure if Piper follows the same approach) made really easy to setup up workspaces with cherry-picked dirs to sync (instead of whole repo clones). So IMHO the debate should not be only about mono-repo vs. multi-repos, but centralized mono-repo vs DVCS multi-repos. In my experience this is also about the practices & cultures allowed/fostered by mono-repo vs. multi-repo ecosystems: - code, dirs & files vs. architecture, modules & interfaces - atomic (backward-breaking) changes vs. extending functionality while honoring APIs - code sharing/reuse/ownership vs. functionality sharing/reuse/ownership * - grep vs. documentation - #include dependency management vs. declarative dep. mgmt - build everything vs. delta builds - E2E testing vs. integration testing - 1 tool for everything (VCS) vs. pipelines (VCS, CI, artifact/release management, CD, etc.) * mono-repos make really hard to share/spread code beyond the repo, BTW, here's (https://www.youtube.com/watch?v=W71BTkUbdqE) a recorded talk about that paper, and it contains a few 'gems' ("Google solves dependencies by statically linking everything", "can do massive backward-incompatible changes... atomically"). Ernesto On Tue, Jun 25, 2019 at 4:24 PM Lenz Grimmer <lgrimmer@suse.com> wrote:
On 6/25/19 1:54 PM, Lenz Grimmer wrote:
1) merge the ceph-iscsi code base into the ceph git repository.
FWIW, there have been a number of articles that promote the "mono repo" approach and list a number of good reasons for going down that route:
https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-l...
https://blog.shippable.com/our-journey-to-microservices-and-a-mono-repositor...
https://github.com/babel/babel/blob/master/doc/design/monorepo.md
Worth a read ;)
Lenz
-- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Ernesto, thanks for your feedback! On 6/25/19 7:49 PM, Ernesto Puerta wrote:
Interesting debate you brought up here, Lenz! Wouldn't want to divert from the original question, but...
The interesting thing is that, as the paper says, BECAUSE of that monolithic approach Google couldn't keep on using P4 (as it no longer scaled enough) and had to build their own VCS.
Because they can :) I personally think that git still scales quite well for a project of the size of Ceph (in terms of contributors and code base).
Besides, P4 (not sure if Piper follows the same approach) made really easy to setup up workspaces with cherry-picked dirs to sync (instead of whole repo clones). So IMHO the debate should not be only about mono-repo vs. multi-repos, but centralized mono-repo vs DVCS multi-repos.
But that would likely require more drastic changes, wouldn't it?
In my experience this is also about the practices & cultures allowed/fostered by mono-repo vs. multi-repo ecosystems: - code, dirs & files vs. architecture, modules & interfaces - atomic (backward-breaking) changes vs. extending functionality while honoring APIs - code sharing/reuse/ownership vs. functionality sharing/reuse/ownership * - grep vs. documentation - #include dependency management vs. declarative dep. mgmt - build everything vs. delta builds - E2E testing vs. integration testing - 1 tool for everything (VCS) vs. pipelines (VCS, CI, artifact/release management, CD, etc.)
* mono-repos make really hard to share/spread code beyond the repo,
BTW, here's (https://www.youtube.com/watch?v=W71BTkUbdqE) a recorded talk about that paper, and it contains a few 'gems' ("Google solves dependencies by statically linking everything", "can do massive backward-incompatible changes... atomically").
Thanks for sharing your insights! I guess we would have to start with collecting current pain points and concerns and then figure out if any of these approaches would help to alleviate them. In my case, I outlined the reasons for my proposal in my initial message. Having worked on openATTIC as an independent project before, we really enjoy working on the dashboard as a "built-in" component. Less surprises... Lenz -- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
On Thu, Jun 27, 2019 at 5:00 AM Lenz Grimmer <lgrimmer@suse.com> wrote:
Hi Ernesto,
thanks for your feedback!
On 6/25/19 7:49 PM, Ernesto Puerta wrote:
Interesting debate you brought up here, Lenz! Wouldn't want to divert from the original question, but...
The interesting thing is that, as the paper says, BECAUSE of that monolithic approach Google couldn't keep on using P4 (as it no longer scaled enough) and had to build their own VCS.
Because they can :)
I personally think that git still scales quite well for a project of the size of Ceph (in terms of contributors and code base).
Besides, P4 (not sure if Piper follows the same approach) made really easy to setup up workspaces with cherry-picked dirs to sync (instead of whole repo clones). So IMHO the debate should not be only about mono-repo vs. multi-repos, but centralized mono-repo vs DVCS multi-repos.
But that would likely require more drastic changes, wouldn't it?
In my experience this is also about the practices & cultures allowed/fostered by mono-repo vs. multi-repo ecosystems: - code, dirs & files vs. architecture, modules & interfaces - atomic (backward-breaking) changes vs. extending functionality while honoring APIs - code sharing/reuse/ownership vs. functionality sharing/reuse/ownership * - grep vs. documentation - #include dependency management vs. declarative dep. mgmt - build everything vs. delta builds - E2E testing vs. integration testing - 1 tool for everything (VCS) vs. pipelines (VCS, CI, artifact/release management, CD, etc.)
* mono-repos make really hard to share/spread code beyond the repo,
BTW, here's (https://www.youtube.com/watch?v=W71BTkUbdqE) a recorded talk about that paper, and it contains a few 'gems' ("Google solves dependencies by statically linking everything", "can do massive backward-incompatible changes... atomically").
Thanks for sharing your insights! I guess we would have to start with collecting current pain points and concerns and then figure out if any of these approaches would help to alleviate them. In my case, I outlined the reasons for my proposal in my initial message. Having worked on openATTIC as an independent project before, we really enjoy working on the dashboard as a "built-in" component. Less surprises...
This is a relief to hear (to collect pain points), because the ceph-volume experience has been tremendously hard: it made testing slower, it needed to release faster because fixes/improvements were produced at a rate that the release cycle couldn't accommodate (this has since slowed down though), and lastly the backporting of everything in master all the way down to Luminous took out a good chunk of our time that we never anticipated. There has been talk of pulling ceph-volume out of the tree but I frankly doubt that the enormous amount of work it would take would benefit us. We wouldn't get the "we no longer need to backport" until Nautilus (!) is no longer supported. Having a conversation about this is great though, happy to add more details of what worked well and what didn't in the case of ceph-volume.
Lenz
-- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (5)
-
Alfredo Deza
-
Ernesto Puerta
-
Jason Dillaman
-
Lenz Grimmer
-
Mark Nelson