Distributions tested and build for
Hi, Kyrlyo wanted to add openSUSE to the set of distributions that get build for and used by teuthology, for example: https://github.com/ceph/ceph-build/pull/1356 https://github.com/ceph/ceph-build/pull/1355 However, seems officially there are "no plans" to add other platforms but Ubuntu and CentOS, so Alfredo closed the PRs. What requirements does a platform need to meet to be added? What's the process for that? (I hope this is the right list) Regards, Lars -- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
Hi,
Kyrlyo wanted to add openSUSE to the set of distributions that get build for and used by teuthology, for example:
https://github.com/ceph/ceph-build/pull/1356 https://github.com/ceph/ceph-build/pull/1355
However, seems officially there are "no plans" to add other platforms but Ubuntu and CentOS, so Alfredo closed the PRs.
Hey Lars. Adding a distro for builds is a very involved problem to solve. We don't keep a detailed list of everything that is needed but I will try to go over some of the well-known items: 1) for RPM-based distros, we must have an actual machine running that distro (for example CentOS 7 for CentOS7 RPMs) - until we start using mock where we can produce RPMs from any base distro. 2) A new distribution added *must* exist in the cloud provider (OVH in this case) that can spin up a VM for builds (as of this writing, there is only an opensuse42 image available from 2016) 3) *All* the building scripts must be revised to ensure that the new distribution is accounted for. I did some of this work when adding Ubuntu Bionic and it was non-trivial, error prone, and it took about two weeks to really get it right with the help of other people. 4) The services that ensure that images come up and are prepared to build Ceph have to be udpated as well to ensure that the minimum requirements are installed so that the machine is operational 5) If the new distro is Python3 only we will need to update all tooling that interacts with a jenkins node - we are not there yet as all our tooling is Python2 exclusive. At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM A few of the things we brought up is that we (the Ceph infrastructure team and our services) aren't prepared to accommodate multiple other distributions and that we are trying to get away from taking on the load of maintenance in our systems and looking to other build/repo solutions. One of these solutions is the CentOS storage-sig which we are trying to coordinate to build and host repos for us there. In addition to that, we mentioned that we would like to see a wider community effort go into building and hosting Ceph in separate systems, maybe with a special signing key (our release signing process is pretty inflexible!) so that others who are building development repositories can ensure their authenticity, while giving us the ability to revoke keys as needed. In the past, we've gotten asked to re-enable the Debian builds, which puts us into a similar problem (maintenance burden, script updates, and other items already mentioned), and we've had to turn that down. As Ken mentions in the presentation, we really want to be helpful and accommodating to the wider community, but we can't do it on our own and with our infrastructure as it is today - we are maxed out. Distributing community signing keys, or allowing other builders to submit status updates into shaman.ceph.com for testing scheduling is yet-to-be-done work, but I am open to have those conversations so that we can move forwards with more distros and better testing.
What requirements does a platform need to meet to be added? What's the process for that?
(I hope this is the right list)
Regards, Lars
-- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Aug 2, 2019, at 7:55 PM, Alfredo Deza <adeza@redhat.com> wrote:
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
Hi,
Kyrlyo wanted to add openSUSE to the set of distributions that get build for and used by teuthology, for example:
https://github.com/ceph/ceph-build/pull/1356 https://github.com/ceph/ceph-build/pull/1355
However, seems officially there are "no plans" to add other platforms but Ubuntu and CentOS, so Alfredo closed the PRs.
Hey Lars. Adding a distro for builds is a very involved problem to solve. We don't keep a detailed list of everything that is needed but I will try to go over some of the well-known items:
1) for RPM-based distros, we must have an actual machine running that distro (for example CentOS 7 for CentOS7 RPMs) - until we start using mock where we can produce RPMs from any base distro. We are not interesting in the «mock» on CentOS or what ever RedHat, we are interested in pure SUSE environment. 2) A new distribution added *must* exist in the cloud provider (OVH in this case) that can spin up a VM for builds (as of this writing, there is only an opensuse42 image available from 2016) This image is outdated and can be dropped. We, at SUSE want to add a new image, like opensuse15 or leap15. I can help with this. Where can I find the instruction on how to do this, or someone who can say me what to do? Who can help us to get access to OVH. I've already spent some time to add support of recent openSUSE distro to ansible scripts. One of the patch to ansible/slave for ceph-build is rejected with not clear reason and without reasonable discussion.
3) *All* the building scripts must be revised to ensure that the new distribution is accounted for. I did some of this work when adding Ubuntu Bionic and it was non-trivial, error prone, and it took about two weeks to really get it right with the help of other people. How can we contribute when adding the distro rather than providing patches and human resources? 4) The services that ensure that images come up and are prepared to build Ceph have to be udpated as well to ensure that the minimum requirements are installed so that the machine is operational Where are those services described? How can we contribute into this?
5) If the new distro is Python3 only we will need to update all tooling that interacts with a jenkins node - we are not there yet as all our tooling is Python2 exclusive.
At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
I haven’t seen in this video any exact recommendation how to proceed if one wish to contribute and add support own system. I had not see any reference where one can start.
A few of the things we brought up is that we (the Ceph infrastructure team and our services) aren't prepared to accommodate multiple other
distributions and that we are trying to get away from taking on the load of maintenance in our systems and looking to other build/repo solutions. One of these solutions is the CentOS storage-sig which we are trying to coordinate to build and host repos for us there.
In addition to that, we mentioned that we would like to see a wider community effort go into building and hosting Ceph in separate systems, maybe with a special signing key (our release signing process is pretty inflexible!) so that others who are building development repositories can ensure their authenticity, while giving us the ability to revoke keys as needed.
Do you guys have clear procedure how to do this? How can we help to develop the one?
In the past, we've gotten asked to re-enable the Debian builds, which puts us into a similar problem (maintenance burden, script updates, and other items already mentioned), and we've had to turn that down. As Ken mentions in the presentation, we really want to be helpful and accommodating to the wider community, but we can't do it on our own and with our infrastructure as it is today - we are maxed out.
Distributing community signing keys, or allowing other builders to submit status updates into shaman.ceph.com for testing scheduling is yet-to-be-done work, but I am open to have those conversations so that we can move forwards with more distros and better testing.
Taking into account that SUSE can provide resources in order to help improve Ceph Infrastructure, can we start from the topic what are clear steps, so at the end we can: 1) Submit a PR and there make check test can run on suse based distro and reported back to the github. 2) There can be scheduled build for our lovely distro based on PR and any branch we demand, artifacts exposed at shaman, and can be used in teuthology. 3) Same as (2) but for the nightly/daily builds. Finally, do we need to improve existing ceph/ceph-build system or we need to build our own system on some other site? If so, can we use Shaman/Chacra or we have to develop our own infrastructure? If Shaman/Chacra is the way, who can provide us corresponding credentials/keys in order to use it, or we can try and grab them from the logs of jenkins? Is there any official documentation how to integrate with Shaman/Chacra. 4) So let say we have already some CI (onsite Jenkins setup) and resources which are running, for example, «make check» for PR against our openSUSE system. We wish to enable it for github.com/ceph/ceph <http://github.com/ceph/ceph>, we need right access to the repo in order to report the status to the PR, who should we ask for granting the access for our robot user? Or how does it work?
What requirements does a platform need to meet to be added? What's the process for that?
(I hope this is the right list)
Regards, Lars
-- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
I've been doing weekly builds of master for leap and tumbleweed on copr for some time now. https://copr.fedorainfracloud.org/coprs/badone/ceph-weeklies/ I realise you are after bare metal builds but I thought it was worth mentioning as a possible alternative while you work on the final solution. On Sat, Aug 3, 2019 at 7:54 AM kyrylo.shatskyy <kyrylo.shatskyy@suse.com> wrote:
On Aug 2, 2019, at 7:55 PM, Alfredo Deza <adeza@redhat.com> wrote:
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
Hi,
Kyrlyo wanted to add openSUSE to the set of distributions that get build for and used by teuthology, for example:
https://github.com/ceph/ceph-build/pull/1356 https://github.com/ceph/ceph-build/pull/1355
However, seems officially there are "no plans" to add other platforms but Ubuntu and CentOS, so Alfredo closed the PRs.
Hey Lars. Adding a distro for builds is a very involved problem to solve. We don't keep a detailed list of everything that is needed but I will try to go over some of the well-known items:
1) for RPM-based distros, we must have an actual machine running that distro (for example CentOS 7 for CentOS7 RPMs) - until we start using mock where we can produce RPMs from any base distro.
We are not interesting in the «mock» on CentOS or what ever RedHat, we are interested in pure SUSE environment.
2) A new distribution added *must* exist in the cloud provider (OVH in this case) that can spin up a VM for builds (as of this writing, there is only an opensuse42 image available from 2016)
This image is outdated and can be dropped. We, at SUSE want to add a new image, like opensuse15 or leap15. I can help with this. Where can I find the instruction on how to do this, or someone who can say me what to do? Who can help us to get access to OVH. I've already spent some time to add support of recent openSUSE distro to ansible scripts. One of the patch to ansible/slave for ceph-build is rejected with not clear reason and without reasonable discussion.
3) *All* the building scripts must be revised to ensure that the new distribution is accounted for. I did some of this work when adding Ubuntu Bionic and it was non-trivial, error prone, and it took about two weeks to really get it right with the help of other people.
How can we contribute when adding the distro rather than providing patches and human resources?
4) The services that ensure that images come up and are prepared to build Ceph have to be udpated as well to ensure that the minimum requirements are installed so that the machine is operational
Where are those services described? How can we contribute into this?
5) If the new distro is Python3 only we will need to update all tooling that interacts with a jenkins node - we are not there yet as all our tooling is Python2 exclusive.
At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
I haven’t seen in this video any exact recommendation how to proceed if one wish to contribute and add support own system. I had not see any reference where one can start.
A few of the things we brought up is that we (the Ceph infrastructure team and our services) aren't prepared to accommodate multiple other
distributions and that we are trying to get away from taking on the load of maintenance in our systems and looking to other build/repo solutions. One of these solutions is the CentOS storage-sig which we are trying to coordinate to build and host repos for us there.
In addition to that, we mentioned that we would like to see a wider community effort go into building and hosting Ceph in separate systems, maybe with a special signing key (our release signing process is pretty inflexible!) so that others who are building development repositories can ensure their authenticity, while giving us the ability to revoke keys as needed.
Do you guys have clear procedure how to do this? How can we help to develop the one?
In the past, we've gotten asked to re-enable the Debian builds, which puts us into a similar problem (maintenance burden, script updates, and other items already mentioned), and we've had to turn that down. As Ken mentions in the presentation, we really want to be helpful and accommodating to the wider community, but we can't do it on our own and with our infrastructure as it is today - we are maxed out.
Distributing community signing keys, or allowing other builders to submit status updates into shaman.ceph.com for testing scheduling is yet-to-be-done work, but I am open to have those conversations so that we can move forwards with more distros and better testing.
Taking into account that SUSE can provide resources in order to help improve Ceph Infrastructure, can we start from the topic what are clear steps, so at the end we can: 1) Submit a PR and there make check test can run on suse based distro and reported back to the github. 2) There can be scheduled build for our lovely distro based on PR and any branch we demand, artifacts exposed at shaman, and can be used in teuthology. 3) Same as (2) but for the nightly/daily builds.
Finally, do we need to improve existing ceph/ceph-build system or we need to build our own system on some other site? If so, can we use Shaman/Chacra or we have to develop our own infrastructure? If Shaman/Chacra is the way, who can provide us corresponding credentials/keys in order to use it, or we can try and grab them from the logs of jenkins? Is there any official documentation how to integrate with Shaman/Chacra.
4) So let say we have already some CI (onsite Jenkins setup) and resources which are running, for example, «make check» for PR against our openSUSE system. We wish to enable it for github.com/ceph/ceph, we need right access to the repo in order to report the status to the PR, who should we ask for granting the access for our robot user? Or how does it work?
What requirements does a platform need to meet to be added? What's the process for that?
(I hope this is the right list)
Regards, Lars
-- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Cheers, Brad
On 8/3/19 7:54 AM, kyrylo.shatskyy wrote:
At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
I haven’t seen in this video any exact recommendation how to proceed if one wish to contribute and add support own system. I had not see any reference where one can start.
IIRC one suggestion from that video was for third parties to listen to the fedora message bus, which should allow someone running their own jenkins slave to trigger builds on their own hardware based on commits/PRs to the ceph git repo. But that still leaves open the question of how to feed build results back to... Where? The PR in question? Shaman? Somewhere else? (Sorry, a bunch of this stuff is still black magic from my perspective) Regards, Tim -- Tim Serong Senior Clustering Engineer SUSE tserong@suse.com
W dniu 02.08.2019 o 19:55, Alfredo Deza pisze:
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
Kyrlyo wanted to add openSUSE to the set of distributions that get build for and used by teuthology, for example:
However, seems officially there are "no plans" to add other platforms but Ubuntu and CentOS, so Alfredo closed the PRs.
At Linaro we would like to help with adding Debian to the list of distributions Ceph is built for. We can provide aarch64 (arm64) machines for it.
Hey Lars. Adding a distro for builds is a very involved problem to solve. We don't keep a detailed list of everything that is needed but I will try to go over some of the well-known items:
2) A new distribution added *must* exist in the cloud provider (OVH in this case) that can spin up a VM for builds (as of this writing, there is only an opensuse42 image available from 2016)
https://www.ovh.co.uk/dedicated_servers/distributions/ lists Debian 10 as available. Not that this page is up-to-date as it does not even list Ubuntu 18.04
3) *All* the building scripts must be revised to ensure that the new distribution is accounted for. I did some of this work when adding Ubuntu Bionic and it was non-trivial, error prone, and it took about two weeks to really get it right with the help of other people.
I can probably work on it.
4) The services that ensure that images come up and are prepared to build Ceph have to be udpated as well to ensure that the minimum requirements are installed so that the machine is operational
5) If the new distro is Python3 only we will need to update all tooling that interacts with a jenkins node - we are not there yet as all our tooling is Python2 exclusive.
Debian 10 'buster' has both Python 2.7 and Python 3.7 so should not be a problem - we can start with py2 and then update to py3 once Ceph move.
At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
Will watch it later.
A few of the things we brought up is that we (the Ceph infrastructure team and our services) aren't prepared to accommodate multiple other distributions and that we are trying to get away from taking on the load of maintenance in our systems and looking to other build/repo solutions. One of these solutions is the CentOS storage-sig which we are trying to coordinate to build and host repos for us there.
In addition to that, we mentioned that we would like to see a wider community effort go into building and hosting Ceph in separate systems, maybe with a special signing key (our release signing process is pretty inflexible!) so that others who are building development repositories can ensure their authenticity, while giving us the ability to revoke keys as needed.
In the past, we've gotten asked to re-enable the Debian builds, which puts us into a similar problem (maintenance burden, script updates, and other items already mentioned), and we've had to turn that down. As Ken mentions in the presentation, we really want to be helpful and accommodating to the wider community, but we can't do it on our own and with our infrastructure as it is today - we are maxed out.
Distributing community signing keys, or allowing other builders to submit status updates into shaman.ceph.com for testing scheduling is yet-to-be-done work, but I am open to have those conversations so that we can move forwards with more distros and better testing.
At Linaro we can arrange machines for building and testing. Space for hosting Debian/arm64 repos too. We are fine on using those machines also to build packages for other arm64 distributions.
I think we are deviating a bit on the discussion: 1) We are not in the capacity of adding any other new distro to our build toolchain, for development or releases. 2) We can own the effort of adding the mechanisms needed in shaman (shaman.ceph.com) so that community-built Ceph packages/repos can report there. This will entail adding authentication so that updates can verify the source. For #2 specifically, the shaman dashboard allows updating the status of a build (started, building, failed, succeeded) as well as the repo. Tools like teuthology query shaman for the state and location of repos. The API is detailed here: https://github.com/ceph/shaman On Mon, Aug 5, 2019 at 4:21 AM Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> wrote:
W dniu 02.08.2019 o 19:55, Alfredo Deza pisze:
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
Kyrlyo wanted to add openSUSE to the set of distributions that get build for and used by teuthology, for example:
However, seems officially there are "no plans" to add other platforms but Ubuntu and CentOS, so Alfredo closed the PRs.
At Linaro we would like to help with adding Debian to the list of distributions Ceph is built for. We can provide aarch64 (arm64) machines for it.
Hey Lars. Adding a distro for builds is a very involved problem to solve. We don't keep a detailed list of everything that is needed but I will try to go over some of the well-known items:
2) A new distribution added *must* exist in the cloud provider (OVH in this case) that can spin up a VM for builds (as of this writing, there is only an opensuse42 image available from 2016)
https://www.ovh.co.uk/dedicated_servers/distributions/ lists Debian 10 as available. Not that this page is up-to-date as it does not even list Ubuntu 18.04
3) *All* the building scripts must be revised to ensure that the new distribution is accounted for. I did some of this work when adding Ubuntu Bionic and it was non-trivial, error prone, and it took about two weeks to really get it right with the help of other people.
I can probably work on it.
4) The services that ensure that images come up and are prepared to build Ceph have to be udpated as well to ensure that the minimum requirements are installed so that the machine is operational
5) If the new distro is Python3 only we will need to update all tooling that interacts with a jenkins node - we are not there yet as all our tooling is Python2 exclusive.
Debian 10 'buster' has both Python 2.7 and Python 3.7 so should not be a problem - we can start with py2 and then update to py3 once Ceph move.
At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
Will watch it later.
A few of the things we brought up is that we (the Ceph infrastructure team and our services) aren't prepared to accommodate multiple other distributions and that we are trying to get away from taking on the load of maintenance in our systems and looking to other build/repo solutions. One of these solutions is the CentOS storage-sig which we are trying to coordinate to build and host repos for us there.
In addition to that, we mentioned that we would like to see a wider community effort go into building and hosting Ceph in separate systems, maybe with a special signing key (our release signing process is pretty inflexible!) so that others who are building development repositories can ensure their authenticity, while giving us the ability to revoke keys as needed.
In the past, we've gotten asked to re-enable the Debian builds, which puts us into a similar problem (maintenance burden, script updates, and other items already mentioned), and we've had to turn that down. As Ken mentions in the presentation, we really want to be helpful and accommodating to the wider community, but we can't do it on our own and with our infrastructure as it is today - we are maxed out.
Distributing community signing keys, or allowing other builders to submit status updates into shaman.ceph.com for testing scheduling is yet-to-be-done work, but I am open to have those conversations so that we can move forwards with more distros and better testing.
At Linaro we can arrange machines for building and testing. Space for hosting Debian/arm64 repos too. We are fine on using those machines also to build packages for other arm64 distributions. _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Alfredo, On Mon, 2019-08-05 at 09:02 -0400, Alfredo Deza wrote:
I think we are deviating a bit on the discussion: 1) We are not in the capacity of adding any other new distro to ourbuild toolchain, for development or releases.
What would be the steps in order to extend the capacity for adding a new distro?What is required from the community? Money, like extend OVH computing resources, dedicated engineer who can support the distroand play role as maintainer?
2) We can own the effort of adding the mechanisms needed in shaman(shaman.ceph.com) so that community-built Ceph packages/repos canreport there. This will entailadding authentication so that updates can verify the source.
Do I understand correctly that there is no mechanism at the moment.
For #2 specifically, the shaman dashboard allows updating the statusof a build (started, building, failed, succeeded) as well as the repo.Tools like teuthology query shaman for the state and locationof repos. The API is detailed here: https://github.com/ceph/shaman I only see that API can support one login/password pair, am I wrong?Who can provide existing credential to us, so we can integrate our buildswithout taking Ceph Infrastructure team engineering resources? On Mon, Aug 5, 2019 at 4:21 AM Marcin Juszkiewicz< marcin.juszkiewicz@linaro.org> wrote:
W dniu 02.08.2019 o 19:55, Alfredo Deza pisze:
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
Kyrlyo wanted to add openSUSE to the set of distributions that getbuild for and used by teuthology, for example: However, seems officially there are "no plans" to add otherplatforms but Ubuntu and CentOS, so Alfredo closed the PRs.
At Linaro we would like to help with adding Debian to the list ofdistributions Ceph is built for. We can provide aarch64 (arm64) machinesfor it.
Hey Lars. Adding a distro for builds is a very involved problem tosolve. We don't keep a detailed list of everything that is neededbut I will try to go over some of the well-known items: 2) A new distribution added *must* exist in the cloud provider (OVHin this case) that can spin up a VM for builds (as of this writing,there is only an opensuse42 image available from 2016)
https://www.ovh.co.uk/dedicated_servers/distributions/ lists Debian 10as available. Not that this page is up-to-date as it does not even listUbuntu 18.04
3) *All* the building scripts must be revised to ensure that the newdistribution is accounted for. I did some of this work when addingUbuntu Bionic and it was non-trivial, error prone, and it took abouttwo weeks to really get it right with the help of other people.
I can probably work on it.
4) The services that ensure that images come up and are prepared tobuild Ceph have to be udpated as well to ensure that the minimumrequirements are installed so that the machine is operational 5) If the new distro is Python3 only we will need to update alltooling that interacts with a jenkins node - we are not there yet asall our tooling is Python2 exclusive.
Debian 10 'buster' has both Python 2.7 and Python 3.7 so should not bea problem - we can start with py2 and then update to py3 once Ceph move.
At Cephalocon, Ken Dreyer and me did a presentation on what exactlyentails building Ceph for both development and releases, theproblems we've faced and where we would like to head next. It mightbe useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
Will watch it later.
A few of the things we brought up is that we (the Cephinfrastructure team and our services) aren't prepared to accommodatemultiple other distributions and that we are trying to get away fromtaking on the load of maintenance in our systems and looking to otherbuild/repo solutions. One of these solutions is the CentOSstorage-sig which we are trying to coordinate to build and host reposfor us there. In addition to that, we mentioned that we would like to see a widercommunity effort go into building and hosting Ceph in separatesystems, maybe with a special signing key (our release signingprocess is pretty inflexible!) so that others who are buildingdevelopment repositories can ensure their authenticity, while givingus the ability to revoke keys as needed. In the past, we've gotten asked to re-enable the Debian builds,which puts us into a similar problem (maintenance burden, scriptupdates, and other items already mentioned), and we've had to turnthat down. As Ken mentions in the presentation, we really want to behelpful and accommodating to the wider community, but we can't do iton our own and with our infrastructure as it is today - we are maxedout. Distributing community signing keys, or allowing other builders tosubmit status updates into shaman.ceph.com for testing scheduling isyet-to-be-done work, but I am open to have those conversations sothat we can move forwards with more distros and better testing.
At Linaro we can arrange machines for building and testing. Space forhosting Debian/arm64 repos too. We are fine on using those machines alsoto build packages for other arm64 distributions._______________________________________________Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io _______________________________________________Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Mon, 5 Aug 2019, Alfredo Deza wrote:
I think we are deviating a bit on the discussion:
1) We are not in the capacity of adding any other new distro to our build toolchain, for development or releases.
2) We can own the effort of adding the mechanisms needed in shaman (shaman.ceph.com) so that community-built Ceph packages/repos can report there. This will entail adding authentication so that updates can verify the source.
For #2 specifically, the shaman dashboard allows updating the status of a build (started, building, failed, succeeded) as well as the repo. Tools like teuthology query shaman for the state and location of repos. The API is detailed here: https://github.com/ceph/shaman
I think this is misunderstanding who "we" are. What distributions are tested and built for shaman/chacra and download.ceph.com is a community decision and depends on who is able to invest the effort. If someone shows up willing to do the work, whoever was doing the work before doesn't get to just say no--especially if they don't want to be stuck with that responsibility for all time. I see two paths forward: (1) we continue with a monolithic approach to builds and expand the pool of people who understand and contribute to maintaining the build infra, or (2) we rearchitect to a federated approach. *Both* paths require knowledge transfer to new people, especially if the old team is too busy with other projects (as I keep hearing). (FWIW, the first path sounds like a lot less effort, and the two presumably also aren't mutually exclusive.) I talked to David a couple weeks back about getting a walk through bringing Debian Buster up in fog so that we could document the process and the response I got was that it is all already documented. It then took him the rest of the day to get a working fog image. :) Where is this documentation? Thanks! sage
On Mon, Aug 5, 2019 at 4:21 AM Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> wrote:
W dniu 02.08.2019 o 19:55, Alfredo Deza pisze:
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
Kyrlyo wanted to add openSUSE to the set of distributions that get build for and used by teuthology, for example:
However, seems officially there are "no plans" to add other platforms but Ubuntu and CentOS, so Alfredo closed the PRs.
At Linaro we would like to help with adding Debian to the list of distributions Ceph is built for. We can provide aarch64 (arm64) machines for it.
Hey Lars. Adding a distro for builds is a very involved problem to solve. We don't keep a detailed list of everything that is needed but I will try to go over some of the well-known items:
2) A new distribution added *must* exist in the cloud provider (OVH in this case) that can spin up a VM for builds (as of this writing, there is only an opensuse42 image available from 2016)
https://www.ovh.co.uk/dedicated_servers/distributions/ lists Debian 10 as available. Not that this page is up-to-date as it does not even list Ubuntu 18.04
3) *All* the building scripts must be revised to ensure that the new distribution is accounted for. I did some of this work when adding Ubuntu Bionic and it was non-trivial, error prone, and it took about two weeks to really get it right with the help of other people.
I can probably work on it.
4) The services that ensure that images come up and are prepared to build Ceph have to be udpated as well to ensure that the minimum requirements are installed so that the machine is operational
5) If the new distro is Python3 only we will need to update all tooling that interacts with a jenkins node - we are not there yet as all our tooling is Python2 exclusive.
Debian 10 'buster' has both Python 2.7 and Python 3.7 so should not be a problem - we can start with py2 and then update to py3 once Ceph move.
At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
Will watch it later.
A few of the things we brought up is that we (the Ceph infrastructure team and our services) aren't prepared to accommodate multiple other distributions and that we are trying to get away from taking on the load of maintenance in our systems and looking to other build/repo solutions. One of these solutions is the CentOS storage-sig which we are trying to coordinate to build and host repos for us there.
In addition to that, we mentioned that we would like to see a wider community effort go into building and hosting Ceph in separate systems, maybe with a special signing key (our release signing process is pretty inflexible!) so that others who are building development repositories can ensure their authenticity, while giving us the ability to revoke keys as needed.
In the past, we've gotten asked to re-enable the Debian builds, which puts us into a similar problem (maintenance burden, script updates, and other items already mentioned), and we've had to turn that down. As Ken mentions in the presentation, we really want to be helpful and accommodating to the wider community, but we can't do it on our own and with our infrastructure as it is today - we are maxed out.
Distributing community signing keys, or allowing other builders to submit status updates into shaman.ceph.com for testing scheduling is yet-to-be-done work, but I am open to have those conversations so that we can move forwards with more distros and better testing.
At Linaro we can arrange machines for building and testing. Space for hosting Debian/arm64 repos too. We are fine on using those machines also to build packages for other arm64 distributions. _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 2019-08-05T13:48:26, Sage Weil <sage@newdream.net> wrote: Hi Sage, thanks for jumping in.
I think this is misunderstanding who "we" are. What distributions are tested and built for shaman/chacra and download.ceph.com is a community decision and depends on who is able to invest the effort. If someone shows up willing to do the work, whoever was doing the work before doesn't get to just say no--especially if they don't want to be stuck with that responsibility for all time.
I think the summary here is that SUSE is willing to shoulder part of this work going forward, and if the respective infrastructure is resource constrained, well, the Foundation exists for specifically that purpose.
I see two paths forward: (1) we continue with a monolithic approach to builds and expand the pool of people who understand and contribute to maintaining the build infra, or (2) we rearchitect to a federated approach.
I think it'd be a mix. Federating the builds - and pushing maintaining distro-packages to, well, the distributors (Fedora, CentOS, Ubuntu, Debian, openSUSE) - seems most sensible. The distros really know best how to build for their platforms. e.g., for (open)SUSE - I can't talk about any other distro -, we'd be willing to either host the builds on our Open Build Service, or help setup a dedicated OBS instance in the lab somewhere (that'd possibly help with making sure we get them in a timely fashion even when the public infra is overloaded) and helping maintain base images for the OS to run on. And then once builds are done, pulling them back to the shared infrastructure for test runs. This is all sort-of a pre-requisite to eventually also run upstream/cross-distro inter-op testings in a more structured fashion, anyway.
*Both* paths require knowledge transfer to new people, especially if the old team is too busy with other projects (as I keep hearing). (FWIW, the first path sounds like a lot less effort, and the two presumably also aren't mutually exclusive.)
Yeah, agreed. So, TL;DR: we're willing to help with extending the distro coverage (for the tests in particular), as long as we know whom to work with to make that happen. We already have openSUSE on https://docs.ceph.com/docs/master/install/get-packages/ for example. (Probably we want to backport that to the Nautilus and previous releases for openSUSE, though.) But https://github.com/ceph/ceph-qa-suite/tree/master/distros/supported is somewhat more limited. Also, last updated 3 years, really? ;-) (That's at least what the docs point to.) Regards, Lars -- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
On Mon, Aug 5, 2019 at 10:49 AM Lars Marowsky-Bree <lmb@suse.com> wrote:
On 2019-08-05T13:48:26, Sage Weil <sage@newdream.net> wrote:
Hi Sage,
thanks for jumping in.
I think this is misunderstanding who "we" are. What distributions are tested and built for shaman/chacra and download.ceph.com is a community decision and depends on who is able to invest the effort. If someone shows up willing to do the work, whoever was doing the work before doesn't get to just say no--especially if they don't want to be stuck with that responsibility for all time.
I think the summary here is that SUSE is willing to shoulder part of this work going forward, and if the respective infrastructure is resource constrained, well, the Foundation exists for specifically that purpose.
I see two paths forward: (1) we continue with a monolithic approach to builds and expand the pool of people who understand and contribute to maintaining the build infra, or (2) we rearchitect to a federated approach.
I think it'd be a mix.
Federating the builds - and pushing maintaining distro-packages to, well, the distributors (Fedora, CentOS, Ubuntu, Debian, openSUSE) - seems most sensible. The distros really know best how to build for their platforms.
e.g., for (open)SUSE - I can't talk about any other distro -, we'd be willing to either host the builds on our Open Build Service, or help setup a dedicated OBS instance in the lab somewhere (that'd possibly help with making sure we get them in a timely fashion even when the public infra is overloaded) and helping maintain base images for the OS to run on.
And then once builds are done, pulling them back to the shared infrastructure for test runs.
This is all sort-of a pre-requisite to eventually also run upstream/cross-distro inter-op testings in a more structured fashion, anyway.
*Both* paths require knowledge transfer to new people, especially if the old team is too busy with other projects (as I keep hearing). (FWIW, the first path sounds like a lot less effort, and the two presumably also aren't mutually exclusive.)
Yeah, agreed.
So, TL;DR: we're willing to help with extending the distro coverage (for the tests in particular), as long as we know whom to work with to make that happen.
That sounds great Lars. If you are already building/hosting then for testing purposes (with Teuthology) it would only need reporting on the status(es) of the builds and repos. From what you are saying, I am thinking that you are looking to have OpenSuse test with teuthology right? In that case there is no need to push the packages anywhere, shaman will happily redirect tools to the right spot and the repo metadata will help teuthology find what it needs via the HTTP API.
We already have openSUSE on https://docs.ceph.com/docs/master/install/get-packages/ for example. (Probably we want to backport that to the Nautilus and previous releases for openSUSE, though.)
But https://github.com/ceph/ceph-qa-suite/tree/master/distros/supported is somewhat more limited. Also, last updated 3 years, really? ;-) (That's at least what the docs point to.)
Regards, Lars
-- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
On Mon, 5 Aug 2019, Alfredo Deza wrote:
On Mon, Aug 5, 2019 at 10:49 AM Lars Marowsky-Bree <lmb@suse.com> wrote:
On 2019-08-05T13:48:26, Sage Weil <sage@newdream.net> wrote:
Hi Sage,
thanks for jumping in.
I think this is misunderstanding who "we" are. What distributions are tested and built for shaman/chacra and download.ceph.com is a community decision and depends on who is able to invest the effort. If someone shows up willing to do the work, whoever was doing the work before doesn't get to just say no--especially if they don't want to be stuck with that responsibility for all time.
I think the summary here is that SUSE is willing to shoulder part of this work going forward, and if the respective infrastructure is resource constrained, well, the Foundation exists for specifically that purpose.
I see two paths forward: (1) we continue with a monolithic approach to builds and expand the pool of people who understand and contribute to maintaining the build infra, or (2) we rearchitect to a federated approach.
I think it'd be a mix.
Federating the builds - and pushing maintaining distro-packages to, well, the distributors (Fedora, CentOS, Ubuntu, Debian, openSUSE) - seems most sensible. The distros really know best how to build for their platforms.
e.g., for (open)SUSE - I can't talk about any other distro -, we'd be willing to either host the builds on our Open Build Service, or help setup a dedicated OBS instance in the lab somewhere (that'd possibly help with making sure we get them in a timely fashion even when the public infra is overloaded) and helping maintain base images for the OS to run on.
And then once builds are done, pulling them back to the shared infrastructure for test runs.
This is all sort-of a pre-requisite to eventually also run upstream/cross-distro inter-op testings in a more structured fashion, anyway.
*Both* paths require knowledge transfer to new people, especially if the old team is too busy with other projects (as I keep hearing). (FWIW, the first path sounds like a lot less effort, and the two presumably also aren't mutually exclusive.)
Yeah, agreed.
So, TL;DR: we're willing to help with extending the distro coverage (for the tests in particular), as long as we know whom to work with to make that happen.
That sounds great Lars. If you are already building/hosting then for testing purposes (with Teuthology) it would only need reporting on the status(es) of the builds and repos. From what you are saying, I am thinking that you are looking to have OpenSuse test with teuthology right?
In that case there is no need to push the packages anywhere, shaman will happily redirect tools to the right spot and the repo metadata will help teuthology find what it needs via the HTTP API.
The one other piece is getting working images for fog, right? s
We already have openSUSE on https://docs.ceph.com/docs/master/install/get-packages/ for example. (Probably we want to backport that to the Nautilus and previous releases for openSUSE, though.)
But https://github.com/ceph/ceph-qa-suite/tree/master/distros/supported is somewhat more limited. Also, last updated 3 years, really? ;-) (That's at least what the docs point to.)
Regards, Lars
-- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
On Mon, Aug 5, 2019 at 12:56 PM Sage Weil <sage@newdream.net> wrote:
On Mon, 5 Aug 2019, Alfredo Deza wrote:
On Mon, Aug 5, 2019 at 10:49 AM Lars Marowsky-Bree <lmb@suse.com> wrote:
On 2019-08-05T13:48:26, Sage Weil <sage@newdream.net> wrote:
Hi Sage,
thanks for jumping in.
I think this is misunderstanding who "we" are. What distributions are tested and built for shaman/chacra and download.ceph.com is a community decision and depends on who is able to invest the effort. If someone shows up willing to do the work, whoever was doing the work before doesn't get to just say no--especially if they don't want to be stuck with that responsibility for all time.
I think the summary here is that SUSE is willing to shoulder part of this work going forward, and if the respective infrastructure is resource constrained, well, the Foundation exists for specifically that purpose.
I see two paths forward: (1) we continue with a monolithic approach to builds and expand the pool of people who understand and contribute to maintaining the build infra, or (2) we rearchitect to a federated approach.
I think it'd be a mix.
Federating the builds - and pushing maintaining distro-packages to, well, the distributors (Fedora, CentOS, Ubuntu, Debian, openSUSE) - seems most sensible. The distros really know best how to build for their platforms.
e.g., for (open)SUSE - I can't talk about any other distro -, we'd be willing to either host the builds on our Open Build Service, or help setup a dedicated OBS instance in the lab somewhere (that'd possibly help with making sure we get them in a timely fashion even when the public infra is overloaded) and helping maintain base images for the OS to run on.
And then once builds are done, pulling them back to the shared infrastructure for test runs.
This is all sort-of a pre-requisite to eventually also run upstream/cross-distro inter-op testings in a more structured fashion, anyway.
*Both* paths require knowledge transfer to new people, especially if the old team is too busy with other projects (as I keep hearing). (FWIW, the first path sounds like a lot less effort, and the two presumably also aren't mutually exclusive.)
Yeah, agreed.
So, TL;DR: we're willing to help with extending the distro coverage (for the tests in particular), as long as we know whom to work with to make that happen.
That sounds great Lars. If you are already building/hosting then for testing purposes (with Teuthology) it would only need reporting on the status(es) of the builds and repos. From what you are saying, I am thinking that you are looking to have OpenSuse test with teuthology right?
In that case there is no need to push the packages anywhere, shaman will happily redirect tools to the right spot and the repo metadata will help teuthology find what it needs via the HTTP API.
The one other piece is getting working images for fog, right?
If testing in the lab I think that is one thing yes, not entirely sure what else is there, David should have better details
s
We already have openSUSE on https://docs.ceph.com/docs/master/install/get-packages/ for example. (Probably we want to backport that to the Nautilus and previous releases for openSUSE, though.)
But https://github.com/ceph/ceph-qa-suite/tree/master/distros/supported is somewhat more limited. Also, last updated 3 years, really? ;-) (That's at least what the docs point to.)
Regards, Lars
-- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
Alfredo Deza wrote:
On Mon, Aug 5, 2019 at 12:56 PM Sage Weil <sage(a)newdream.net> wrote:
On Mon, Aug 5, 2019 at 10:49 AM Lars Marowsky-Bree <lmb(a)suse.com> wrote:
On 2019-08-05T13:48:26, Sage Weil <sage(a)newdream.net> wrote:
Hi Sage,
thanks for jumping in.
I think this is misunderstanding who "we" are. What
distributions are
tested and built for shaman/chacra and download.ceph.com is a community decision and depends on who is able to invest the effort. If someone shows up willing to do the work, whoever was doing the work before doesn't get to just say no--especially if they don't want to be stuck with
On Mon, 5 Aug 2019, Alfredo Deza wrote: that
responsibility for all time.
I think the summary here is that SUSE is willing to shoulder part of this work going forward, and if the respective infrastructure is resource constrained, well, the Foundation exists for specifically that purpose.
I see two paths forward: (1) we continue with a monolithic approach to builds and expand the pool of people who understand and contribute to maintaining the build infra, or (2) we rearchitect to a federated approach.
I think it'd be a mix.
Federating the builds - and pushing maintaining distro-packages to, well, the distributors (Fedora, CentOS, Ubuntu, Debian, openSUSE) - seems most sensible. The distros really know best how to build for their platforms.
e.g., for (open)SUSE - I can't talk about any other distro -, we'd be willing to either host the builds on our Open Build Service, or help setup a dedicated OBS instance in the lab somewhere (that'd possibly help with making sure we get them in a timely fashion even when the public infra is overloaded) and helping maintain base images for the OS to run on.
And then once builds are done, pulling them back to the shared infrastructure for test runs.
This is all sort-of a pre-requisite to eventually also run upstream/cross-distro inter-op testings in a more structured fashion, anyway.
*Both* paths require knowledge transfer to new people, especially if the old team is too busy with other projects (as I keep hearing). (FWIW, the first path sounds like a lot less effort, and the two presumably also aren't mutually exclusive.)
Yeah, agreed.
So, TL;DR: we're willing to help with extending the distro coverage (for the tests in particular), as long as we know whom to work with to make that happen.
That sounds great Lars. If you are already building/hosting then for testing purposes (with Teuthology) it would only need reporting on the status(es) of the builds and repos. From what you are saying, I am thinking that you are looking to have OpenSuse test with teuthology right?
In that case there is no need to push the packages anywhere, shaman will happily redirect tools to the right spot and the repo metadata will help teuthology find what it needs via the HTTP API.
The one other piece is getting working images for fog, right?
If testing in the lab I think that is one thing yes, not entirely sure what else is there, David should have better details
A cursory glance at teuthology code suggests Leap might already be supported so that's covered. I think ceph-cm-ansible is covered. FOG needs some work. @Sage you asked where docs were. This doc was written for adding an existing distro's major version release but it's mostly applicable to a new distro entirely: https://wiki.sepia.ceph.com/doku.php?id=tasks:major-distro-release The CI is a different animal. I think it's great that Suse is willing to file PRs to get the tools working with OpenSUSE. I will admit, however, I don't know how much time/resources goes into reviewing/testing code PRs. My expertise is more on the infra/services side of things. My primary concern is with system resources. We don't have spare baremetal or VMs to dedicate as OpenSUSE builders unless we take resources away from lab testnodes or existing RPM/DEB builders. I made this clear when Kyr reached out. This means we'd be increasing our spending in OVH's public cloud. Who should ultimately approve or deny the additional spend?
s
We already have openSUSE on https://docs.ceph.com/docs/master/install/get-packages/ for example. (Probably we want to backport that to the Nautilus and previous releases for openSUSE, though.)
But https://github.com/ceph/ceph-qa-suite/tree/master/distros/supported is somewhat more limited. Also, last updated 3 years, really? ;-) (That's at least what the docs point to.)
Regards, Lars
-- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284
(AG Nürnberg)
"Architects should open possibilities and not determine everything." (Ueli Zbinden)
On 2019-08-06T13:25:12, dgallowa@redhat.com wrote:
My primary concern is with system resources. We don't have spare baremetal or VMs to dedicate as OpenSUSE builders unless we take resources away from lab testnodes or existing RPM/DEB builders. I made this clear when Kyr reached out. This means we'd be increasing our spending in OVH's public cloud. Who should ultimately approve or deny the additional spend?
I think that's a tab the foundation should be able to pick up. Funnelling money towards hardware and services to support Ceph development is literally the worthwhile reason why it exists ;-) Regards, Lars -- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
So what are our next step if we want to have, for example, opensuse node running in community OVH?Let say, we modified provisioning scripts and tested them. How should "approve" funding and adding them to release/development builds on jenkins.ceph.com ? Best regards,Kyr On Wed, 2019-08-07 at 14:50 +0200, Lars Marowsky-Bree wrote:
On 2019-08-06T13:25:12, dgallowa@redhat.com wrote:
My primary concern is with system resources. We don't have spare baremetal or VMs to dedicate as OpenSUSE builders unless we take resources away from lab testnodes or existing RPM/DEB builders. I made this clear when Kyr reached out. This means we'd be increasing our spending in OVH's public cloud. Who should ultimately approve or deny the additional spend?
I think that's a tab the foundation should be able to pick up.Funnelling money towards hardware and services to support Cephdevelopment is literally the worthwhile reason why it exists ;-)
Regards, Lars
On 2019-08-14T18:14:15, kyr <kshatskyy@suse.de> wrote:
So what are our next step if we want to have, for example, opensuse node running in community OVH?Let say, we modified provisioning scripts and tested them. How should "approve" funding and adding them to release/development builds on jenkins.ceph.com ?
I don't know what's technically required. David, Sage, Alfredo? But this can't be a lot - who controls Ceph's OVH investment? If they could just say how much money we need for this (roughly), I'm sure the board can approve. I don't expect this to be a huge sum. Thanks, Lars -- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
On Mon, Aug 5, 2019 at 9:48 AM Sage Weil <sage@newdream.net> wrote:
On Mon, 5 Aug 2019, Alfredo Deza wrote:
I think we are deviating a bit on the discussion:
1) We are not in the capacity of adding any other new distro to our build toolchain, for development or releases.
2) We can own the effort of adding the mechanisms needed in shaman (shaman.ceph.com) so that community-built Ceph packages/repos can report there. This will entail adding authentication so that updates can verify the source.
For #2 specifically, the shaman dashboard allows updating the status of a build (started, building, failed, succeeded) as well as the repo. Tools like teuthology query shaman for the state and location of repos. The API is detailed here: https://github.com/ceph/shaman
I think this is misunderstanding who "we" are. What distributions are tested and built for shaman/chacra and download.ceph.com is a community decision and depends on who is able to invest the effort.
I agree here, I am not implying that the decision is (or should) not driven by the community.
If someone shows up willing to do the work, whoever was doing the work before doesn't get to just say no--especially if they don't want to be stuck with that responsibility for all time.
Our infrastructure (Jenkins, repo builders, OVH services, shaman) wasn't built with the ability to allow community contributions. At some point we tried with CERN who graciously spared a few nodes to build Ceph but this had lots of issues on both ends and ended up not moving forward with that way of extending how we build. Note that this is different from "what distro is going to be built" - adding nodes is just a piece of the puzzle.
I see two paths forward: (1) we continue with a monolithic approach to builds and expand the pool of people who understand and contribute to maintaining the build infra, or (2) we rearchitect to a federated approach. *Both* paths require knowledge transfer to new people, especially if the old team is too busy with other projects (as I keep hearing). (FWIW, the first path sounds like a lot less effort, and the two presumably also aren't mutually exclusive.)
I am all for anyone wanting to step up and do the work, but I see that functioning well if we stick with #2 which is what I am proposing: allow the community to build and maintain what is needed for other builds/distro combinations and allow those to be reported in shaman. Again, this will take some effort that I am happy to assist with.
I talked to David a couple weeks back about getting a walk through bringing Debian Buster up in fog so that we could document the process and the response I got was that it is all already documented. It then took him the rest of the day to get a working fog image. :) Where is this documentation?
Thanks! sage
On Mon, Aug 5, 2019 at 4:21 AM Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> wrote:
W dniu 02.08.2019 o 19:55, Alfredo Deza pisze:
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
Kyrlyo wanted to add openSUSE to the set of distributions that get build for and used by teuthology, for example:
However, seems officially there are "no plans" to add other platforms but Ubuntu and CentOS, so Alfredo closed the PRs.
At Linaro we would like to help with adding Debian to the list of distributions Ceph is built for. We can provide aarch64 (arm64) machines for it.
Hey Lars. Adding a distro for builds is a very involved problem to solve. We don't keep a detailed list of everything that is needed but I will try to go over some of the well-known items:
2) A new distribution added *must* exist in the cloud provider (OVH in this case) that can spin up a VM for builds (as of this writing, there is only an opensuse42 image available from 2016)
https://www.ovh.co.uk/dedicated_servers/distributions/ lists Debian 10 as available. Not that this page is up-to-date as it does not even list Ubuntu 18.04
3) *All* the building scripts must be revised to ensure that the new distribution is accounted for. I did some of this work when adding Ubuntu Bionic and it was non-trivial, error prone, and it took about two weeks to really get it right with the help of other people.
I can probably work on it.
4) The services that ensure that images come up and are prepared to build Ceph have to be udpated as well to ensure that the minimum requirements are installed so that the machine is operational
5) If the new distro is Python3 only we will need to update all tooling that interacts with a jenkins node - we are not there yet as all our tooling is Python2 exclusive.
Debian 10 'buster' has both Python 2.7 and Python 3.7 so should not be a problem - we can start with py2 and then update to py3 once Ceph move.
At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
Will watch it later.
A few of the things we brought up is that we (the Ceph infrastructure team and our services) aren't prepared to accommodate multiple other distributions and that we are trying to get away from taking on the load of maintenance in our systems and looking to other build/repo solutions. One of these solutions is the CentOS storage-sig which we are trying to coordinate to build and host repos for us there.
In addition to that, we mentioned that we would like to see a wider community effort go into building and hosting Ceph in separate systems, maybe with a special signing key (our release signing process is pretty inflexible!) so that others who are building development repositories can ensure their authenticity, while giving us the ability to revoke keys as needed.
In the past, we've gotten asked to re-enable the Debian builds, which puts us into a similar problem (maintenance burden, script updates, and other items already mentioned), and we've had to turn that down. As Ken mentions in the presentation, we really want to be helpful and accommodating to the wider community, but we can't do it on our own and with our infrastructure as it is today - we are maxed out.
Distributing community signing keys, or allowing other builders to submit status updates into shaman.ceph.com for testing scheduling is yet-to-be-done work, but I am open to have those conversations so that we can move forwards with more distros and better testing.
At Linaro we can arrange machines for building and testing. Space for hosting Debian/arm64 repos too. We are fine on using those machines also to build packages for other arm64 distributions. _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Mon, 5 Aug 2019, Alfredo Deza wrote:
On Mon, Aug 5, 2019 at 9:48 AM Sage Weil <sage@newdream.net> wrote:
On Mon, 5 Aug 2019, Alfredo Deza wrote:
I think we are deviating a bit on the discussion:
1) We are not in the capacity of adding any other new distro to our build toolchain, for development or releases.
2) We can own the effort of adding the mechanisms needed in shaman (shaman.ceph.com) so that community-built Ceph packages/repos can report there. This will entail adding authentication so that updates can verify the source.
For #2 specifically, the shaman dashboard allows updating the status of a build (started, building, failed, succeeded) as well as the repo. Tools like teuthology query shaman for the state and location of repos. The API is detailed here: https://github.com/ceph/shaman
I think this is misunderstanding who "we" are. What distributions are tested and built for shaman/chacra and download.ceph.com is a community decision and depends on who is able to invest the effort.
I agree here, I am not implying that the decision is (or should) not driven by the community.
If someone shows up willing to do the work, whoever was doing the work before doesn't get to just say no--especially if they don't want to be stuck with that responsibility for all time.
Our infrastructure (Jenkins, repo builders, OVH services, shaman) wasn't built with the ability to allow community contributions.
Again, I'm unclear what "our" and "community" mean here. All of the build infrastructure (with the exception of the signing host) are in the sepia lab, which is a shared community (upstream) resource. Adjusting access to individual hosts within the lab is trivial.
At some point we tried with CERN who graciously spared a few nodes to build Ceph but this had lots of issues on both ends and ended up not moving forward with that way of extending how we build.
Agree on the external jenkins workers, or integration with anything that is outside of sepia--that's a different ball of wax! sage
Note that this is different from "what distro is going to be built" - adding nodes is just a piece of the puzzle.
I see two paths forward: (1) we continue with a monolithic approach to builds and expand the pool of people who understand and contribute to maintaining the build infra, or (2) we rearchitect to a federated approach. *Both* paths require knowledge transfer to new people, especially if the old team is too busy with other projects (as I keep hearing). (FWIW, the first path sounds like a lot less effort, and the two presumably also aren't mutually exclusive.)
I am all for anyone wanting to step up and do the work, but I see that functioning well if we stick with #2 which is what I am proposing: allow the community to build and maintain what is needed for other builds/distro combinations and allow those to be reported in shaman. Again, this will take some effort that I am happy to assist with.
I talked to David a couple weeks back about getting a walk through bringing Debian Buster up in fog so that we could document the process and the response I got was that it is all already documented. It then took him the rest of the day to get a working fog image. :) Where is this documentation?
Thanks! sage
On Mon, Aug 5, 2019 at 4:21 AM Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> wrote:
W dniu 02.08.2019 o 19:55, Alfredo Deza pisze:
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
Kyrlyo wanted to add openSUSE to the set of distributions that get build for and used by teuthology, for example:
However, seems officially there are "no plans" to add other platforms but Ubuntu and CentOS, so Alfredo closed the PRs.
At Linaro we would like to help with adding Debian to the list of distributions Ceph is built for. We can provide aarch64 (arm64) machines for it.
Hey Lars. Adding a distro for builds is a very involved problem to solve. We don't keep a detailed list of everything that is needed but I will try to go over some of the well-known items:
2) A new distribution added *must* exist in the cloud provider (OVH in this case) that can spin up a VM for builds (as of this writing, there is only an opensuse42 image available from 2016)
https://www.ovh.co.uk/dedicated_servers/distributions/ lists Debian 10 as available. Not that this page is up-to-date as it does not even list Ubuntu 18.04
3) *All* the building scripts must be revised to ensure that the new distribution is accounted for. I did some of this work when adding Ubuntu Bionic and it was non-trivial, error prone, and it took about two weeks to really get it right with the help of other people.
I can probably work on it.
4) The services that ensure that images come up and are prepared to build Ceph have to be udpated as well to ensure that the minimum requirements are installed so that the machine is operational
5) If the new distro is Python3 only we will need to update all tooling that interacts with a jenkins node - we are not there yet as all our tooling is Python2 exclusive.
Debian 10 'buster' has both Python 2.7 and Python 3.7 so should not be a problem - we can start with py2 and then update to py3 once Ceph move.
At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
Will watch it later.
A few of the things we brought up is that we (the Ceph infrastructure team and our services) aren't prepared to accommodate multiple other distributions and that we are trying to get away from taking on the load of maintenance in our systems and looking to other build/repo solutions. One of these solutions is the CentOS storage-sig which we are trying to coordinate to build and host repos for us there.
In addition to that, we mentioned that we would like to see a wider community effort go into building and hosting Ceph in separate systems, maybe with a special signing key (our release signing process is pretty inflexible!) so that others who are building development repositories can ensure their authenticity, while giving us the ability to revoke keys as needed.
In the past, we've gotten asked to re-enable the Debian builds, which puts us into a similar problem (maintenance burden, script updates, and other items already mentioned), and we've had to turn that down. As Ken mentions in the presentation, we really want to be helpful and accommodating to the wider community, but we can't do it on our own and with our infrastructure as it is today - we are maxed out.
Distributing community signing keys, or allowing other builders to submit status updates into shaman.ceph.com for testing scheduling is yet-to-be-done work, but I am open to have those conversations so that we can move forwards with more distros and better testing.
At Linaro we can arrange machines for building and testing. Space for hosting Debian/arm64 repos too. We are fine on using those machines also to build packages for other arm64 distributions. _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Mon, Aug 5, 2019 at 12:55 PM Sage Weil <sage@newdream.net> wrote:
On Mon, 5 Aug 2019, Alfredo Deza wrote:
On Mon, Aug 5, 2019 at 9:48 AM Sage Weil <sage@newdream.net> wrote:
On Mon, 5 Aug 2019, Alfredo Deza wrote:
I think we are deviating a bit on the discussion:
1) We are not in the capacity of adding any other new distro to our build toolchain, for development or releases.
2) We can own the effort of adding the mechanisms needed in shaman (shaman.ceph.com) so that community-built Ceph packages/repos can report there. This will entail adding authentication so that updates can verify the source.
For #2 specifically, the shaman dashboard allows updating the status of a build (started, building, failed, succeeded) as well as the repo. Tools like teuthology query shaman for the state and location of repos. The API is detailed here: https://github.com/ceph/shaman
I think this is misunderstanding who "we" are. What distributions are tested and built for shaman/chacra and download.ceph.com is a community decision and depends on who is able to invest the effort.
I agree here, I am not implying that the decision is (or should) not driven by the community.
If someone shows up willing to do the work, whoever was doing the work before doesn't get to just say no--especially if they don't want to be stuck with that responsibility for all time.
Our infrastructure (Jenkins, repo builders, OVH services, shaman) wasn't built with the ability to allow community contributions.
Again, I'm unclear what "our" and "community" mean here. All of the build infrastructure (with the exception of the signing host) are in the sepia lab, which is a shared community (upstream) resource. Adjusting access to individual hosts within the lab is trivial.
At some point we tried with CERN who graciously spared a few nodes to build Ceph but this had lots of issues on both ends and ended up not moving forward with that way of extending how we build.
Agree on the external jenkins workers, or integration with anything that is outside of sepia--that's a different ball of wax!
I think that we may have been discussing different parts of the infrastructure... so yes, I agree that the lab should have the ability to run tests and get access to others, seems like that has always been in place. My concerns are more with what produces and hosts packages, which I think can be solved by allowing shaman to get reports. It also seems that I may have come up as a sort of gate-keeper here and I want to ensure that is not the case. From having introduced the build system a while ago, I can tell that there are issues if we wanted to go monolithic (vs. federated). I am happy to assist where I can.
sage
Note that this is different from "what distro is going to be built" - adding nodes is just a piece of the puzzle.
I see two paths forward: (1) we continue with a monolithic approach to builds and expand the pool of people who understand and contribute to maintaining the build infra, or (2) we rearchitect to a federated approach. *Both* paths require knowledge transfer to new people, especially if the old team is too busy with other projects (as I keep hearing). (FWIW, the first path sounds like a lot less effort, and the two presumably also aren't mutually exclusive.)
I am all for anyone wanting to step up and do the work, but I see that functioning well if we stick with #2 which is what I am proposing: allow the community to build and maintain what is needed for other builds/distro combinations and allow those to be reported in shaman. Again, this will take some effort that I am happy to assist with.
I talked to David a couple weeks back about getting a walk through bringing Debian Buster up in fog so that we could document the process and the response I got was that it is all already documented. It then took him the rest of the day to get a working fog image. :) Where is this documentation?
Thanks! sage
On Mon, Aug 5, 2019 at 4:21 AM Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> wrote:
W dniu 02.08.2019 o 19:55, Alfredo Deza pisze:
On Fri, Aug 2, 2019 at 12:20 PM Lars Marowsky-Bree <lmb@suse.com> wrote:
> Kyrlyo wanted to add openSUSE to the set of distributions that get > build for and used by teuthology, for example:
> However, seems officially there are "no plans" to add other > platforms but Ubuntu and CentOS, so Alfredo closed the PRs.
At Linaro we would like to help with adding Debian to the list of distributions Ceph is built for. We can provide aarch64 (arm64) machines for it.
Hey Lars. Adding a distro for builds is a very involved problem to solve. We don't keep a detailed list of everything that is needed but I will try to go over some of the well-known items:
2) A new distribution added *must* exist in the cloud provider (OVH in this case) that can spin up a VM for builds (as of this writing, there is only an opensuse42 image available from 2016)
https://www.ovh.co.uk/dedicated_servers/distributions/ lists Debian 10 as available. Not that this page is up-to-date as it does not even list Ubuntu 18.04
3) *All* the building scripts must be revised to ensure that the new distribution is accounted for. I did some of this work when adding Ubuntu Bionic and it was non-trivial, error prone, and it took about two weeks to really get it right with the help of other people.
I can probably work on it.
4) The services that ensure that images come up and are prepared to build Ceph have to be udpated as well to ensure that the minimum requirements are installed so that the machine is operational
5) If the new distro is Python3 only we will need to update all tooling that interacts with a jenkins node - we are not there yet as all our tooling is Python2 exclusive.
Debian 10 'buster' has both Python 2.7 and Python 3.7 so should not be a problem - we can start with py2 and then update to py3 once Ceph move.
At Cephalocon, Ken Dreyer and me did a presentation on what exactly entails building Ceph for both development and releases, the problems we've faced and where we would like to head next. It might be useful to go through if you haven't already: https://www.youtube.com/watch?v=seHyiQT8YJM
Will watch it later.
A few of the things we brought up is that we (the Ceph infrastructure team and our services) aren't prepared to accommodate multiple other distributions and that we are trying to get away from taking on the load of maintenance in our systems and looking to other build/repo solutions. One of these solutions is the CentOS storage-sig which we are trying to coordinate to build and host repos for us there.
In addition to that, we mentioned that we would like to see a wider community effort go into building and hosting Ceph in separate systems, maybe with a special signing key (our release signing process is pretty inflexible!) so that others who are building development repositories can ensure their authenticity, while giving us the ability to revoke keys as needed.
In the past, we've gotten asked to re-enable the Debian builds, which puts us into a similar problem (maintenance burden, script updates, and other items already mentioned), and we've had to turn that down. As Ken mentions in the presentation, we really want to be helpful and accommodating to the wider community, but we can't do it on our own and with our infrastructure as it is today - we are maxed out.
Distributing community signing keys, or allowing other builders to submit status updates into shaman.ceph.com for testing scheduling is yet-to-be-done work, but I am open to have those conversations so that we can move forwards with more distros and better testing.
At Linaro we can arrange machines for building and testing. Space for hosting Debian/arm64 repos too. We are fine on using those machines also to build packages for other arm64 distributions. _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (9)
-
Alfredo Deza
-
Brad Hubbard
-
dgallowa＠redhat.com
-
kyr
-
kyrylo.shatskyy
-
Lars Marowsky-Bree
-
Marcin Juszkiewicz
-
Sage Weil
-
Tim Serong