Hi Dan & others, A few months back we (the Orchestration team working on cephadm), discussed the new compiled [1] cephadm in a CLT call and we briefly discussed it on the list [2]. I wanted to revisit that conversation as Reef's release is quickly coming up. To summarize: Previous versions of Ceph contained a single python source file for cephadm. Upstream users were instructed to copy this source file directly out of the Ceph git tree and execute the file using python. Now, the ceph build process creates an executable python zipapp from the source file and the old instructions are out of date. We would like to request assistance with making it possible for users who are planning on bootstrapping clusters to download an already compiled version of cephadm from a canonical location. A secondary goal would also be to sign that binary. However, I would treat that as a nice to have since the current workflow doesn't have this. I'd love to discuss the technical aspects of this and get some code to implement this in place. I'd be happy to continue the conversation here - or if you prefer - in the discussion forum of your choice. Thanks very much! [1] - It is still python, and it's not compiled to native binaries but is rather a zipapp - https://docs.python.org/3/library/zipapp.html [2] - I wanted to link to the previous discussion but neither the Archived-At header url or search turn it up. So maybe it got lost in the infra issues a while back. If you need references to the previous thread, I can forward them because I still have them on my mailserver.
A long long time ago when this first came up, I suggested we have instructions that mechanized finding the package from download.ceph.com and downloading/extracting the zipapp from it. I felt strongly at the time that we should try our best to avoid having two sources of truth for the Real Tested Binary, and packages were the obvious answer, but using the actual package management software was not suitable, so wrote the extraction code. We didn't ever release that code for extracting from the package, but the technique found its way into the build jobs in Oct 2021: At present, at least for the CI builds, the cephadm binary/zipapp is extracted from packages during the build and placed, separately, in the package repo as a plain file, unmanaged by the package managers. See https://github.com/ceph/ceph-build/blob/main/ceph-dev-new-build/build/build_... and https://github.com/ceph/ceph-build/blob/main/scripts/build_utils.sh#L984-L98... (the asymmetry was there when I got there). That code is also present in the "ceph-dev-build" and "ceph-build" jobs (which do candidate builds for named releases and final release builds, respectively), and indeed cephadm is present in the rpm repos (for example, http://download.ceph.com/rpm-17.2.6/el8/noarch/cephadm) but I don't see the cephadm binary on download.ceph.com for deb repos. (It's probably the "sync to download.ceph.com" step being unaware it should copy anything besides .deb.) My suggestion would be to debug the issue with the deb release process, and document that the binary can be found in the appropriate path corresponding to the release in download.ceph.com. On 5/16/2023 10:10 AM, John Mulligan wrote:
Hi Dan & others,
A few months back we (the Orchestration team working on cephadm), discussed the new compiled [1] cephadm in a CLT call and we briefly discussed it on the list [2]. I wanted to revisit that conversation as Reef's release is quickly coming up.
To summarize: Previous versions of Ceph contained a single python source file for cephadm. Upstream users were instructed to copy this source file directly out of the Ceph git tree and execute the file using python. Now, the ceph build process creates an executable python zipapp from the source file and the old instructions are out of date.
We would like to request assistance with making it possible for users who are planning on bootstrapping clusters to download an already compiled version of cephadm from a canonical location. A secondary goal would also be to sign that binary. However, I would treat that as a nice to have since the current workflow doesn't have this.
I'd love to discuss the technical aspects of this and get some code to implement this in place. I'd be happy to continue the conversation here - or if you prefer - in the discussion forum of your choice. Thanks very much!
[1] - It is still python, and it's not compiled to native binaries but is rather a zipapp - https://docs.python.org/3/library/zipapp.html
[2] - I wanted to link to the previous discussion but neither the Archived-At header url or search turn it up. So maybe it got lost in the infra issues a while back. If you need references to the previous thread, I can forward them because I still have them on my mailserver.
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Wednesday, May 17, 2023 6:00:01 PM EDT Dan Mick wrote:
A long long time ago when this first came up, I suggested we have instructions that mechanized finding the package from download.ceph.com and downloading/extracting the zipapp from it. I felt strongly at the time that we should try our best to avoid having two sources of truth for the Real Tested Binary, and packages were the obvious answer, but using the actual package management software was not suitable, so wrote the extraction code.
We didn't ever release that code for extracting from the package, but the technique found its way into the build jobs in Oct 2021:
At present, at least for the CI builds, the cephadm binary/zipapp is extracted from packages during the build and placed, separately, in the package repo as a plain file, unmanaged by the package managers. See https://github.com/ceph/ceph-build/blob/main/ceph-dev-new-build/build/build_ rpm#L59-L63 and https://github.com/ceph/ceph-build/blob/main/scripts/build_utils.sh#L984-L98 8 (the asymmetry was there when I got there).
OK, that's a good start. :-)
That code is also present in the "ceph-dev-build" and "ceph-build" jobs (which do candidate builds for named releases and final release builds, respectively), and indeed cephadm is present in the rpm repos (for example, http://download.ceph.com/rpm-17.2.6/el8/noarch/cephadm) but I don't see the cephadm binary on download.ceph.com for deb repos. (It's probably the "sync to download.ceph.com" step being unaware it should copy anything besides .deb.)
Unfortunately, I don't think this RPM derived cephadm is appropriate for general uses. RPM mangles the shebang line to something RH-distro specific so this `cephadm` wouldn't work properly on distros like debian, ubuntu, etc. I think it does this to the zipapp too - but I would need to verify it. I'm aware of chacra.ceph.com (and 1.chacra.ceph.com) but all I can get from it is JSON and I would like to inspect the actual binary. I'm probably not using the proper interface but I don't know where to look. If the file extracted from the ubuntu deb package does not get mangled and the shebang simply refers to `python3` then we'd be good there once the issue with the deb release process is resolved.
My suggestion would be to debug the issue with the deb release process, and document that the binary can be found in the appropriate path corresponding to the release in download.ceph.com.
Sounds like a plan, is that something you expect us to do, or is it something you are planning to do? I'm happy to help but I'm afraid I wouldn't know where to start so at least I'd need some initial "hand holding". Thanks!
On 5/16/2023 10:10 AM, John Mulligan wrote:
Hi Dan & others,
A few months back we (the Orchestration team working on cephadm), discussed the new compiled [1] cephadm in a CLT call and we briefly discussed it on the list [2]. I wanted to revisit that conversation as Reef's release is quickly coming up.
To summarize: Previous versions of Ceph contained a single python source file for cephadm. Upstream users were instructed to copy this source file directly out of the Ceph git tree and execute the file using python. Now, the ceph build process creates an executable python zipapp from the source file and the old instructions are out of date.
We would like to request assistance with making it possible for users who are planning on bootstrapping clusters to download an already compiled version of cephadm from a canonical location. A secondary goal would also be to sign that binary. However, I would treat that as a nice to have since the current workflow doesn't have this.
I'd love to discuss the technical aspects of this and get some code to implement this in place. I'd be happy to continue the conversation here - or if you prefer - in the discussion forum of your choice. Thanks very much!
[1] - It is still python, and it's not compiled to native binaries but is rather a zipapp - https://docs.python.org/3/library/zipapp.html
[2] - I wanted to link to the previous discussion but neither the Archived-At header url or search turn it up. So maybe it got lost in the infra issues a while back. If you need references to the previous thread, I can forward them because I still have them on my mailserver.
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 5/18/2023 6:56 AM, John Mulligan wrote:
That code is also present in the "ceph-dev-build" and "ceph-build" jobs (which do candidate builds for named releases and final release builds, respectively), and indeed cephadm is present in the rpm repos (for example, http://download.ceph.com/rpm-17.2.6/el8/noarch/cephadm) but I don't see the cephadm binary on download.ceph.com for deb repos. (It's probably the "sync to download.ceph.com" step being unaware it should copy anything besides .deb.)
Unfortunately, I don't think this RPM derived cephadm is appropriate for general uses. RPM mangles the shebang line to something RH-distro specific so this `cephadm` wouldn't work properly on distros like debian, ubuntu, etc. I think it does this to the zipapp too - but I would need to verify it.
Oh, I wasn't suggesting to use the RPM version for deb releases, just that half the solution makes it all the way to the release website.
I'm aware of chacra.ceph.com (and 1.chacra.ceph.com) but all I can get from it is JSON and I would like to inspect the actual binary. I'm probably not using the proper interface but I don't know where to look.
shaman is the querier for where to find binaries; chacra is the storage, and generally not meant to be queried directly (although you can, shaman has the better interfaces). See the readme at github.com/ceph/shaman for how to query programmatically, or poke around the website interface. The problem is that the repo construction doesn't carry that binary over by default; that's the missing piece. I was wrong about it only affecting download.ceph.com; it doesn't show up in the actual deb repos for ceph-ci builds either. You *can* drill down and find it, but you have to find the specific chacra host (of [1234].chacra) that contains the binary and then go find it in that repo, which involves either logging in or knowing something magic about the URLs that I've currently forgotten :) so the deb repo construction bit needs some work.
If the file extracted from the ubuntu deb package does not get mangled and the shebang simply refers to `python3` then we'd be good there once the issue with the deb release process is resolved.
I'm pretty sure the binary in the package doesn't get mangled at install time, so whatever's there by definition should run on the OS it's built for.
My suggestion would be to debug the issue with the deb release process, and document that the binary can be found in the appropriate path corresponding to the release in download.ceph.com.
Sounds like a plan, is that something you expect us to do, or is it something you are planning to do? I'm happy to help but I'm afraid I wouldn't know where to start so at least I'd need some initial "hand holding".
Thanks!
I'm sorta up to my ears in fires with the lab at this moment, but I can work with whoever to help and advise; if you have time, DM me somehow and we can look into it together.
On 5/16/2023 10:10 AM, John Mulligan wrote:
Hi Dan & others,
A few months back we (the Orchestration team working on cephadm), discussed the new compiled [1] cephadm in a CLT call and we briefly discussed it on the list [2]. I wanted to revisit that conversation as Reef's release is quickly coming up.
To summarize: Previous versions of Ceph contained a single python source file for cephadm. Upstream users were instructed to copy this source file directly out of the Ceph git tree and execute the file using python. Now, the ceph build process creates an executable python zipapp from the source file and the old instructions are out of date.
We would like to request assistance with making it possible for users who are planning on bootstrapping clusters to download an already compiled version of cephadm from a canonical location. A secondary goal would also be to sign that binary. However, I would treat that as a nice to have since the current workflow doesn't have this.
I'd love to discuss the technical aspects of this and get some code to implement this in place. I'd be happy to continue the conversation here - or if you prefer - in the discussion forum of your choice. Thanks very much!
[1] - It is still python, and it's not compiled to native binaries but is rather a zipapp - https://docs.python.org/3/library/zipapp.html
[2] - I wanted to link to the previous discussion but neither the Archived-At header url or search turn it up. So maybe it got lost in the infra issues a while back. If you need references to the previous thread, I can forward them because I still have them on my mailserver.
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Thu, May 18, 2023 at 9:57 AM John Mulligan <phlogistonjohn@asynchrono.us> wrote:
Unfortunately, I don't think this RPM derived cephadm is appropriate for general uses. RPM mangles the shebang line to something RH-distro specific so
That's not entirely accurate. Packaging guidelines these days say that the python shebang should be /usr/bin/python3. I believe that's true even for Debian and Ubuntu. Some developers think it's smart to use #! /usr/bin/env python. In gluster we had a mix of that and some #! /usr/bin/python or #! /usr/bin/python2 shebangs. These were all changed to #! /usr/bin/python3. (Obviously on old distros like RHEL7 it needs to be /usr/bin/python2 and in gluster we added some sed magic to the glusterfs.spec to fix them at build time. Before its EOL, RHEL7 was one of the last major distros still using python2.) On Fedora, RHEL, and SUSE, rpmbuild will whine about incorrect shebangs during the build. It may even change them, I don't remember at this point. IIRC, if it changes them, it changes them to /usr/bin/python3.
this `cephadm` wouldn't work properly on distros like debian, ubuntu, etc.
If all the shebangs are #! /usr/bin/python3 — either in the source to start with, or changed by the rpmbuild — they work on Debian and Ubuntu just fine. Or they should. If they don't, there's a different problem. If cephadm is a separate package (for some definition of package) I have to wonder why you're *not* working on adding them to Fedora and CentOS Storage SIG, where Ceph and several related packages are already being packaged for wide consumption. -- Kaleb
On Thu, May 18, 2023 at 4:58 PM Kaleb Keithley <kkeithle@redhat.com> wrote:
On Thu, May 18, 2023 at 9:57 AM John Mulligan < phlogistonjohn@asynchrono.us> wrote:
Unfortunately, I don't think this RPM derived cephadm is appropriate for general uses. RPM mangles the shebang line to something RH-distro specific so
That's not entirely accurate.
Packaging guidelines these days say that the python shebang should be /usr/bin/python3. I believe that's true even for Debian and Ubuntu.
It is true. See https://www.debian.org/doc/packaging-manuals/python-policy/index.html#interp... -- Kaleb
On Thursday, May 18, 2023 4:58:30 PM EDT Kaleb Keithley wrote:
On Thu, May 18, 2023 at 9:57 AM John Mulligan <phlogistonjohn@asynchrono.us> wrote:
Unfortunately, I don't think this RPM derived cephadm is appropriate for general uses. RPM mangles the shebang line to something RH-distro specific so
That's not entirely accurate.
Packaging guidelines these days say that the python shebang should be /usr/bin/python3. I believe that's true even for Debian and Ubuntu.
Agreed!
Some developers think it's smart to use #! /usr/bin/env python. In gluster we had a mix of that and some #! /usr/bin/python or #! /usr/bin/python2 shebangs. These were all changed to #! /usr/bin/python3. (Obviously on old distros like RHEL7 it needs to be /usr/bin/python2 and in gluster we added some sed magic to the glusterfs.spec to fix them at build time. Before its EOL, RHEL7 was one of the last major distros still using python2.)
On Fedora, RHEL, and SUSE, rpmbuild will whine about incorrect shebangs during the build. It may even change them, I don't remember at this point. IIRC, if it changes them, it changes them to /usr/bin/python3.
Interesting.... when I wrote that I had just looked at a non-zipapp version of cephadm that was extracted from the RPM and it had a weird "system python" shebang (`/usr/libexec/platform-python -s`) see: `https://download.ceph.com/rpm-17.2.6/el8/noarch/ and download the `cephadm` the system extracted there. I was worried, but had not confirmed that rpmbuild was still doing that. I should probably get access to a proper main/reef build and see what the result is before saying more on this. :-)
this `cephadm` wouldn't work properly on distros like debian, ubuntu, etc.
If all the shebangs are #! /usr/bin/python3 — either in the source to start with, or changed by the rpmbuild — they work on Debian and Ubuntu just fine. Or they should. If they don't, there's a different problem.
OK, that's good news. With the new zipapp compilation we have the ability to force the shebang to be something we explicitly set (like "/usr/bin/python3"). As of this moment it takes whatever shebang is default for python's zipapp module & function.
If cephadm is a separate package (for some definition of package) I have to wonder why you're *not* working on adding them to Fedora and CentOS Storage SIG, where Ceph and several related packages are already being packaged for wide consumption.
cephadm is part of the packages. For RPM it is one of the many sub-packages in ceph.spec.in. For another example, see the same dir linked above and the file `cephadm-17.2.6-0.el8.noarch.rpm`. That's probably the "best" way to get it on one's system. However, there's been a long standing workflow that one can aqucuire a "standalone" cephadm and bootstrap using that. See https://docs.ceph.com/en/pacific/cephadm/install/#install-cephadm and the curl based installation section. This is a workflow that is currently broken on "main" and "reef" and we're trying to replace/replicate. For exactly what systems and workflows we expect to use the curl based workflow, I'm not clear on but the idea may be to support other distros that have docker/podman support (because containers!). But I haven't been working on cephadm long enough to know - perhaps Adam will chime in here. :-)
On Thu, May 18, 2023 at 5:19 PM John Mulligan <phlogistonjohn@asynchrono.us> wrote:
On Thu, May 18, 2023 at 9:57 AM John Mulligan <
On Thursday, May 18, 2023 4:58:30 PM EDT Kaleb Keithley wrote: phlogistonjohn@asynchrono.us>
wrote: ... If cephadm is a separate package (for some definition of package) I have to wonder why you're *not* working on adding them to Fedora and CentOS Storage SIG, where Ceph and several related packages are already being packaged for wide consumption.
cephadm is part of the packages. For RPM it is one of the many sub-packages in ceph.spec.in.
Yeah, my bad. I didn't remember every one of the dozens of subpackages. But there is a cephadm subpackage in the 17.2.6 (and earlier) packages in fedora and centos storage sig. I leapt to an incorrect conclusion when you said you were looking for someplace to distribute cephadm from. The implication seemed to be that it was a completely separate build from the rest of ceph. -- Kaleb
On Thursday, May 18, 2023 8:08:55 PM EDT Kaleb Keithley wrote:
On Thu, May 18, 2023 at 5:19 PM John Mulligan <phlogistonjohn@asynchrono.us> wrote:
On Thursday, May 18, 2023 4:58:30 PM EDT Kaleb Keithley wrote:
On Thu, May 18, 2023 at 9:57 AM John Mulligan <
phlogistonjohn@asynchrono.us> [[snip]]
Yeah, my bad. I didn't remember every one of the dozens of subpackages. But there is a cephadm subpackage in the 17.2.6 (and earlier) packages in fedora and centos storage sig. I leapt to an incorrect conclusion when you said you were looking for someplace to distribute cephadm from. The implication seemed to be that it was a completely separate build from the rest of ceph.
No problem. I'm fairly new to this area of the code myself. I'm frequently doubling back to check things I just was previously assuming. Anyway I looked into what packages set what shebang and now it's looking like the centos 9 based builds might be the best for general non-package distribution. ----- Examining Reef builds from https://shaman.ceph.com/builds/ceph/reef/ 35c3c9bdcee409e4b0cf5c304febcf8bd62a139a/ https://2.chacra.ceph.com/r/ceph/reef/ 35c3c9bdcee409e4b0cf5c304febcf8bd62a139a/ubuntu/jammy/flavors/default/pool/ main/c/ceph/ - Ubuntu "Jammy" builds - cephadm _not_ extracted from the package - cephadm package exists: cephadm_18.0.0-3148-g35c3c9bd-1jammy_amd64.deb - shebang command in zipapp: /usr/bin/python3.10 https://2.chacra.ceph.com/r/ceph/reef/ 35c3c9bdcee409e4b0cf5c304febcf8bd62a139a/centos/8/flavors/default/noarch/ - CentoOS 8 builds (noarch) - cephadm is extracted from the package - cephadm package exists: cephadm-18.0.0-3148.g35c3c9bd.el8.noarch.rpm - shebang command in zipapp: /usr/libexec/platform-python -s https://3.chacra.ceph.com/r/ceph/reef/ 35c3c9bdcee409e4b0cf5c304febcf8bd62a139a/centos/9/flavors/default/noarch/ - CentOS 9 builds (noarch) - cephadm is extracted from the package - cephadm package exists: cephadm-18.0.0-3148.g35c3c9bd.el9.noarch.rpm - shebang command in zipapp: /usr/bin/python3 -s ----- The -s option tells python not to add the user site directory to the python module path. This is fine for cephadm's purposes. So the centos 8 package does change the shebang to something I don't expect to work outside of Fedora/CentOS/RHEL, so I wouldn't recommend downloading that. The CentOS 9 package simply has /usr/bin/python3 which is the most general, so I think that's ideal. The Ubuntu package has /usr/bin/python3.10 which would not be ideal because not all of these "other" platforms we're trying to support would all have python 3.10, some might have 3.9 or 3.11, etc. This might be an artifact of how we build cephadm on ubuntu though, so I still need to investigate that. At the same time I think I have found the code that needs changing to support extracting cephadm from unbuntu packages and I'll be contacting Dan directly to work on that.
participants (3)
-
Dan Mick
-
John Mulligan
-
Kaleb Keithley