Thank you, John! BTW I looked for that "[1]" footnote in your reply but couldn't find any? Kind Regards, Ernesto On Mon, Mar 8, 2021 at 9:57 PM John Spray <jcspray@gmail.com> wrote:
Hey Ernesto,
I haven't used pex, it looks like it wants to generate executables, but they're zip files that include a dump of vendored packages (and it has an --inherit-path option to respect systemwide packages) so maybe it's a useful building block.
To be honest I never ended up with an approach that made me happy[1]. My hazy recollection from hacking builds out of virtualenvs is that virtualenv tends to get a bit confusing in the interplay between --system-site-packages and dependency resolution for packages within the env (sorry to be vague, this was a few years ago).
If I was doing it today, my starting point would be to keep it as simple as possible and unzip specific versions of packages into a tree, avoiding virtualenv (and the games it plays with import paths) if possible. That also keeps your vendoring very intentional, and avoids risk of sprawling second order dependencies + the associated supply chain risks.
John
On Mon, Mar 8, 2021 at 4:14 PM Ernesto Puerta <epuertat@redhat.com> wrote:
Glad to see you here, John!
Any past experience, recommendations on packaging and releasing formats
or patterns for those virtualenvs? I remember that Juanmi was exploring pex (https://github.com/pantsbuild/pex) for cephadm.
Kind Regards,
Ernesto
On Mon, Mar 8, 2021 at 2:44 PM John Spray <jcspray@gmail.com> wrote:
From experience of doing this on a couple of other projects, the thing to watch out for is any python package with native/binary dependencies. Pure python dependencies are easy to embed, things with a binary component are usually not worth the trouble - your vendored python modules can and will break when the underlying .so gets updated in an unexpected way. Or to look at it another way, things with binary dependencies are worthwhile to manage as RPMs, because they have natural RPM dependencies for the libwhatever/libwhatever-devel that they wrap.
You might also want to make sure anything that involves crypto is RPM packaged, to keep the single path for delivering critical fixes to those components.
Annoyingly, the python packaging format doesn't make it easy to distinguish them - lots of things that "pip install" are actually compiling C code with undeclared header dependencies - so it comes down to careful review.
John
On Mon, Mar 8, 2021 at 1:23 PM Ernesto Puerta <epuertat@redhat.com>
wrote:
Additionally, to make this comparison fairer, we should also take
into account not only the code we currently have, but the quality and features we could deliver by relying on mature yet-not-packaged dependencies (I'm thinking on typing-extensions, pydantic, websockets or fastapi among others). Bringing fresh dependencies into the project right now involves ensuring that they're available for a combination of ~5-6 distro-release pairs (Ubuntu, CentOS, OpenSUSE, FreeBSD ports) and that's often a showstopper.
-- Ernesto
On Mon, Mar 8, 2021 at 1:26 PM Alfonso Martinez Hidalgo <
almartin@redhat.com> wrote:
Hi all,
In the last weeks the mgr dashboard module has suffered several bugs related to python package versions:
1) mgr/dashboard: dashboard hangs when accessing it https://tracker.ceph.com/issues/48973 cheroot 8.5.1 was causing a hang: needed to upgrade to cheroot distro version 8.5.2
2) mgr/dashboard: ERROR: test_a_set_login_credentials (tasks.mgr.dashboard.test_auth.AuthTest) https://tracker.ceph.com/issues/49574
Adapt code to PyJWT version >= 2.0.0 (our tests were relying on
version==1.6.4)
3) No possibility to use (in nautilus branch) features from "six" package version like "ensure_text" (available in version >=1.12) due to the fact that distros like ubuntu bionic seem to be stuck on version 1.11
Taking into consideration all the above and the fact that version pinning/constraint can provide stability & reliability, e.g.: https://github.com/psf/requests/blob/master/setup.py#L44 https://github.com/tiangolo/fastapi/blob/master/pyproject.toml#L34
PROPOSAL: Including a virtualenv (or equivalent) in mgr(dashboard) packag(es) (created during build process) with some pinned dependencies so we are no longer dependent on system packages for the specified packages.
We're already doing it for Dashboard Frontend (Angular Single Page App) npm dependencies:
https://github.com/ceph/ceph/blob/master/src/pybind/mgr/dashboard/frontend/p...
PROS: * Stability: no longer exposed to potential bugs coming from side-effects of third-party decisions, like distro maintainers package upgrade policy.
* Reliability: code tested against well-known package versions; avoid the problem of having disparity of package versions in several distros & distro releases.
* Flexibility: possibility of using packages that right now are not available on all supported distros (or required version not available).
CONS: * Package build size increase: we should measure the new size and maybe considering an incremental approach: as a first step, pinning only those packages that are "problematic" (have high version disparity across distros/releases, those not available on supported distros, etc).
* Taking care of CVE-related security updates for pinned versions: 1 possibility is tracking the security updates with tools like this:
https://docs.github.com/en/github/managing-security-vulnerabilities/configur...
Let me know any other considerations that we should take into
account.
Regards, --
Alfonso Martínez
Senior Software Engineer, Ceph Storage
Red Hat
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io