8 Mar
2021
8 Mar
'21
1:43 p.m.
From experience of doing this on a couple of other projects, the thing
to watch out for is any python package with native/binary
dependencies. Pure python dependencies are easy to embed, things with
a binary component are usually not worth the trouble - your vendored
python modules can and will break when the underlying .so gets updated
in an unexpected way. Or to look at it another way, things with
binary dependencies are worthwhile to manage as RPMs, because they
have natural RPM dependencies for the libwhatever/libwhatever-devel
that they wrap.
You might also want to make sure anything that involves crypto is RPM
packaged, to keep the single path for delivering critical fixes to
those components.
Annoyingly, the python packaging format doesn't make it easy to
distinguish them - lots of things that "pip install" are actually
compiling C code with undeclared header dependencies - so it comes
down to careful review.
John
On Mon, Mar 8, 2021 at 1:23 PM Ernesto Puerta <epuertat(a)redhat.com> wrote:
>
> Additionally, to make this comparison fairer, we should also take into account not
only the code we currently have, but the quality and features we could deliver by relying
on mature yet-not-packaged dependencies (I'm thinking on typing-extensions, pydantic,
websockets or fastapi among others). Bringing fresh dependencies into the project right
now involves ensuring that they're available for a combination of ~5-6 distro-release
pairs (Ubuntu, CentOS, OpenSUSE, FreeBSD ports) and that's often a showstopper.
>
> --
> Ernesto
>
>
> On Mon, Mar 8, 2021 at 1:26 PM Alfonso Martinez Hidalgo <almartin(a)redhat.com>
wrote:
>>
>> Hi all,
>>
>> In the last weeks the mgr dashboard module has suffered several bugs
>> related to python package versions:
>>
>> 1) mgr/dashboard: dashboard hangs when accessing it
>> https://tracker.ceph.com/issues/48973
>> cheroot 8.5.1 was causing a hang: needed to upgrade to cheroot distro
>> version 8.5.2
>>
>> 2) mgr/dashboard: ERROR: test_a_set_login_credentials
>> (tasks.mgr.dashboard.test_auth.AuthTest)
>> https://tracker.ceph.com/issues/49574
>>
>> Adapt code to PyJWT version >= 2.0.0 (our tests were relying on
version==1.6.4)
>>
>> 3) No possibility to use (in nautilus branch) features from "six"
>> package version
>> like "ensure_text" (available in version >=1.12) due to the fact
that
>> distros like ubuntu bionic seem to be stuck on version 1.11
>>
>> Taking into consideration all the above and the fact that version
>> pinning/constraint can provide
>> stability & reliability, e.g.:
>> https://github.com/psf/requests/blob/master/setup.py#L44
>> https://github.com/tiangolo/fastapi/blob/master/pyproject.toml#L34
>>
>> PROPOSAL:
>> Including a virtualenv (or equivalent) in mgr(dashboard) packag(es)
>> (created during build process)
>> with some pinned dependencies so we are no longer dependent on system
>> packages for the specified packages.
>>
>> We're already doing it for Dashboard Frontend (Angular Single Page
>> App) npm dependencies:
>>
https://github.com/ceph/ceph/blob/master/src/pybind/mgr/dashboard/frontend/…
>>
>> PROS:
>> * Stability: no longer exposed to potential bugs coming from
>> side-effects of third-party decisions,
>> like distro maintainers package upgrade policy.
>>
>> * Reliability: code tested against well-known package versions; avoid
>> the problem of having
>> disparity of package versions in several distros & distro releases.
>>
>> * Flexibility: possibility of using packages that right now are not
>> available on all supported distros
>> (or required version not available).
>>
>> CONS:
>> * Package build size increase: we should measure the new size and
>> maybe considering an incremental approach:
>> as a first step, pinning only those packages that are "problematic"
>> (have high version disparity across distros/releases, those not
>> available on supported distros, etc).
>>
>> * Taking care of CVE-related security updates for pinned versions:
>> 1 possibility is tracking the security updates with tools like this:
>>
https://docs.github.com/en/github/managing-security-vulnerabilities/configu…
>>
>>
>> Let me know any other considerations that we should take into account.
>>
>> Regards,
>> --
>>
>> Alfonso Martínez
>>
>> Senior Software Engineer, Ceph Storage
>>
>> Red Hat
>>
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io