Re: Debian 12 (bookworm) / Reef 18.2.1 problems
I have logged this as https://tracker.ceph.com/issues/64213 On 16/01/2024 14:18, DERUMIER, Alexandre wrote:
Hi,
ImportError: PyO3 modules may only be initialized once per interpreter process
and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process We have the same problem on proxmox8 (based on debian12) with ceph quincy or reef.
It seem to be related to python version on debian12
(we have no fix for this currently)
Chris, Thanks for all the investigations you are doing here. We're on quincy/debian11. Is there any working path at this point to reef/debian12? Ideally I want to go in two steps. Upgrade ceph first or upgrade debian first, then do the upgrade to the other one. Most of our infra is already upgraded to debian 12, except ceph. On 2024-01-29 07:27, Chris Palmer wrote:
I have logged this as https://tracker.ceph.com/issues/64213
On 16/01/2024 14:18, DERUMIER, Alexandre wrote:
Hi,
ImportError: PyO3 modules may only be initialized once per interpreter process
and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process We have the same problem on proxmox8 (based on debian12) with ceph quincy or reef.
It seem to be related to python version on debian12
(we have no fix for this currently)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Matthew AFAIK the upgrade from quincy/deb11 to reef/deb12 is not possible: * The packaging problem you can work around, and a fix is pending * You have to upgrade both the OS and Ceph in one step * The MGR will not run under deb12 due to the PyO3 lack of support for subinterpreters. If you do attempt an upgrade, you will end up stuck with a partially upgraded cluster. The MONs will be on deb12/reef and cannot be downgraded, and the MGR will be stuck on deb11/quincy, We have a test cluster in that state with no way forward or back. I fear the MGR problem will spread as time goes on and PyO3 updates occur. And it's not good that it can silently corrupt in the existing apparently-working installations. No-one has picked up issue 64213 that I raised yet. I'm tempted to raise another issue for qa : the debian 12 package cannot have been tested as it just won't work either as an upgrade or a new install. Regards, Chris On 02/02/2024 14:40, Matthew Darwin wrote:
Chris,
Thanks for all the investigations you are doing here. We're on quincy/debian11. Is there any working path at this point to reef/debian12? Ideally I want to go in two steps. Upgrade ceph first or upgrade debian first, then do the upgrade to the other one. Most of our infra is already upgraded to debian 12, except ceph.
On 2024-01-29 07:27, Chris Palmer wrote:
I have logged this as https://tracker.ceph.com/issues/64213
On 16/01/2024 14:18, DERUMIER, Alexandre wrote:
Hi,
ImportError: PyO3 modules may only be initialized once per interpreter process
and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process We have the same problem on proxmox8 (based on debian12) with ceph quincy or reef.
It seem to be related to python version on debian12
(we have no fix for this currently)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Feb 2, 2024 at 11:21 AM Chris Palmer <chris.palmer@idnet.com> wrote:
Hi Matthew
AFAIK the upgrade from quincy/deb11 to reef/deb12 is not possible:
* The packaging problem you can work around, and a fix is pending * You have to upgrade both the OS and Ceph in one step * The MGR will not run under deb12 due to the PyO3 lack of support for subinterpreters.
If you do attempt an upgrade, you will end up stuck with a partially upgraded cluster. The MONs will be on deb12/reef and cannot be downgraded, and the MGR will be stuck on deb11/quincy, We have a test cluster in that state with no way forward or back.
I fear the MGR problem will spread as time goes on and PyO3 updates occur. And it's not good that it can silently corrupt in the existing apparently-working installations.
No-one has picked up issue 64213 that I raised yet.
I'm tempted to raise another issue for qa : the debian 12 package cannot have been tested as it just won't work either as an upgrade or a new install.
you're right that the debian packages don't get tested: https://docs.ceph.com/en/reef/start/os-recommendations/#platforms
Regards, Chris
On 02/02/2024 14:40, Matthew Darwin wrote:
Chris,
Thanks for all the investigations you are doing here. We're on quincy/debian11. Is there any working path at this point to reef/debian12? Ideally I want to go in two steps. Upgrade ceph first or upgrade debian first, then do the upgrade to the other one. Most of our infra is already upgraded to debian 12, except ceph.
On 2024-01-29 07:27, Chris Palmer wrote:
I have logged this as https://tracker.ceph.com/issues/64213
On 16/01/2024 14:18, DERUMIER, Alexandre wrote:
Hi,
ImportError: PyO3 modules may only be initialized once per interpreter process
and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process We have the same problem on proxmox8 (based on debian12) with ceph quincy or reef.
It seem to be related to python version on debian12
(we have no fix for this currently)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Would migrating to a cephadm orchestrated docker/podman cluster be an acceptable workaround? We are running that config with reef containers on Debian 12 hosts, with a couple of debian 12 clients successfully mounting cephfs mounts, using the reef client packages directly on Debian. On Fri, Feb 2, 2024, 8:21 AM Chris Palmer <chris.palmer@idnet.com> wrote:
Hi Matthew
AFAIK the upgrade from quincy/deb11 to reef/deb12 is not possible:
* The packaging problem you can work around, and a fix is pending * You have to upgrade both the OS and Ceph in one step * The MGR will not run under deb12 due to the PyO3 lack of support for subinterpreters.
If you do attempt an upgrade, you will end up stuck with a partially upgraded cluster. The MONs will be on deb12/reef and cannot be downgraded, and the MGR will be stuck on deb11/quincy, We have a test cluster in that state with no way forward or back.
I fear the MGR problem will spread as time goes on and PyO3 updates occur. And it's not good that it can silently corrupt in the existing apparently-working installations.
No-one has picked up issue 64213 that I raised yet.
I'm tempted to raise another issue for qa : the debian 12 package cannot have been tested as it just won't work either as an upgrade or a new install.
Regards, Chris
On 02/02/2024 14:40, Matthew Darwin wrote:
Chris,
Thanks for all the investigations you are doing here. We're on quincy/debian11. Is there any working path at this point to reef/debian12? Ideally I want to go in two steps. Upgrade ceph first or upgrade debian first, then do the upgrade to the other one. Most of our infra is already upgraded to debian 12, except ceph.
On 2024-01-29 07:27, Chris Palmer wrote:
I have logged this as https://tracker.ceph.com/issues/64213
On 16/01/2024 14:18, DERUMIER, Alexandre wrote:
Hi,
ImportError: PyO3 modules may only be initialized once per interpreter process
and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process We have the same problem on proxmox8 (based on debian12) with ceph quincy or reef.
It seem to be related to python version on debian12
(we have no fix for this currently)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We have fundamental problems with the concept of cephadm and its direction of travel. But that's a different story. The nub of this problem is a design incompatibility with MGR and the PyO3 package that python-cryptography relies on. It's actually unsafe as it is, and the new package just stops you performing the unsafe operations. So that affects all distributions and containers and versions of ceph. Eventually the updated PyO3 will find its way into other distributions and containers bringing things to a head. On 02/02/2024 16:45, Brian Chow wrote:
Would migrating to a cephadm orchestrated docker/podman cluster be an acceptable workaround?
We are running that config with reef containers on Debian 12 hosts, with a couple of debian 12 clients successfully mounting cephfs mounts, using the reef client packages directly on Debian.
On Fri, Feb 2, 2024, 8:21 AM Chris Palmer <chris.palmer@idnet.com> wrote:
Hi Matthew
AFAIK the upgrade from quincy/deb11 to reef/deb12 is not possible:
* The packaging problem you can work around, and a fix is pending * You have to upgrade both the OS and Ceph in one step * The MGR will not run under deb12 due to the PyO3 lack of support for subinterpreters.
If you do attempt an upgrade, you will end up stuck with a partially upgraded cluster. The MONs will be on deb12/reef and cannot be downgraded, and the MGR will be stuck on deb11/quincy, We have a test cluster in that state with no way forward or back.
I fear the MGR problem will spread as time goes on and PyO3 updates occur. And it's not good that it can silently corrupt in the existing apparently-working installations.
No-one has picked up issue 64213 that I raised yet.
I'm tempted to raise another issue for qa : the debian 12 package cannot have been tested as it just won't work either as an upgrade or a new install.
Regards, Chris
On 02/02/2024 14:40, Matthew Darwin wrote:
Chris,
Thanks for all the investigations you are doing here. We're on quincy/debian11. Is there any working path at this point to reef/debian12? Ideally I want to go in two steps. Upgrade ceph first or upgrade debian first, then do the upgrade to the other one. Most of our infra is already upgraded to debian 12, except ceph.
On 2024-01-29 07:27, Chris Palmer wrote:
I have logged this as https://tracker.ceph.com/issues/64213
On 16/01/2024 14:18, DERUMIER, Alexandre wrote:
Hi,
> ImportError: PyO3 modules may only be initialized once per > interpreter > process > > and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 > modules may only be initialized once per interpreter process We have the same problem on proxmox8 (based on debian12) with ceph quincy or reef.
It seem to be related to python version on debian12
(we have no fix for this currently)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Just looking back through PyO3 issues, it would appear this functionality was never supported: https://github.com/PyO3/pyo3/issues/3451 https://github.com/PyO3/pyo3/issues/576 It just appears attempting to use this functionality (which does not work/exist) wasn't successfully prevented previously, and is now. I see a few PRs in associated projects (such as bcrypt) where they attempted to rollback (example): https://github.com/pyca/bcrypt/pull/714 This will restore functionality to the way it was before (not sure if the same exists for other libraries) but they are basically stop-gaps until proper support exists in PyO3, which may or may not happen in the near future. It sounds like the rollbacks are being considered if the upstream issue isn't resolved in some undefined timeline. I just wanted to add this information to further the discussion, as I know it does not resolve your immediate problem(s). It sounds like we need to discuss the reliance on PyO3 if this is necessary functionality from the library which isn't actually implemented but was just permitted in error and has an undefined or ill-defined target date of resolution (sounds like a large upstream project). I don't pretend to know the complexities around an alternative implementation, but it seems worth at least a cursory investigation, as behavior right now (prior to the blocking change) may be somewhat undefined even if not throwing errors, according to the above PyO3 issues. David On Fri, Feb 2, 2024, at 10:20, Chris Palmer wrote:
Hi Matthew
AFAIK the upgrade from quincy/deb11 to reef/deb12 is not possible:
* The packaging problem you can work around, and a fix is pending * You have to upgrade both the OS and Ceph in one step * The MGR will not run under deb12 due to the PyO3 lack of support for subinterpreters.
If you do attempt an upgrade, you will end up stuck with a partially upgraded cluster. The MONs will be on deb12/reef and cannot be downgraded, and the MGR will be stuck on deb11/quincy, We have a test cluster in that state with no way forward or back.
I fear the MGR problem will spread as time goes on and PyO3 updates occur. And it's not good that it can silently corrupt in the existing apparently-working installations.
No-one has picked up issue 64213 that I raised yet.
I'm tempted to raise another issue for qa : the debian 12 package cannot have been tested as it just won't work either as an upgrade or a new install.
Regards, Chris
On 02/02/2024 14:40, Matthew Darwin wrote:
Chris,
Thanks for all the investigations you are doing here. We're on quincy/debian11. Is there any working path at this point to reef/debian12? Ideally I want to go in two steps. Upgrade ceph first or upgrade debian first, then do the upgrade to the other one. Most of our infra is already upgraded to debian 12, except ceph.
On 2024-01-29 07:27, Chris Palmer wrote:
I have logged this as https://tracker.ceph.com/issues/64213
On 16/01/2024 14:18, DERUMIER, Alexandre wrote:
Hi,
ImportError: PyO3 modules may only be initialized once per interpreter process
and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process We have the same problem on proxmox8 (based on debian12) with ceph quincy or reef.
It seem to be related to python version on debian12
(we have no fix for this currently)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[mgr modules failing because pyO3 can't be imported more than once] On 29/01/2024 12:27, Chris Palmer wrote:
I have logged this as https://tracker.ceph.com/issues/64213
I've noted there that it's related to https://tracker.ceph.com/issues/63529 (an earlier report relating to the dashboard); there is a MR to fix just the dashboard issue which got merged into main. I've opened a MR to backport that change to Reef: https://github.com/ceph/ceph/pull/55689 I don't know what the devs' plans are for dealing with the broader pyO3 issue, but I'll ask on the dev list... Regards, Matthew
participants (6)
-
Brian Chow
-
Casey Bodley
-
Chris Palmer
-
David Orman
-
Matthew Darwin
-
Matthew Vernon