Failing upgrade 18.2.7 to 19.2.3 - failing to activate via raw takes 5 minutes (before proceeding to lvm)
I'm fighting with a ceph upgrade, going 18.2.7 to 19.2.3. This time again the ceph-volume activate step is taking to long, triggering failures due to systemd service timing out so the orch daemon fails (though the osd does eventually come up, the daemon is still dead, and upgrade halts). I can also reproduce the slowdown of startup with cephadm ceph-volume raw list (I don't use raw devices, but the ceph-volume activation method hardcodes checking raw first https://github.com/ceph/ceph/blob/4d5ad8c1ef04f38d14402f0d89f2df2b7d254c2c/s... ) That's takes 6s on 18.2.7, but 4m32s minutes on 19.2.3 ! I have 42 spinning drives per host (with multipath). It's spending all of it's time in the new method: self.exclude_lvm_osd_devices() and the list of items to scan, given all the duplication from multipath + and mapper names, it ends up with 308 items to scan in my setup. With good old print debugging, i found that while the threadpool speeds things up a bit, it simply takes to long to construct all those Device() objects. In fact, just creating a single Device() object, since it needs to call disk.get_devices() at least once, since this list does not include all devices, it filters out things like "/dev/mapper/mpathxx" from the list, but the code always regenerates (the same) device list if the path isn't found: if not sys_info.devices.get(self.path): sys_info.devices = disk.get_devices() will now force it to re-generate this list >400 times (initial 32 times in parallel, followed by about 400 more which will never match the device name). In the end, it's again O(n^2) computational time to list these raw devices with ceph-volume. So with 32 threads in the pool, it's also now requires running heavy load for 5 minutes before completing this trivial task every time the deamon needs to start.
This is interesting, I was planning to upgrade our own cluster next week from 18.2.7 to 19.2.3 as well, now I'm hesitating. Although we don't have that many OSDs per node, so we probably will not have this issue you're describing. But I can confirm that 'cephadm ceph-volume raw list' on my virtual test environment with only 3 OSDs per node takes around 11 seconds (and empty output). On Reef the output is not empty (probably because exclude_lvm_osd_devices is not present there as I understand it) and it only takes 4 seconds to complete with around 10 OSDs per node. I'll have to check with my colleagues if we should still move forward with the upgrade... Thanks for reporting that! Did you check if there's a tracker issue for that? Thanks, Eugen Zitat von Mikael Öhman <micketeer@gmail.com>:
I'm fighting with a ceph upgrade, going 18.2.7 to 19.2.3.
This time again the ceph-volume activate step is taking to long, triggering failures due to systemd service timing out so the orch daemon fails (though the osd does eventually come up, the daemon is still dead, and upgrade halts).
I can also reproduce the slowdown of startup with cephadm ceph-volume raw list
(I don't use raw devices, but the ceph-volume activation method hardcodes checking raw first https://github.com/ceph/ceph/blob/4d5ad8c1ef04f38d14402f0d89f2df2b7d254c2c/s... )
That's takes 6s on 18.2.7, but 4m32s minutes on 19.2.3 ! I have 42 spinning drives per host (with multipath).
It's spending all of it's time in the new method: self.exclude_lvm_osd_devices() and the list of items to scan, given all the duplication from multipath + and mapper names, it ends up with 308 items to scan in my setup.
With good old print debugging, i found that while the threadpool speeds things up a bit, it simply takes to long to construct all those Device() objects. In fact, just creating a single Device() object, since it needs to call disk.get_devices() at least once, since this list does not include all devices, it filters out things like "/dev/mapper/mpathxx" from the list, but the code always regenerates (the same) device list if the path isn't found:
if not sys_info.devices.get(self.path): sys_info.devices = disk.get_devices()
will now force it to re-generate this list >400 times (initial 32 times in parallel, followed by about 400 more which will never match the device name). In the end, it's again O(n^2) computational time to list these raw devices with ceph-volume. So with 32 threads in the pool, it's also now requires running heavy load for 5 minutes before completing this trivial task every time the deamon needs to start. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Mikael, Thanks for the report. I was also considering upgrading from 19.2.2 to 19.23. Should be related to a change between those 2 versions as I experienced no problem during the 18.2.7 to 19.2.2. upgrade... It reminds me a problem in one of the Quincy update if I'm right with something similar (but probably a different cause) where the device activation was doing just too many times the same command (at that time was a trivial indentation issue in the code)... but at least it seems that activation of many OSDs per node was unsufficiently tested. I don't know if testing was improved... Best regards, Michel Le 14/09/2025 à 10:23, Eugen Block a écrit :
This is interesting, I was planning to upgrade our own cluster next week from 18.2.7 to 19.2.3 as well, now I'm hesitating. Although we don't have that many OSDs per node, so we probably will not have this issue you're describing. But I can confirm that 'cephadm ceph-volume raw list' on my virtual test environment with only 3 OSDs per node takes around 11 seconds (and empty output). On Reef the output is not empty (probably because exclude_lvm_osd_devices is not present there as I understand it) and it only takes 4 seconds to complete with around 10 OSDs per node. I'll have to check with my colleagues if we should still move forward with the upgrade...
Thanks for reporting that! Did you check if there's a tracker issue for that?
Thanks, Eugen
Zitat von Mikael Öhman <micketeer@gmail.com>:
I'm fighting with a ceph upgrade, going 18.2.7 to 19.2.3.
This time again the ceph-volume activate step is taking to long, triggering failures due to systemd service timing out so the orch daemon fails (though the osd does eventually come up, the daemon is still dead, and upgrade halts).
I can also reproduce the slowdown of startup with cephadm ceph-volume raw list
(I don't use raw devices, but the ceph-volume activation method hardcodes checking raw first https://github.com/ceph/ceph/blob/4d5ad8c1ef04f38d14402f0d89f2df2b7d254c2c/s...
)
That's takes 6s on 18.2.7, but 4m32s minutes on 19.2.3 ! I have 42 spinning drives per host (with multipath).
It's spending all of it's time in the new method: self.exclude_lvm_osd_devices() and the list of items to scan, given all the duplication from multipath + and mapper names, it ends up with 308 items to scan in my setup.
With good old print debugging, i found that while the threadpool speeds things up a bit, it simply takes to long to construct all those Device() objects. In fact, just creating a single Device() object, since it needs to call disk.get_devices() at least once, since this list does not include all devices, it filters out things like "/dev/mapper/mpathxx" from the list, but the code always regenerates (the same) device list if the path isn't found:
if not sys_info.devices.get(self.path): sys_info.devices = disk.get_devices()
will now force it to re-generate this list >400 times (initial 32 times in parallel, followed by about 400 more which will never match the device name). In the end, it's again O(n^2) computational time to list these raw devices with ceph-volume. So with 32 threads in the pool, it's also now requires running heavy load for 5 minutes before completing this trivial task every time the deamon needs to start. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all Yes, there was also a similar O(n^2) bug caused by indentation (in lsblk_all if i recall correctly). That time it took well over 15 minutes for me to run through it, so it was even worse. This time, it's not quite such a subtle bug. I plan to write up a bug report on this with the details, I just got my bug tracker account approved. I suspect only users with large jbods + multipath would see this issue as bad as I do. If I didn't have multipath devices that caused re-triggering the expensive "disk.get_devices()" repeatedly, it would "only" have taken me an extra ~30 seconds to launch an OSD daemon. Not good, but it would be well within the systemd timeout and wouldn't break the daemon completely. It's slow because "ceph-volume activate" also attempts to find raw devices before proceeding to lvm. The change was introduced in https://github.com/ceph/ceph/pull/60395 and I can confirm from https://github.com/ceph/ceph/blob/v19.2.2/src/ceph-volume/ceph_volume/device... it does not have the specific problematic code. In 19.2.2 it's still technically O(n^2) but since it uses a local info_devices variable that's just generated once, it won't have the multipath issue that makes it 10x worse. https://github.com/ceph/ceph/blob/v19.2.2/src/ceph-volume/ceph_volume/device... I worked around this problem by setting up a little container mirror locally where i monkeypatched out the raw part from ceph-volume activate. Dockerfile: FROM quay.io/ceph/ceph:v19.2.3 RUN sed -i '46,52d' /usr/lib/python3.9/site-packages/ceph_volume/activate/main.py which just deletes the "first try raw" section from ceph-volume activate: https://github.com/ceph/ceph/blob/50d6a3d454763cea76ca45a846cde9702364c773/s... since like all recommended setups these days I use LVMs for all devices (I don't understand why one must try raw first) ceph-volume raw list still takes 5 minutes (and it correctly outputs 0 devices as i don't use raw) but i don't care about that since i will only use "ceph-volume lvm list". At least this way activation is fast. On Sun, Sep 14, 2025 at 11:46 AM Michel Jouvin < michel.jouvin@ijclab.in2p3.fr> wrote:
Hi Mikael,
Thanks for the report. I was also considering upgrading from 19.2.2 to 19.23. Should be related to a change between those 2 versions as I experienced no problem during the 18.2.7 to 19.2.2. upgrade... It reminds me a problem in one of the Quincy update if I'm right with something similar (but probably a different cause) where the device activation was doing just too many times the same command (at that time was a trivial indentation issue in the code)... but at least it seems that activation of many OSDs per node was unsufficiently tested. I don't know if testing was improved...
Best regards,
Michel
Le 14/09/2025 à 10:23, Eugen Block a écrit :
This is interesting, I was planning to upgrade our own cluster next week from 18.2.7 to 19.2.3 as well, now I'm hesitating. Although we don't have that many OSDs per node, so we probably will not have this issue you're describing. But I can confirm that 'cephadm ceph-volume raw list' on my virtual test environment with only 3 OSDs per node takes around 11 seconds (and empty output). On Reef the output is not empty (probably because exclude_lvm_osd_devices is not present there as I understand it) and it only takes 4 seconds to complete with around 10 OSDs per node. I'll have to check with my colleagues if we should still move forward with the upgrade...
Thanks for reporting that! Did you check if there's a tracker issue for that?
Thanks, Eugen
Zitat von Mikael Öhman <micketeer@gmail.com>:
I'm fighting with a ceph upgrade, going 18.2.7 to 19.2.3.
This time again the ceph-volume activate step is taking to long, triggering failures due to systemd service timing out so the orch daemon fails (though the osd does eventually come up, the daemon is still dead, and upgrade halts).
I can also reproduce the slowdown of startup with cephadm ceph-volume raw list
(I don't use raw devices, but the ceph-volume activation method hardcodes checking raw first
https://github.com/ceph/ceph/blob/4d5ad8c1ef04f38d14402f0d89f2df2b7d254c2c/s...
)
That's takes 6s on 18.2.7, but 4m32s minutes on 19.2.3 ! I have 42 spinning drives per host (with multipath).
It's spending all of it's time in the new method: self.exclude_lvm_osd_devices() and the list of items to scan, given all the duplication from multipath + and mapper names, it ends up with 308 items to scan in my setup.
With good old print debugging, i found that while the threadpool speeds things up a bit, it simply takes to long to construct all those Device() objects. In fact, just creating a single Device() object, since it needs to call disk.get_devices() at least once, since this list does not include all devices, it filters out things like "/dev/mapper/mpathxx" from the list, but the code always regenerates (the same) device list if the path isn't found:
if not sys_info.devices.get(self.path): sys_info.devices = disk.get_devices()
will now force it to re-generate this list >400 times (initial 32 times in parallel, followed by about 400 more which will never match the device name). In the end, it's again O(n^2) computational time to list these raw devices with ceph-volume. So with 32 threads in the pool, it's also now requires running heavy load for 5 minutes before completing this trivial task every time the deamon needs to start. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I finally got around to open my first issue on this, cf. https://tracker.ceph.com/issues/73107 On Sun, Sep 14, 2025 at 2:22 PM Mikael Öhman <micketeer@gmail.com> wrote:
Hi all
Yes, there was also a similar O(n^2) bug caused by indentation (in lsblk_all if i recall correctly). That time it took well over 15 minutes for me to run through it, so it was even worse. This time, it's not quite such a subtle bug.
I plan to write up a bug report on this with the details, I just got my bug tracker account approved.
I suspect only users with large jbods + multipath would see this issue as bad as I do. If I didn't have multipath devices that caused re-triggering the expensive "disk.get_devices()" repeatedly, it would "only" have taken me an extra ~30 seconds to launch an OSD daemon. Not good, but it would be well within the systemd timeout and wouldn't break the daemon completely. It's slow because "ceph-volume activate" also attempts to find raw devices before proceeding to lvm.
The change was introduced in https://github.com/ceph/ceph/pull/60395 and I can confirm from https://github.com/ceph/ceph/blob/v19.2.2/src/ceph-volume/ceph_volume/device... it does not have the specific problematic code. In 19.2.2 it's still technically O(n^2) but since it uses a local info_devices variable that's just generated once, it won't have the multipath issue that makes it 10x worse.
https://github.com/ceph/ceph/blob/v19.2.2/src/ceph-volume/ceph_volume/device...
I worked around this problem by setting up a little container mirror locally where i monkeypatched out the raw part from ceph-volume activate. Dockerfile: FROM quay.io/ceph/ceph:v19.2.3 RUN sed -i '46,52d' /usr/lib/python3.9/site-packages/ceph_volume/activate/main.py
which just deletes the "first try raw" section from ceph-volume activate:
https://github.com/ceph/ceph/blob/50d6a3d454763cea76ca45a846cde9702364c773/s... since like all recommended setups these days I use LVMs for all devices (I don't understand why one must try raw first) ceph-volume raw list still takes 5 minutes (and it correctly outputs 0 devices as i don't use raw) but i don't care about that since i will only use "ceph-volume lvm list". At least this way activation is fast.
On Sun, Sep 14, 2025 at 11:46 AM Michel Jouvin < michel.jouvin@ijclab.in2p3.fr> wrote:
Hi Mikael,
Thanks for the report. I was also considering upgrading from 19.2.2 to 19.23. Should be related to a change between those 2 versions as I experienced no problem during the 18.2.7 to 19.2.2. upgrade... It reminds me a problem in one of the Quincy update if I'm right with something similar (but probably a different cause) where the device activation was doing just too many times the same command (at that time was a trivial indentation issue in the code)... but at least it seems that activation of many OSDs per node was unsufficiently tested. I don't know if testing was improved...
Best regards,
Michel
Le 14/09/2025 à 10:23, Eugen Block a écrit :
This is interesting, I was planning to upgrade our own cluster next week from 18.2.7 to 19.2.3 as well, now I'm hesitating. Although we don't have that many OSDs per node, so we probably will not have this issue you're describing. But I can confirm that 'cephadm ceph-volume raw list' on my virtual test environment with only 3 OSDs per node takes around 11 seconds (and empty output). On Reef the output is not empty (probably because exclude_lvm_osd_devices is not present there as I understand it) and it only takes 4 seconds to complete with around 10 OSDs per node. I'll have to check with my colleagues if we should still move forward with the upgrade...
Thanks for reporting that! Did you check if there's a tracker issue for that?
Thanks, Eugen
Zitat von Mikael Öhman <micketeer@gmail.com>:
I'm fighting with a ceph upgrade, going 18.2.7 to 19.2.3.
This time again the ceph-volume activate step is taking to long, triggering failures due to systemd service timing out so the orch daemon fails (though the osd does eventually come up, the daemon is still dead, and upgrade halts).
I can also reproduce the slowdown of startup with cephadm ceph-volume raw list
(I don't use raw devices, but the ceph-volume activation method hardcodes checking raw first
https://github.com/ceph/ceph/blob/4d5ad8c1ef04f38d14402f0d89f2df2b7d254c2c/s...
)
That's takes 6s on 18.2.7, but 4m32s minutes on 19.2.3 ! I have 42 spinning drives per host (with multipath).
It's spending all of it's time in the new method: self.exclude_lvm_osd_devices() and the list of items to scan, given all the duplication from multipath + and mapper names, it ends up with 308 items to scan in my setup.
With good old print debugging, i found that while the threadpool speeds things up a bit, it simply takes to long to construct all those
Device()
objects. In fact, just creating a single Device() object, since it needs to call disk.get_devices() at least once, since this list does not include all devices, it filters out things like "/dev/mapper/mpathxx" from the list, but the code always regenerates (the same) device list if the path isn't found:
if not sys_info.devices.get(self.path): sys_info.devices = disk.get_devices()
will now force it to re-generate this list >400 times (initial 32 times in parallel, followed by about 400 more which will never match the device name). In the end, it's again O(n^2) computational time to list these raw devices with ceph-volume. So with 32 threads in the pool, it's also now requires running heavy load for 5 minutes before completing this trivial task every time the deamon needs to start. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Michel Jouvin
-
Mikael Öhman