[ceph-users] Re: lvm / udev regression, was Re: Re: v20.2.2 Tentacle released

18 Jun 2026

      Hi,

PR #67989 intentionally stopped excluding all LVM mapper devices so that
public LVs show up in 'ceph-volume inventory --list-all'. The regression
was that we dropped both the RuntimeError guard and any filtering of
LVM internal RAID legs (_rmeta_/_rimage_/ etc.), which have dm nodes in
/sys/block but no standalone udev node

The follow up PF (tracker #77486) addresses this without reverting #67989:
1/ in get_devices(), it catches RuntimeError from UdevData and skips the device.
2/ it sskips LVM internal RAID/thin-raid legs via is_internal_lv
   (_rmeta_/_rimage_/_rtmeta_/_rtimage_ and DM_LV_LAYER).
3/ + skip LVs whose preferred path is not an accessible block device.
4/ finally, in Device._parse(), it doesn't rewrite the path to a missing /dev/vg/lv.
Public LVs with a resolvable node remain included, only internal/hidden
legs and broken udev paths are dropped. The tests cover internal RAID
exclusion and missing node exclusion.

Regards,

--
Guillaume Abrioux
Software Engineer
From: Harry G Coin via ceph-users <ceph-users@ceph.io>
Date: Wednesday, 17 June 2026 at 18:19
To: ceph-users@ceph.io <ceph-users@ceph.io>
Subject: [EXTERNAL] [ceph-users] lvm / udev regression, was Re: Re: v20.2.2 Tentacle released

Confirming Reto's report and adding the root cause, verified against the
diff.  That regression hit me as well.  Basically, using the lvm system
for anything else on the same host that uses ceph wedges.  For example,
booting the host that uses ceph...

Detail:

The regression is PR #67989 ("ceph-volume: include LVM mapper devices in
get_devices()"). It removed this block from get_devices() in
ceph_volume/util/disk.py:

    python

    # If the mapper device is a logical volume it gets excluded

    try:
         if UdevData(diskname).is_lvm:
             continue
         except RuntimeError:
             logger.debug("get_devices(): device {} couldn't be
    found.".format(diskname))
             continue

That block did two things, both now gone: it skipped LVM devices, and —
decisively — it caught the RuntimeError that UdevData.__init__ raises
when a device path has no udev node.

RAID-internal sub-LVs (<lv>_rmeta_N, <lv>_rimage_N) have no standalone
udev node. Post-PR, get_devices() calls UdevData() on every dm node
including those sub-LVs, the RuntimeError is uncaught, and the whole
inventory aborts — hence CEPHADM_REFRESH_FAILED on any host with an LVM
RAID1 (or RAID5/6/10) LV.

The PR's added test (test_lvm_device_is_included) only covers an LV with
a resolvable udev node; the raising path is untested.

Trigger condition: present on v20.2.2; absent on v20.2.1 (where the
exclude-and-catch block is still in place — verified here, inventory
clean on a host with multiple RAID1 LVs).

Any host whose OS/boot mirror uses LVM RAID1 is affected; the run aborts
on the first RAID-internal sub-LV enumerated.
LVM filter/global_filter is not a workaround — those act on PV device
paths, not LV names, so a reject of *_rmeta_*/*_rimage_* leaves the
sub-LVs fully visible to ceph-volume's /sys/block walk.

Suggested fix: restore handling for LVs whose UdevData can't resolve a
udev node — re-add the except RuntimeError: continue, or skip LVs whose
LVM role is RAID-internal (the [bracketed], private,raid,metadata/image
LVs) during enumeration. Reto's "only enumerate public-role LVs" framing
is the same idea.

Workaround until fixed: stay on v20.2.1 for hosts with RAID-backed OS
mirrors.

On 6/17/26 05:07, Reto Gysi via ceph-users wrote:
...
Hi
I've noticed the following bug/regression with Tentacle release v20.2.2
Ceph health checks are failing because there are existing raid mirror
volumes on the node hosts, and ceph-volume is trying to read the "raid
internal" logical volumes
[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
    host zephir `cephadm ceph-volume` failed: cephadm exited with an error
code: 1, stderr: Inferring config
/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/mon.zephir/config
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/cep
h/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e
CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v
/var/log/ceph/27923302-87a5-11e
c-ac5b-976d21a49941:/var/log/ceph:z -v
/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v
/run/udev:/run/ude
v -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v
/:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2
inventory
--format=json-pretty --with-lsm --filter-for-batch
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr   File "/usr/sbin/ceph-volume", line 33, in <module>
/usr/bin/podman: stderr     sys.exit(load_entry_point('ceph-volume==1.0.0',
'console_scripts', 'ceph-volume')())
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__
/usr/bin/podman: stderr     self.main(self.argv)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in
newfunc
/usr/bin/podman: stderr     return f(*a, **kw)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main
/usr/bin/podman: stderr     terminal.dispatch(self.mapper, subcommand_args)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in
dispatch
/usr/bin/podman: stderr     instance.main()
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58,
in main
/usr/bin/podman: stderr
     self.format_report(Devices(filter_for_batch=self.args.filter_for_batch,
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in
__init__
/usr/bin/podman: stderr     self._devices = [Device(k,
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in
<listcomp>
/usr/bin/podman: stderr     self._devices = [Device(k,
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in
__init__
/usr/bin/podman: stderr     self.device_id = self._get_device_id()
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in
_get_device_id
/usr/bin/podman: stderr     udev_data = disk.UdevData(self.path)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in
__init__
/usr/bin/podman: stderr     raise RuntimeError(f'{path} not found.')
/usr/bin/podman: stderr RuntimeError: /dev/debian/var_rmeta_0 not found.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 5296, in <module>
    main()
    ~~~~^^
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 5284, in main
    r = ctx.func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 397, in _infer_config
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 312, in _infer_fsid
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 425, in _infer_image
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 299, in _validate_fsid
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 3231, in command_ceph_volume
    out, err, code = call_throws(ctx, c.run_cmd(),
verbosity=CallVerbosity.QUIET_UNLESS_ERROR)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py",
line 310, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io
/ceph/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes
-e CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v
/var/log/ceph/27923302-87a5
-11ec-ac5b-976d21a49941:/var/log/ceph:z -v
/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run
/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v
/:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inven
tory --format=json-pretty --with-lsm --filter-for-batch
lvs output example:
root@zephir:~# lvs -a -o +role -o +devices |head -1; lvs -a -o +role -o
+devices| grep var
  LV                                             VG
                                        Attr       LSize    Pool Origin
                         Data%  Meta%  Move Log Cpy%Sync Convert Role
                               Devices
var                                            debian
                                    rwi-aor--- <139.70g
                                                            100.00
           public
                               var_rimage_0(0),var_rimage_1(0)
var_backup                                     debian
                                    -wi-ao----  120.00g
public
/dev/mapper/sdc1_crypt(23040)
[var_rimage_0]                                 debian
                                    iwi-aor--- <139.70g
private,ra
id,image                       /dev/mapper/nvme0n1p3_crypt(71525)
[var_rimage_1]                                 debian
                                    iwi-aor--- <139.70g
private,ra
id,image                       /dev/mapper/nvme1n1p1_crypt(214578)
[var_rmeta_0]                                  debian
                                    ewi-aor---    4.00m
private,ra
id,metadata                    /dev/mapper/nvme0n1p3_crypt(0)
[var_rmeta_1]                                  debian
                                    ewi-aor---    4.00m
private,ra
id,metadata                    /dev/mapper/nvme1n1p1_crypt(214577)
var-lib-ceph                                   debian-4k
                                 rwi-aor---   40.00g
                                                            100.00
           public
                               var-lib-ceph_rimage_0(0),var-lib-ceph_rimage_1(0)
[var-lib-ceph_rimage_0]                        debian-4k
                                 iwi-aor---   40.00g
private,ra
id,image                       /dev/mapper/nvme3n1_crypt(55050)
[var-lib-ceph_rimage_1]                        debian-4k
                                 iwi-aor---   40.00g
private,ra
id,image                       /dev/mapper/nvme4n1_crypt(786433)
[var-lib-ceph_rmeta_0]                         debian-4k
                                 ewi-aor---    4.00m
private,ra
id,metadata                    /dev/mapper/nvme3n1_crypt(55049)
[var-lib-ceph_rmeta_1]                         debian-4k
                                 ewi-aor---    4.00m
private,ra
id,metadata                    /dev/mapper/nvme4n1_crypt(786432)
root@zephir:~#
Maybe ceph-volume should only try to read the logical volumes with Role
'public' and not the lv raid internal ones.
Those volumes are used to host the standard OS installation directories
(/,/var,etc) on mirrored volumes.
...
ceph-volume
-----------
* Inventory Scanning: Enhanced inventory discovery logic to
automatically skip RAM disk devices (/dev/ram*).
To unsubscribe send an email toceph-users-leave@ceph.io
Thanks & Regards
Reto
_______________________________________________
ceph-users mailing list --ceph-users@ceph.io
To unsubscribe send an email toceph-users-leave@ceph.io

ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io

Unless otherwise stated above:

Compagnie IBM France
Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex
RCS Nanterre 552 118 465
Forme Sociale : S.A.S.
Capital Social : 664 614 175,50 €
SIRET : 552 118 465 03644 - Code NAF 6203Z

[ceph-users] Re: lvm / udev regression, was Re: Re: v20.2.2 Tentacle released

Guillaume ABRIOUX