lvm / udev regression, was Re: Re: v20.2.2 Tentacle released

17 Jun 2026

      Confirming Reto's report and adding the root cause, verified against the 
diff.  That regression hit me as well.  Basically, using the lvm system 
for anything else on the same host that uses ceph wedges.  For example, 
booting the host that uses ceph...

Detail:

The regression is PR #67989 ("ceph-volume: include LVM mapper devices in 
get_devices()"). It removed this block from get_devices() in 
ceph_volume/util/disk.py:

    python

    # If the mapper device is a logical volume it gets excluded

    try:
         if UdevData(diskname).is_lvm:
             continue
         except RuntimeError:
             logger.debug("get_devices(): device {} couldn't be
    found.".format(diskname))
             continue

That block did two things, both now gone: it skipped LVM devices, and — 
decisively — it caught the RuntimeError that UdevData.__init__ raises 
when a device path has no udev node.

RAID-internal sub-LVs (<lv>_rmeta_N, <lv>_rimage_N) have no standalone 
udev node. Post-PR, get_devices() calls UdevData() on every dm node 
including those sub-LVs, the RuntimeError is uncaught, and the whole 
inventory aborts — hence CEPHADM_REFRESH_FAILED on any host with an LVM 
RAID1 (or RAID5/6/10) LV.

The PR's added test (test_lvm_device_is_included) only covers an LV with 
a resolvable udev node; the raising path is untested.

Trigger condition: present on v20.2.2; absent on v20.2.1 (where the 
exclude-and-catch block is still in place — verified here, inventory 
clean on a host with multiple RAID1 LVs).

Any host whose OS/boot mirror uses LVM RAID1 is affected; the run aborts 
on the first RAID-internal sub-LV enumerated.
LVM filter/global_filter is not a workaround — those act on PV device 
paths, not LV names, so a reject of *_rmeta_*/*_rimage_* leaves the 
sub-LVs fully visible to ceph-volume's /sys/block walk.

Suggested fix: restore handling for LVs whose UdevData can't resolve a 
udev node — re-add the except RuntimeError: continue, or skip LVs whose 
LVM role is RAID-internal (the [bracketed], private,raid,metadata/image 
LVs) during enumeration. Reto's "only enumerate public-role LVs" framing 
is the same idea.

Workaround until fixed: stay on v20.2.1 for hosts with RAID-backed OS 
mirrors.

On 6/17/26 05:07, Reto Gysi via ceph-users wrote:
...
Hi
I've noticed the following bug/regression with Tentacle release v20.2.2
Ceph health checks are failing because there are existing raid mirror
volumes on the node hosts, and ceph-volume is trying to read the "raid
internal" logical volumes
[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
    host zephir `cephadm ceph-volume` failed: cephadm exited with an error
code: 1, stderr: Inferring config
/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/mon.zephir/config
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/cep
h/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e
CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v
/var/log/ceph/27923302-87a5-11e
c-ac5b-976d21a49941:/var/log/ceph:z -v
/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v
/run/udev:/run/ude
v -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v
/:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2
inventory
--format=json-pretty --with-lsm --filter-for-batch
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr   File "/usr/sbin/ceph-volume", line 33, in <module>
/usr/bin/podman: stderr     sys.exit(load_entry_point('ceph-volume==1.0.0',
'console_scripts', 'ceph-volume')())
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__
/usr/bin/podman: stderr     self.main(self.argv)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in
newfunc
/usr/bin/podman: stderr     return f(*a, **kw)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main
/usr/bin/podman: stderr     terminal.dispatch(self.mapper, subcommand_args)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in
dispatch
/usr/bin/podman: stderr     instance.main()
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58,
in main
/usr/bin/podman: stderr
     self.format_report(Devices(filter_for_batch=self.args.filter_for_batch,
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in
__init__
/usr/bin/podman: stderr     self._devices = [Device(k,
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in
<listcomp>
/usr/bin/podman: stderr     self._devices = [Device(k,
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in
__init__
/usr/bin/podman: stderr     self.device_id = self._get_device_id()
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in
_get_device_id
/usr/bin/podman: stderr     udev_data = disk.UdevData(self.path)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in
__init__
/usr/bin/podman: stderr     raise RuntimeError(f'{path} not found.')
/usr/bin/podman: stderr RuntimeError: /dev/debian/var_rmeta_0 not found.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 5296, in <module>
    main()
    ~~~~^^
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 5284, in main
    r = ctx.func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 397, in _infer_config
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 312, in _infer_fsid
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 425, in _infer_image
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 299, in _validate_fsid
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 3231, in command_ceph_volume
    out, err, code = call_throws(ctx, c.run_cmd(),
verbosity=CallVerbosity.QUIET_UNLESS_ERROR)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py",
line 310, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io
/ceph/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes
-e CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v
/var/log/ceph/27923302-87a5
-11ec-ac5b-976d21a49941:/var/log/ceph:z -v
/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run
/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v
/:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inven
tory --format=json-pretty --with-lsm --filter-for-batch
lvs output example:
root@zephir:~# lvs -a -o +role -o +devices |head -1; lvs -a -o +role -o
+devices| grep var
  LV                                             VG
                                        Attr       LSize    Pool Origin
                         Data%  Meta%  Move Log Cpy%Sync Convert Role
                               Devices
var                                            debian
                                    rwi-aor--- <139.70g
                                                            100.00
           public
                               var_rimage_0(0),var_rimage_1(0)
var_backup                                     debian
                                    -wi-ao----  120.00g
public
/dev/mapper/sdc1_crypt(23040)
[var_rimage_0]                                 debian
                                    iwi-aor--- <139.70g
private,ra
id,image                       /dev/mapper/nvme0n1p3_crypt(71525)
[var_rimage_1]                                 debian
                                    iwi-aor--- <139.70g
private,ra
id,image                       /dev/mapper/nvme1n1p1_crypt(214578)
[var_rmeta_0]                                  debian
                                    ewi-aor---    4.00m
private,ra
id,metadata                    /dev/mapper/nvme0n1p3_crypt(0)
[var_rmeta_1]                                  debian
                                    ewi-aor---    4.00m
private,ra
id,metadata                    /dev/mapper/nvme1n1p1_crypt(214577)
var-lib-ceph                                   debian-4k
                                 rwi-aor---   40.00g
                                                            100.00
           public
                               var-lib-ceph_rimage_0(0),var-lib-ceph_rimage_1(0)
[var-lib-ceph_rimage_0]                        debian-4k
                                 iwi-aor---   40.00g
private,ra
id,image                       /dev/mapper/nvme3n1_crypt(55050)
[var-lib-ceph_rimage_1]                        debian-4k
                                 iwi-aor---   40.00g
private,ra
id,image                       /dev/mapper/nvme4n1_crypt(786433)
[var-lib-ceph_rmeta_0]                         debian-4k
                                 ewi-aor---    4.00m
private,ra
id,metadata                    /dev/mapper/nvme3n1_crypt(55049)
[var-lib-ceph_rmeta_1]                         debian-4k
                                 ewi-aor---    4.00m
private,ra
id,metadata                    /dev/mapper/nvme4n1_crypt(786432)
root@zephir:~#
Maybe ceph-volume should only try to read the logical volumes with Role
'public' and not the lv raid internal ones.
Those volumes are used to host the standard OS installation directories
(/,/var,etc) on mirrored volumes.
...
ceph-volume
-----------
* Inventory Scanning: Enhanced inventory discovery logic to
automatically skip RAM disk devices (/dev/ram*).
To unsubscribe send an email toceph-users-leave@ceph.io
Thanks & Regards
Reto
_______________________________________________
ceph-users mailing list --ceph-users@ceph.io
To unsubscribe send an email toceph-users-leave@ceph.io

Harry G Coin

tags

participants (1)