- ceph-users - lists.ceph.io

Experiences migrating conventional CRUSH rules to MSR at scale?
by Jane Zhu (BLOOMBERG/ 120 PARK) 17 Jun '26

17 Jun '26

Hi all, We're planning a migration from conventional CRUSH rules (chooseleaf firstn/indep) to MSR rules (choosemsr) on large scale production clusters (thousands of OSDs). We've validated the procedure on a test cluster using pg_upmap pinning to avoid data movement during the rule change, then releasing pins in batches to control backfill. I'd appreciate hearing from anyone who has done this at scale. A few specific areas we're trying to get more data on: 1. Peering storm on rule change — changing a pool's crush_rule triggers re-peering for pretty much every PG in that pool. How long did peering take to settle on your clusters, and was client I/O meaningfully impacted? 2. OSDMap bloat from massive pg_upmap — did you hit any issues with OSD map distribution lag, OSDs falling behind on epochs, or MON store growth? Did you batch the upmap application, and if so, what batch size worked well? 3. pg_upmap apply rate — how fast can ceph osd pg-upmap commands be applied in practice at scale? Did you see any MON bottlenecks when applying tens of thousands of upmaps in rapid succession? 4. Pin release strategy — When releasing pg_upmap pins to allow data movement to the new MSR placement, what batch size and pacing worked for you? 5. Anything else we should watch for? For context, in our testing on a Squid cluster, we confirmed that pg_upmap_items does NOT survive a rule change (they are pruned from the OSDMap), so we're using pg_upmap (full acting set override). Thanks in advance for any data points or stories! -Jane

1 0

lvm / udev regression, was Re: Re: v20.2.2 Tentacle released
by Harry G Coin 17 Jun '26

17 Jun '26

Confirming Reto's report and adding the root cause, verified against the diff. That regression hit me as well. Basically, using the lvm system for anything else on the same host that uses ceph wedges. For example, booting the host that uses ceph... Detail: The regression is PR #67989 ("ceph-volume: include LVM mapper devices in get_devices()"). It removed this block from get_devices() in ceph_volume/util/disk.py: python # If the mapper device is a logical volume it gets excluded try: if UdevData(diskname).is_lvm: continue except RuntimeError: logger.debug("get_devices(): device {} couldn't be found.".format(diskname)) continue That block did two things, both now gone: it skipped LVM devices, and — decisively — it caught the RuntimeError that UdevData.__init__ raises when a device path has no udev node. RAID-internal sub-LVs (<lv>_rmeta_N, <lv>_rimage_N) have no standalone udev node. Post-PR, get_devices() calls UdevData() on every dm node including those sub-LVs, the RuntimeError is uncaught, and the whole inventory aborts — hence CEPHADM_REFRESH_FAILED on any host with an LVM RAID1 (or RAID5/6/10) LV. The PR's added test (test_lvm_device_is_included) only covers an LV with a resolvable udev node; the raising path is untested. Trigger condition: present on v20.2.2; absent on v20.2.1 (where the exclude-and-catch block is still in place — verified here, inventory clean on a host with multiple RAID1 LVs). Any host whose OS/boot mirror uses LVM RAID1 is affected; the run aborts on the first RAID-internal sub-LV enumerated. LVM filter/global_filter is not a workaround — those act on PV device paths, not LV names, so a reject of *_rmeta_*/*_rimage_* leaves the sub-LVs fully visible to ceph-volume's /sys/block walk. Suggested fix: restore handling for LVs whose UdevData can't resolve a udev node — re-add the except RuntimeError: continue, or skip LVs whose LVM role is RAID-internal (the [bracketed], private,raid,metadata/image LVs) during enumeration. Reto's "only enumerate public-role LVs" framing is the same idea. Workaround until fixed: stay on v20.2.1 for hosts with RAID-backed OS mirrors. On 6/17/26 05:07, Reto Gysi via ceph-users wrote: > Hi > > I've noticed the following bug/regression with Tentacle release v20.2.2 > Ceph health checks are failing because there are existing raid mirror > volumes on the node hosts, and ceph-volume is trying to read the "raid > internal" logical volumes > > [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices > host zephir `cephadm ceph-volume` failed: cephadm exited with an error > code: 1, stderr: Inferring config > /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/mon.zephir/config > Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host > --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume > --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/cep > h/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e > CEPH_VOLUME_DEBUG=1 -v > /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v > /var/log/ceph/27923302-87a5-11e > c-ac5b-976d21a49941:/var/log/ceph:z -v > /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z > -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v > /run/udev:/run/ude > v -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v > /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v > /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 > inventory > --format=json-pretty --with-lsm --filter-for-batch > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr Traceback (most recent call last): > /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module> > /usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', > 'console_scripts', 'ceph-volume')()) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__ > /usr/bin/podman: stderr self.main(self.argv) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in > newfunc > /usr/bin/podman: stderr return f(*a, **kw) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main > /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in > dispatch > /usr/bin/podman: stderr instance.main() > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58, > in main > /usr/bin/podman: stderr > self.format_report(Devices(filter_for_batch=self.args.filter_for_batch, > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in > __init__ > /usr/bin/podman: stderr self._devices = [Device(k, > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in > <listcomp> > /usr/bin/podman: stderr self._devices = [Device(k, > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in > __init__ > /usr/bin/podman: stderr self.device_id = self._get_device_id() > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in > _get_device_id > /usr/bin/podman: stderr udev_data = disk.UdevData(self.path) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in > __init__ > /usr/bin/podman: stderr raise RuntimeError(f'{path} not found.') > /usr/bin/podman: stderr RuntimeError: /dev/debian/var_rmeta_0 not found. > Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 5296, in <module> > main() > ~~~~^^ > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 5284, in main > r = ctx.func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 397, in _infer_config > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 312, in _infer_fsid > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 425, in _infer_image > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 299, in _validate_fsid > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 3231, in command_ceph_volume > out, err, code = call_throws(ctx, c.run_cmd(), > verbosity=CallVerbosity.QUIET_UNLESS_ERROR) > > ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py", > line 310, in call_throws > raise RuntimeError('Failed command: %s' % ' '.join(command)) > RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host > --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume > --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io > /ceph/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes > -e CEPH_VOLUME_DEBUG=1 -v > /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v > /var/log/ceph/27923302-87a5 > -11ec-ac5b-976d21a49941:/var/log/ceph:z -v > /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z > -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run > /udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v > /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v > /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inven > tory --format=json-pretty --with-lsm --filter-for-batch > > > lvs output example: > root@zephir:~# lvs -a -o +role -o +devices |head -1; lvs -a -o +role -o > +devices| grep var > LV VG > Attr LSize Pool Origin > Data% Meta% Move Log Cpy%Sync Convert Role > Devices > > var debian > rwi-aor--- <139.70g > 100.00 > public > var_rimage_0(0),var_rimage_1(0) > > var_backup debian > -wi-ao---- 120.00g > > public > > /dev/mapper/sdc1_crypt(23040) > > [var_rimage_0] debian > iwi-aor--- <139.70g > > private,ra > id,image /dev/mapper/nvme0n1p3_crypt(71525) > > [var_rimage_1] debian > iwi-aor--- <139.70g > > private,ra > id,image /dev/mapper/nvme1n1p1_crypt(214578) > > [var_rmeta_0] debian > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme0n1p3_crypt(0) > > [var_rmeta_1] debian > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme1n1p1_crypt(214577) > > var-lib-ceph debian-4k > rwi-aor--- 40.00g > 100.00 > public > var-lib-ceph_rimage_0(0),var-lib-ceph_rimage_1(0) > > [var-lib-ceph_rimage_0] debian-4k > iwi-aor--- 40.00g > > private,ra > id,image /dev/mapper/nvme3n1_crypt(55050) > > [var-lib-ceph_rimage_1] debian-4k > iwi-aor--- 40.00g > > private,ra > id,image /dev/mapper/nvme4n1_crypt(786433) > > [var-lib-ceph_rmeta_0] debian-4k > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme3n1_crypt(55049) > > [var-lib-ceph_rmeta_1] debian-4k > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme4n1_crypt(786432) > > root@zephir:~# > > Maybe ceph-volume should only try to read the logical volumes with Role > 'public' and not the lv raid internal ones. > Those volumes are used to host the standard OS installation directories > (/,/var,etc) on mirrored volumes. > > >> ceph-volume >> ----------- >> >> * Inventory Scanning: Enhanced inventory discovery logic to >> automatically skip RAM disk devices (/dev/ram*). >> >> To unsubscribe send an email toceph-users-leave(a)ceph.io > > > Thanks & Regards > > Reto > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

Re: v20.2.2 Tentacle released
by Reto Gysi 17 Jun '26

17 Jun '26

Hi I've noticed the following bug/regression with Tentacle release v20.2.2 Ceph health checks are failing because there are existing raid mirror volumes on the node hosts, and ceph-volume is trying to read the "raid internal" logical volumes [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host zephir `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/mon.zephir/config Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/cep h/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v /var/log/ceph/27923302-87a5-11e c-ac5b-976d21a49941:/var/log/ceph:z -v /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/ude v -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inventory --format=json-pretty --with-lsm --filter-for-batch /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module> /usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__ /usr/bin/podman: stderr self.main(self.argv) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr return f(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58, in main /usr/bin/podman: stderr self.format_report(Devices(filter_for_batch=self.args.filter_for_batch, /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in __init__ /usr/bin/podman: stderr self._devices = [Device(k, /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in <listcomp> /usr/bin/podman: stderr self._devices = [Device(k, /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in __init__ /usr/bin/podman: stderr self.device_id = self._get_device_id() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in _get_device_id /usr/bin/podman: stderr udev_data = disk.UdevData(self.path) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in __init__ /usr/bin/podman: stderr raise RuntimeError(f'{path} not found.') /usr/bin/podman: stderr RuntimeError: /dev/debian/var_rmeta_0 not found. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 5296, in <module> main() ~~~~^^ File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 5284, in main r = ctx.func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 397, in _infer_config return func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 312, in _infer_fsid return func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 425, in _infer_image return func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 299, in _validate_fsid return func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 3231, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py", line 310, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io /ceph/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v /var/log/ceph/27923302-87a5 -11ec-ac5b-976d21a49941:/var/log/ceph:z -v /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run /udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inven tory --format=json-pretty --with-lsm --filter-for-batch lvs output example: root@zephir:~# lvs -a -o +role -o +devices |head -1; lvs -a -o +role -o +devices| grep var LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Role Devices var debian rwi-aor--- <139.70g 100.00 public var_rimage_0(0),var_rimage_1(0) var_backup debian -wi-ao---- 120.00g public /dev/mapper/sdc1_crypt(23040) [var_rimage_0] debian iwi-aor--- <139.70g private,ra id,image /dev/mapper/nvme0n1p3_crypt(71525) [var_rimage_1] debian iwi-aor--- <139.70g private,ra id,image /dev/mapper/nvme1n1p1_crypt(214578) [var_rmeta_0] debian ewi-aor--- 4.00m private,ra id,metadata /dev/mapper/nvme0n1p3_crypt(0) [var_rmeta_1] debian ewi-aor--- 4.00m private,ra id,metadata /dev/mapper/nvme1n1p1_crypt(214577) var-lib-ceph debian-4k rwi-aor--- 40.00g 100.00 public var-lib-ceph_rimage_0(0),var-lib-ceph_rimage_1(0) [var-lib-ceph_rimage_0] debian-4k iwi-aor--- 40.00g private,ra id,image /dev/mapper/nvme3n1_crypt(55050) [var-lib-ceph_rimage_1] debian-4k iwi-aor--- 40.00g private,ra id,image /dev/mapper/nvme4n1_crypt(786433) [var-lib-ceph_rmeta_0] debian-4k ewi-aor--- 4.00m private,ra id,metadata /dev/mapper/nvme3n1_crypt(55049) [var-lib-ceph_rmeta_1] debian-4k ewi-aor--- 4.00m private,ra id,metadata /dev/mapper/nvme4n1_crypt(786432) root@zephir:~# Maybe ceph-volume should only try to read the logical volumes with Role 'public' and not the lv raid internal ones. Those volumes are used to host the standard OS installation directories (/,/var,etc) on mirrored volumes. > > ceph-volume > ----------- > > * Inventory Scanning: Enhanced inventory discovery logic to > automatically skip RAM disk devices (/dev/ram*). > > To unsubscribe send an email to ceph-users-leave(a)ceph.io Thanks & Regards Reto

1 0

Re: v20.2.2 Tentacle released (incomplete EU mirror sync)
by Florian Haas 17 Jun '26

17 Jun '26

Hello, it looks like after the 20.2.2 release, the sync to the EU package mirror was incomplete. Compare: https://eu.ceph.com/debian-tentacle/pool/main/c/ceph/ to https://download.ceph.com/debian-tentacle/pool/main/c/ceph/ Apparently the contents of https://eu.ceph.com/debian-tentacle/dists/noble/ *were* fully replicated though, so this means that the EU mirror now contains an index that refers to .deb files that don't exist there, leading to 404s on "apt upgrade" if this is the only Ceph mirror configured. This issue can be worked around by reconfiguring one's APT sources to download.ceph.com, but it would be great if the EU mirror could also be fixed. Thanks! Cheers, Florian On 16/06/2026 21:51, Patrick Donnelly via ceph-users wrote: > Greetings, > > This is the second minor release in the Tentacle series. > > We recommend that all users update to this release.

1 0

Re: v20.2.2 Tentacle released
by Robert Sander 17 Jun '26

17 Jun '26

Hi, Am 16.06.26 um 9:52 PM schrieb Patrick Donnelly via ceph-users: > ceph-volume > ----------- > > * Inventory Scanning: Enhanced inventory discovery logic to > automatically skip RAM disk devices (/dev/ram*). Why was https://tracker.ceph.com/issues/76043 not included in this release? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

v20.2.2 Tentacle released
by Patrick Donnelly 16 Jun '26

16 Jun '26

Greetings, This is the second minor release in the Tentacle series. We recommend that all users update to this release. Notable Changes --------------- Rocky 10 package-based installs are now supported starting with v20.2.2. Please see the supported platforms ( https://docs.ceph.com/en/latest/start/os-recommendations/#platforms ) or current and planned support in Ceph. MDS (Metadata Server) --------------------- * Segmentation fault fixed due to incorrect queueing of request retries. OSD (Object Storage Daemon) --------------------------- * PGLog Missed List: Fixed a bug to ensure the correct version is attached to the missing list when ignoring log entries. * Data Integrity Asserts: Added assertions to explicitly catch potential corruption in the OSD missing list. RGW (RADOS Gateway) ------------------- * Lifecycle Management: Fixed lifecycle transition issues affecting encrypted multipart objects. * REST & Query Handling: RESTArgs::get_string() now properly URL-decodes incoming query parameters. RADOS / librados / neorados --------------------------- * Linger Operations: Rewrote safety checks to eliminate use-after-free vulnerabilities and LingerOp memory leaks when an unwatch operation returns ENOTCONN. Replaced the flawed is_valid_watch() check with a safe linger_by_cookie() lookup which safely manages LingerOp references using intrusive_ptr. Ensured librados linger callbacks hold a persistent reference to LingerOp to protect against races with simultaneous linger_cancel() requests. Configured librados::IoCtxImpl::aio_unwatch() to asynchronously deliver ENOTCONN to AioCompletion instead of returning the error directly. * Watch/Notify: Fixed neorados notification queue bounds enforcement so that an overflow marker is only appended on the first message exceeding capacity, rather than duplicating it on every subsequent message. Prevented double-cleanup triggers in watch/notify operations when incoming errors arrive after maybe_cleanup() runs. Fixed a bug where notify would lose original error values by ensuring it no longer attempts to decode empty responses. Enhanced io_context shutdown procedures to clear handlers and route cleanly through linger_cancel to avoid use-after-free states. * Async Utilities: Fixed an issue regarding the improper removal of objects from the async service list. Dashboard ----------------- * NVMeoF Revamp: Complete UI overhaul with DHCHAP controller key support, namespace encryption, and secure listeners configuration. * Pools & RGW: Added stretch cluster validation for pools, fixed RGW restart/stop issues, storage class restrictions, sync policy fixes, and MSR EC Profile support. * NFS: Toggle visibility for CephFS snapshots, fixed export creation and path value consistency issues ceph-volume ----------- * Inventory Scanning: Enhanced inventory discovery logic to automatically skip RAM disk devices (/dev/ram*). External Block Device (extblkdev) --------------------------------- * Plugin Stability: Fixed an assertion failure in the FCM plugin when encountering multivolume devices. Testing & QA (Suites / Tooling) ------------------------------- * OS Testing: Added CentOS 9 Stream as a supported host OS for the tentacle-p2p stress-split suite. * Dependency & Compilation: Configured bare-metal upgrade testing workloads (rbd-python) to inject python3-pytest via system packages to prevent test failures. Patched the blogbench workunit in qa/cephfs to gracefully handle "implicit declaration of function" warnings generated by newer GCC versions. Updated Lua scripts in qa/cephfs to respect missing kernels in the YAML configurations. * Boto3 Migration: Replaced outdated boto2 dependencies with boto3 across qa/radosgw_admin and qa/radosgw_admin_rest test frameworks, removing stale request log components and ACL test cases. * Tools: Resolved a crash occurring in tools/ceph-kvstore-tool during database closures, reduced BlueStore header exposure, and added missing compiler guards (WITH_BLUESTORE). We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-2-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.2.2.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 0fcffee29411e3a38036764817b6e1afc59741cc -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: tentacle 20.2.2 QE validation status
by Venky Shankar 16 Jun '26

16 Jun '26

Hi Yuri, On Wed, Jun 10, 2026 at 10:56 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Still seeking approvals: > > fs: Venky fs approved. Failures are https://tracker.ceph.com/projects/cephfs/wiki/Tentacle#Tentacle-v2022 > orch/ceph-volume: Redo & Guillaume > nvmeof: Aviv, Vallari > > > On Tue, Jun 9, 2026 at 1:14 AM Nitzan Mordechai via dev <dev(a)ceph.io> wrote: >> >> Rados approved: https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2022-additional-PR-a… >> ________________________________ >> From: Yuri Weinstein via dev <dev(a)ceph.io> >> Sent: Monday, June 8, 2026 11:54 PM >> To: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> >> Subject: [EXTERNAL] Re: tentacle 20.2.2 QE validation status >> >> We merged several PRs listed here https: //tracker. ceph. com/issues/77140 and re-ran several suites. Seeking approvals: rados: Radek, Nitzan rgw: Adam Emerson fs: Venky orch/ceph-volume: Redo & Guillaume rbd: Ilya nvmeof: Aviv, Vallari upgrade: >> We merged several PRs listed here https://tracker.ceph.com/issues/77140 and re-ran several suites. >> Seeking approvals: >> >> rados: Radek, Nitzan >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Redo & Guillaume >> rbd: Ilya >> nvmeof: Aviv, Vallari >> >> upgrade: Patrick, Nitzan >> >> On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/75878#note-6 >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya >> nvmeof: Aviv, Vallari >> >> upgrade: Patrick >> >> upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD >> upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD >> >> perf-basic: PASSED >> >> crimson-rados: N/A >> >> powercycle: PASSED >> >> TIA -- Cheers, Venky

1 0

Re: How to build ceph containers?
by John Mulligan 12 Jun '26

12 Jun '26

On Friday, June 12, 2026 7:49:18 AM Eastern Daylight Time Adam Prycki via ceph-users wrote: > I have a newbie question. > > How does one build ceph container which can be tested in cephadm cluster? > > I've had success compiling ceph with this command from the documentation > ./src/script/build-with-container.py -d ubuntu22.04 -b build.u2204 -e build > > but container it leaves behind has no ceph binaries in $PATH or /usr/bin. > That's intentional. The container that script produces is meant purely to build ceph. To produce a container for running ceph - one first needs to build the (rpm) packages. You can use the build-with-container.py script and the -e packages with a distro base of rocky10 (`-d rocky10 -e packages`) or centos 9 (`-d centos9 -e packages`), depending on what ceph branch you are targeting. Once you have packages you need to host them in a yum/dnf repo. Then you can use the script ./container/build.sh with the special env var CUSTOM_CEPH_REPO_URL set to your new repo. It's been a few months since I last tried it myself, but that's the process I used. We've batted around the idea of teaching build-with-container.py how to help automate some of the steps after package build (like creating the repo and running a local webserver to host it) - but nothing concrete has really come of that so far. > I was testing on ubunut 22.04 with podman and v19.2.4 git tag. > As of today, we regularly build packages for centos 9, rocky 10, Ubuntu 22.04 (jammy), and Ubuntu 24.04 (noble) on main. The container/Containerfile builds the images based on EL distro base - centos9 and rocky 10. The default was recently updated to rocky 10. To use something other than those two - you'd probably be better off creating your own {Contatainerfile,Dockerfile}. > Second question, how distro specific are these container build scripts? > My colleague couldn't build any images on ubuntu 26.04. > I can build only ubnutu 22.04 images, centos9 and centos10 builds fail > on v19.2.4 and v20.2.1 > Ubuntu 26.04 (resolute) support exists in build-with-container.py but it is not being tested in the CI. It's being discussed as a target for a future ceph release so I expect some more activity on that front soon. I don't expect either the basic build or the package build to work at the moment but I have not tried it myself recently. Each branch will have it's own "supported build matrix" so I am unsure how to help with the 2nd sentence there. Maybe more specific, "I used branch X on host os Y and saw Z failure" would help. > Third question, is there a reliable way to set number of jobs used > during compilation? BUILD_MAKEOPTS=-j32 seems to work for some build > targets but not for all. By default build system chooses -j16 on 32 core VM. > Some of the build targets have different behaviors because they're driven by different scripts internally. The stuff that drives the basic build and 'make check' is a bit different from the package builds for instance. You should be able to pass any env var you want via an environment file. Take a look at this example from the ceph CI : https://github.com/ceph/ceph-build/blob/ b8a350545b563843fc96a7e0f0fb2c1fbeb98246/ceph-pull-requests/build/build#L13- L20 But what vars will be honored is dependent on what you run. Also, feel free to file tracker.ceph.com issues for specific problems you may encounter while trying these tools. You can also reach us on Ceph community slack in the #ceph-build-sig channel if you prefer chat over email. We also hold weekly meetings on 4:30PM Eastern on Wednesdays. Check the ceph community calendar for details. Although, I realize that's pretty late for folks in Europe.

1 0

How to build ceph containers?
by Adam Prycki 12 Jun '26

12 Jun '26

I have a newbie question. How does one build ceph container which can be tested in cephadm cluster? I've had success compiling ceph with this command from the documentation ./src/script/build-with-container.py -d ubuntu22.04 -b build.u2204 -e build but container it leaves behind has no ceph binaries in $PATH or /usr/bin. I was testing on ubunut 22.04 with podman and v19.2.4 git tag. Second question, how distro specific are these container build scripts? My colleague couldn't build any images on ubuntu 26.04. I can build only ubnutu 22.04 images, centos9 and centos10 builds fail on v19.2.4 and v20.2.1 Third question, is there a reliable way to set number of jobs used during compilation? BUILD_MAKEOPTS=-j32 seems to work for some build targets but not for all. By default build system chooses -j16 on 32 core VM. Best regards Adam Prycki

1 0

Re: Scrubbing behind after Quincy to Reef upgrade
by Eugen Block 11 Jun '26

11 Jun '26

I feel like this has been discussed multiple times on this list, I just don't have any links at hand. I suspect mclock settings, most likely some things changed between Quincy and Reef. You could fall back to wpq instead of mclock, it's still a general recommendation at the moment, and then see if anyting improves. Zitat von Reed Dier via ceph-users <ceph-users(a)ceph.io>: > Hello all, > > TL;DR is that the number of concurrent PGs scrubbing, deep or > otherwise, has appeared to increase by about 5-10x, while the number > of PGs complaining that they haven't been scrubbed, deep or > otherwise, has continued to tick higher. >> HEALTH_WARN 217 pgs not deep-scrubbed in time; 187 pgs not scrubbed in time > > > Hoping there may be something that I must have missed in release > notes and mailing list that explains why my scrubs both exploded in > concurrency, as well as fell behind after upgrading from quincy > (17.2.9) to reef (18.2.8). > > Non-cephadm, U22.04, rather heterogenous OSD hardware. > Mix of 8T and 2T HDD, as well as 2T SSD. > HDD's have NVMe WAL/DB of various sizes depending on when they were deployed. > Mix of replicated and EC pools, as well as some replicated pools > across different device classes. > > The vast majority of the PG's that are behind on scrubbing are on EC > pools, and the vast majority of that, is our EC82 cephfs pool (40) > that holds the bulk of our stored data, and the other largest pool > is an older EC73 cephfs pool (37). > My quick and dirty approximation based on PGs last scrubbed last month. > >> ceph pg dump | grep 2026-05 | awk '{print $1" "$27}' | grep -v >> periodic | cut -d '.' -f1 | sort | uniq -c >> dumped all >> 1 17 >> 1 20 >> 116 37 >> 224 40 > > > I didn't make any changes to scrub intervals or mclock profiles > before/during/after the upgrade. >> ceph config dump | grep mclock_profile | awk '{print $4}' | uniq -c ; >> 313 balanced >> ceph config dump | grep scrub_interval >> global class:ssd advanced osd_deep_scrub_interval >> 604800.000000 >> mon advanced osd_deep_scrub_interval >> 604800.000000 >> mon.* advanced osd_deep_scrub_interval >> 604800.000000 >> mgr.* advanced osd_deep_scrub_interval >> 604800.000000 >> osd class:hdd advanced osd_deep_scrub_interval >> 604800.000000 >> osd class:ssd advanced osd_deep_scrub_interval >> 604800.000000 >> osd advanced osd_deep_scrub_interval >> 604800.000000 >> osd.* advanced osd_deep_scrub_interval >> 604800.000000 > > I've tried ceph tell osd.$osd osd_max_scrubs $more, which seems to > somewhat momentarily drive the count of > active+clean+scrubbing[+deep] PGs, but doesn't seem to make a > demonstrative difference in terms of getting ahead in the number of > PGs behind (number continues to grow). > I also looked at load15 across OSD hosts, and they don't appear to > be anywhere near the 50% threshold of osd_scrub_load_threshold > either, so I think I can rule that one out for now. > > I'm mostly curious why the change in behavior of concurrent scrubs > ballooning, and yet the number of PGs behind on scrubbing ballooning > as well, without anything actually changing. > And I'm also curious what tunables I can turn to get things back > under control for scrubbing both short and long term as I look > towards getting to squid and 24.04. > Is there an internal mechanism that triggers a deeper scrub during > first deep scrub after upgrading a major release, reef or otherwise? > > Included some graphs of scrub load over the last 60 and 365 day > period to show prior scrub load that only exceedingly rarely ever > generated a PG_NOT_[DEEP_]SCRUBBED warning, > as well as raw load average (smallest cpu count is 16, and it > doesn't even autoscale to 8, so nothing should be complaining there.) > https://imgur.com/a/rixNrCe > > Appreciate any pointers anyone can steer me towards. > Reed

1 0